Latency detection system and method for streaming tasks

Abstract

The application provides a system and method for detecting latency of streaming tasks. The system comprises a task data acquisition component configured to acquire latency sampling data corresponding to a plurality of streaming tasks, the latency sampling data being used to indicate the degree of latency of the corresponding streaming tasks, the plurality of streaming tasks including N first streaming tasks, and the N first streaming tasks belonging to a first task scenario; and a latency anomaly processing component configured to obtain a latency detection result according to N latency sampling data corresponding to the N first streaming tasks and an anomaly detection model corresponding to the first task scenario, the latency detection result being used to indicate M1 first streaming tasks with latency anomalies in the N first streaming tasks; and N>M1≥0. The difference in latency tolerance under different task scenarios is considered to improve the accuracy of latency anomaly detection of the streaming tasks.

Description

Technical Field

[0001] This application relates to the field of network technology, and in particular to a latency detection system and method for streaming tasks. Background Technology

[0002] In the process of processing streaming tasks through a streaming task platform, in order to monitor the task processing capabilities of the streaming task platform and resolve abnormal states in a timely manner, it is often necessary to detect whether there are any abnormalities in the latency of the streaming tasks in the streaming task platform.

[0003] Currently, the presence of latency anomalies in streaming tasks is determined by whether the latency exceeds a preset value. However, this approach does not consider the differences in latency tolerance across different task scenarios, resulting in low accuracy in latency anomaly detection. Summary of the Invention

[0004] This application provides a latency detection system and method for streaming tasks, taking into account the differences in latency tolerance under different task scenarios, in order to improve the accuracy of latency anomaly detection for streaming tasks.

[0005] In a first aspect, embodiments of this application provide a latency detection system for streaming tasks, comprising: a task data acquisition component, configured to acquire latency sampling data corresponding to multiple streaming tasks, the latency sampling data being used to indicate the latency level of the corresponding streaming tasks, the multiple streaming tasks including streaming tasks under at least one task scenario, the at least one task scenario including a first task scenario, the first task scenario including N first streaming tasks; and a latency anomaly processing component, the latency anomaly processing component being deployed with an anomaly detection model corresponding to each task scenario, the anomaly processing component being configured to determine the anomaly detection model corresponding to the first task scenario, and obtain a latency detection result based on the N latency sampling data corresponding to the N first streaming tasks and the anomaly detection model corresponding to the first task scenario, the latency detection result being used to indicate that there are M1 first streaming tasks with latency anomalies among the N first streaming tasks; wherein, N>M1≥0.

[0006] Secondly, embodiments of this application provide a latency detection method for streaming tasks, comprising: acquiring latency sampling data corresponding to multiple streaming tasks respectively, the latency sampling data being used to indicate the latency level of the corresponding streaming tasks, the multiple streaming tasks including streaming tasks under at least one task scenario, the at least one task scenario including a first task scenario, the first task scenario including N first streaming tasks; determining an anomaly detection model corresponding to the first task scenario; obtaining a latency detection result based on the N latency sampling data corresponding to the N first streaming tasks and the anomaly detection model corresponding to the first task scenario, the latency detection result being used to indicate that there are M1 first streaming tasks with latency anomalies among the N first streaming tasks; wherein, N>M1≥0.

[0007] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method provided in the second aspect.

[0008] Fourthly, embodiments of this application provide a computer program product including computer instructions that, when executed by a processor, implement the method provided in the second aspect.

[0009] In this embodiment, based on the anomaly detection model corresponding to the first task scenario, N first streaming tasks under the first task scenario are detected, which can accurately detect the streaming tasks under the first task scenario. This avoids the problem of low detection accuracy caused by always using the same detection standard to detect streaming tasks in different task scenarios. Attached Figure Description

[0010] Figure 1 A schematic diagram illustrating a scenario for latency detection of streaming tasks provided in an embodiment of this application;

[0011] Figure 2 A schematic diagram of the structure of a latency detection system for streaming tasks provided in an embodiment of this application;

[0012] Figure 3 A flowchart illustrating a latency detection method for streaming tasks provided in an embodiment of this application;

[0013] Figure 4 This is a schematic structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] In big data applications, streaming data (or data streams) refers to a dynamic collection of data that is boundless in terms of time distribution and quantity. The value of the data decreases over time, therefore, rapid (e.g., second-level) responses are needed to enable real-time computation and realize the data's value. Streaming tasks are precisely the real-time computational tasks that process streaming data.

[0015] Figure 1 This is a schematic diagram illustrating a scenario for latency detection of streaming tasks, provided as an embodiment of this application. Figure 1 As shown, the streaming task platform 110 can process received streaming data through streaming tasks to realize streaming computing processes in one or more task processing scenarios. These task scenarios can include, but are not limited to, risk control, advertising, and recommendation scenarios. A streaming data processing process for a single task scenario can be achieved through one or more streaming tasks.

[0016] The streaming task platform 110 can be implemented as a streaming computing cluster, or it can be part of a streaming computing cluster. The streaming computing cluster can be a functional unit with streaming computing capabilities implemented based on streaming computing frameworks such as Flink, Sparak Streams, and Storm. The streaming task platform 110 can be deployed on a server (cluster) or a cloud server (cluster).

[0017] During the processing of streaming tasks, the task processing capability of the streaming task platform 110 can be monitored by detecting whether there are any latency anomalies in the streaming tasks, so as to resolve any abnormal states of the streaming task platform 100 in a timely manner.

[0018] See Figure 1 The latency detection system 120 is connected to the streaming task platform 110 via wired or wireless means. The latency detection system 120 obtains the latency of the streaming task from the streaming task platform 110, and determines that the streaming task has a latency anomaly when the latency of the streaming task is greater than or equal to a preset value. However, this solution does not take into account the differences in latency tolerance under different task scenarios, resulting in low accuracy of latency anomaly detection.

[0019] To address the issue of poor accuracy in detecting latency anomalies, this application provides a latency detection system for streaming tasks, which can determine latency detection results based on latency sampling data of streaming tasks and an anomaly detection model corresponding to the task scenario.

[0020] The latency detection system for streaming tasks provided in this application will be described exemplarily below with reference to the accompanying drawings.

[0021] Figure 2This is a schematic diagram of the structure of a latency detection system 200 for streaming tasks provided in an embodiment of this application. Figure 2 As shown, the latency detection system 200 includes at least: a task data acquisition component 210 and a latency anomaly handling component 220.

[0022] It should be noted that the latency detection system 200 can be used to detect the latency of streaming tasks running in the streaming task platform 110. The latency detection system 200 can be deployed on a server (cluster) or a cloud server (cluster). The latency detection system 200 and the streaming task platform 110 can be deployed on the same server (cluster) or the same cloud server (cluster), and the latency detection system 200 and the streaming task platform 110 on the same server (cluster) or the same cloud server (cluster) can be deployed in a distributed or centralized manner; alternatively, the latency detection system 200 and the streaming task platform 110 can be deployed on different servers (clusters) or cloud servers (clusters), and this application does not impose any limitations on this.

[0023] The task data acquisition component 210 can be used to acquire latency sampling data corresponding to multiple streaming tasks. These multiple streaming tasks can be streaming tasks in the streaming task platform 110 (including running and stopped streaming tasks). These multiple streaming tasks can include streaming tasks under different task scenarios, such as streaming tasks under recommendation scenarios, streaming tasks under search scenarios, streaming tasks under advertising scenarios, streaming tasks under risk control scenarios, etc.

[0024] The task data acquisition component 210 can acquire latency sampling data for each streaming task. For example, the task data acquisition component 210 can collect latency sampling data from multiple streaming tasks from the streaming task platform 110, or it can receive latency sampling data from multiple streaming tasks reported by the streaming task platform 110. For instance, the streaming task management component in the streaming task platform 110 collects latency sampling data for each streaming task and reports the latency sampling data corresponding to each of the multiple streaming tasks to the task data acquisition component 210. Optionally, multiple streaming tasks can be periodically sampled to obtain their latency sampling data.

[0025] It should be noted that latency sampling data can be used to indicate the latency level of the corresponding streaming task. For example, latency sampling data can be related to the sampling time and the start time of the corresponding streaming task. The sampling time is the time taken to sample multiple streaming tasks, i.e., the time to collect the sampled data. The time interval between the sampling time and the start time of the streaming task can reflect the latency of that streaming task. For example, latency sampling data can include the sampling time and the start time of the corresponding streaming task; alternatively, latency sampling data can include the difference between the sampling time and the start time of the corresponding streaming task.

[0026] The task data acquisition component 210 can send the acquired latency sampling data corresponding to multiple streaming tasks to the latency anomaly processing component 220. As mentioned above, multiple streaming tasks can include streaming tasks under multiple different task scenarios. Of course, this application does not exclude the possibility that multiple streaming tasks belong to the same task scenario. The first task scenario can be any one of the multiple task scenarios, and the multiple streaming tasks include N first streaming tasks under the first task scenario.

[0027] In some embodiments, the task data acquisition component 210 may include a task data acquisition unit 211 and a distributed message unit 212. The task data acquisition unit 211 acquires task identifiers and latency sampling data corresponding to multiple streaming tasks, and sends the task identifiers and latency sampling data corresponding to the multiple streaming tasks as multiple latency parameters corresponding to the multiple streaming tasks to the distributed message unit. The task identifier may include at least one of task ID, task name, and reporting machine instance. The distributed message unit 212 sends the latency parameters of the first streaming task to the latency exception handling component 220 through a message consumer.

[0028] In some embodiments, the latency detection system 200 further includes a timing database 260. The timing database 260 receives multiple latency parameters corresponding to multiple streaming tasks from the task data acquisition unit 211, and sends the multiple latency parameters to the distributed message unit 212. This enables timing management and consumption of multiple streaming tasks based on their sampling timing.

[0029] The aforementioned distributed message unit 212 can be implemented as a distributed messaging system. For example, a distributed messaging system can be implemented using the open-source stream processing platform Kafka.

[0030] The aforementioned time-series database 260 can be an implementation database using the Prometheus database, and a data forwarding function has been added based on this database to forward data to a distributed messaging system.

[0031] To improve the accuracy of latency detection, different task scenarios can be assigned to different anomaly detection models. That is, by using an anomaly detection model specific to a task scenario to detect streaming tasks within that scenario, accurate detection of streaming tasks within that scenario can be achieved. This avoids the problem of low detection accuracy caused by always using the same detection criteria to detect streaming tasks across different task scenarios.

[0032] The latency anomaly processing component 220 can perform latency detection on streaming tasks within a specific task scenario using anomaly detection models for different task scenarios. These anomaly detection models can be implemented serially or in parallel, and this application does not impose any limitations on this. Taking the first task scenario as an example, the latency anomaly processing component 220 can determine the anomaly detection model corresponding to the first task scenario and obtain the latency detection result based on the N latency sampling data corresponding to the N first streaming tasks and the anomaly detection model corresponding to the first task scenario. The latency detection result is used to indicate M1 first streaming tasks among the N first streaming tasks that have latency anomalies, where N > M1 ≥ 0, and both N and M1 are integers.

[0033] For example, this anomaly detection model can be deployed on, for instance, Figure 2 The delay anomaly detection unit (221-1 to 221-n) in the middle.

[0034] The aforementioned anomaly detection model can be trained based on the Isolation Forest (iForest) algorithm, also known as the Island Forest algorithm. Isolation Forest is a fast outlier detection method based on model ensemble, featuring linear time complexity and high accuracy, making it suitable for big data processing. It is applicable to anomaly detection in continuous numerical data. Unlike other anomaly detection algorithms that characterize the degree of alienation between samples using quantitative metrics such as distance and density, the Isolation Forest algorithm detects outliers by isolating sample points. Specifically, this algorithm uses a binary search tree structure called an Isolation Tree (iTree) to isolate samples. Due to the smaller number of outliers and their alienation from most samples, outliers are isolated earlier; that is, outliers are closer to the root node of the iTree, while normal values ​​are farther away. In this embodiment, the anomaly detection model uses the isolation forest algorithm to quickly segment outliers based on the latency sampling data corresponding to multiple input streaming tasks (e.g., N first streaming tasks) and identifies the first streaming tasks corresponding to the outliers as having latency anomalies.

[0035] Optionally, embodiments of this application can implement the Isolation Forest algorithm based on the scikit-learn algorithm framework. Optionally, the parameters in this algorithm framework can be default parameters.

[0036] In this application embodiment, the anomaly detection model can be: Method 1, pre-trained; or Method 2, trained in real time during the above-mentioned latency anomaly detection process. This application does not limit this method. If the anomaly detection model is pre-trained, each latency anomaly detection unit (e.g., ...) in the latency anomaly processing component 220 Figure 2 In the components 221-1 to 221-n, pre-trained anomaly detection models for different task scenarios are deployed respectively. If the anomaly detection model is trained in real time during the above-mentioned latency anomaly detection process, each latency anomaly detection unit (such as...) in the latency anomaly processing component 220... Figure 2 In the initial stage, initial anomaly detection models are deployed for each of the 221-1 to 221-n units. Generally, the initial anomaly detection models deployed in each latency anomaly detection unit are the same. However, it is possible that different initial models are set for different task scenarios.

[0037] In the second method described above, the delay anomaly detection unit (such as...) Figure 2 In the initial stage, the initial anomaly detection model (221-1 to 221-n) can be trained first, and then the trained anomaly detection model can be used to detect latency anomalies in one or more streaming tasks after the initial stage is completed.

[0038] Continuing with the above method two, taking the latency anomaly detection unit 221-1 corresponding to the first task scenario as an example, the latency anomaly processing component 220 identifies the task scenario to which the streaming task belongs based on the task identifiers corresponding to the multiple streaming tasks. That is, there is a correspondence between the task identifier of the streaming task and the task scenario described by the streaming task. Multiple streaming tasks are clustered to obtain at least one streaming task set. Each streaming task set corresponds to a task scenario, and one streaming task set in at least one streaming task set may include the above N first streaming tasks. Generally speaking, the first streaming tasks in one streaming task set should meet the traffic threshold in order to provide sufficient training samples for the initial anomaly detection model.

[0039] Furthermore, the latency anomaly detection unit 221-1 can train the initial anomaly detection model based on the latency sampling data corresponding to these N first streaming tasks, thereby obtaining the trained anomaly detection model and the initial latency detection results for the N first streaming tasks. It should be noted that N satisfies the traffic threshold requirement in this case.

[0040] It should also be understood that, in the above method two, after the latency anomaly detection unit 221-1 trains the anomaly detection model, it can continue to perform latency anomaly detection on the received first streaming task. It can output a latency anomaly detection result once based on the latency sampling data of each input first streaming task, or output a latency anomaly detection result once based on the multiple latency sampling data corresponding to the multiple input first streaming tasks. The output latency anomaly detection result includes the detection results of whether there is latency anomaly in multiple first streaming tasks.

[0041] Taking the first task scenario as an example, in the first example, the latency anomaly processing component 220 inputs the N latency sampling data corresponding to the N first streaming tasks into the anomaly detection model of the latency anomaly detection unit 221-1 corresponding to the first task scenario to obtain the latency detection result; in the second example, the latency anomaly processing component 220 may also include a latency anomaly generation unit 222. The latency anomaly processing component 220 inputs the N latency sampling data corresponding to the N first streaming tasks into the anomaly detection model of the latency anomaly detection unit 221-1 corresponding to the first task scenario to obtain an initial latency detection result. This initial latency detection result is used to indicate that there are M2 first streaming tasks with latency anomalies among the N first streaming tasks, where M2≥M1 and M2 is an integer. The latency anomaly generation unit 222 verifies the initial latency detection result according to the task status corresponding to the M2 first streaming tasks to obtain the latency detection result.

[0042] In the second example above, to further improve the accuracy of latency anomaly detection, the latency anomaly processing component 220 filters the streaming tasks with latency anomalies identified by the anomaly detection model. As one implementation of the second example, for each of the M2 first streaming tasks, the latency anomaly generation unit 222 can determine that the first streaming task has a latency anomaly when the task state corresponding to that first streaming task is running; or, when the task state corresponding to that first streaming task is stopped, determine that the first streaming task does not have a latency anomaly. It should be understood that when the task state of a first streaming task is stopped, even if the latency sampling data of that first streaming task indicates a large latency level, it may have terminated task execution for various reasons, and this does not mean that the first streaming task has a latency anomaly.

[0043] In some embodiments, the latency detection system 200 further includes a detection database 230 and a metadata database 240. The detection database may store the initial latency detection results described above, for example, latency anomaly detection units (such as those in the first task scenario)... Figure 2(221-1) The obtained initial latency detection result can be sent to the detection database 230 for storage. The metadata database 240 can store the metadata of the above-mentioned multiple streaming tasks, for example, the metadata of the above-mentioned multiple streaming tasks can be stored in the form of database tables. Optionally, the metadata may include at least one of the task status, task identifier, and operator identifier of the corresponding streaming task, wherein the task status of any streaming task may include running or stopped.

[0044] In some embodiments, the task data acquisition component 210 can store the metadata corresponding to multiple streaming tasks in a metadata database.

[0045] For example, the latency anomaly generation unit 222 can use a dimension table association module, such as a dimension table join statement, to establish an association between the initial latency detection result in the detection database and the metadata of the M2 first streaming tasks in the metadata database, so as to determine the task status corresponding to each of the M2 first streaming tasks. In this case, the metadata includes at least the task status. Then, the latency anomaly generation unit 222 verifies the initial latency detection result according to the task status corresponding to each of the M2 first streaming tasks to obtain the latency detection result.

[0046] In some embodiments, the latency anomaly processing component 220 is further configured to determine the operator identifiers corresponding to the M1 first streaming tasks. For example, the latency anomaly generation unit 222 can establish an association between the initial latency detection results in the detection database and the operator identifiers of the M1 first streaming tasks in the metadata database through a dimension table association module, thereby determining the operator identifiers corresponding to the M1 first streaming tasks. The specific process by which the latency anomaly generation unit 222 determines the operator identifiers corresponding to the M1 first streaming tasks through the dimension table association module is the same as or similar to the process described above for determining the task states corresponding to the M2 first streaming tasks through the dimension table association module, and will not be elaborated further here.

[0047] In some embodiments, the latency detection system 200 may further include an alarm unit 250. The alarm unit 250 receives latency detection results from the latency anomaly processing component 220, which carry operator identifiers for M1 first streaming tasks. It should be noted that the M1 first streaming tasks may each correspond to a different operator identifier, or some or all of the M1 first streaming tasks may correspond to the same operator identifier. Furthermore, the alarm unit 250 can push anomaly indication information for the M1 first streaming tasks to the operator's client based on the operator identifiers indicated in the latency detection results, so that the operator can promptly maintain the corresponding unit modules in the streaming task platform 110.

[0048] Optionally, both the metadata database 240 and the detection database 230 mentioned above can be relational databases, such as MySQL or Oracle databases.

[0049] In this application embodiment, in addition to providing a latency detection system for streaming tasks, a latency detection method 300 for streaming tasks is also provided. The execution entity of the method 300 can be the aforementioned latency detection system 200 for streaming tasks, such as... Figure 3 As shown, the method 300 includes:

[0050] S310, acquire latency sampling data corresponding to multiple streaming tasks respectively. The latency sampling data is used to indicate the latency level of the corresponding streaming task. The multiple streaming tasks include streaming tasks under at least one task scenario. The at least one task scenario includes a first task scenario. The first task scenario includes N first streaming tasks.

[0051] S320, Determine the anomaly detection model corresponding to the first task scenario;

[0052] S330, based on the N latency sampling data corresponding to the N first streaming tasks and the anomaly detection model corresponding to the first task scenario, obtain the latency detection result, which is used to indicate M1 first streaming tasks with latency anomalies among the N first streaming tasks;

[0053] Where N>M1≥0.

[0054] In some embodiments, obtaining a latency detection result based on the N latency sampling data corresponding to the N first streaming tasks and the anomaly detection model corresponding to the first task scenario includes: inputting the N latency sampling data into the anomaly detection model corresponding to the first task scenario to obtain an initial latency detection result, which is used to indicate that there are M2 first streaming tasks with latency anomalies among the N first streaming tasks, where M2≥M1; and verifying the initial latency detection result according to the task status corresponding to the M2 first streaming tasks to obtain the latency detection result.

[0055] In some embodiments, the anomaly detection model is trained based on the isolated forest algorithm. The step of inputting the N time-delay sampling data into the anomaly detection model corresponding to the first task scenario to obtain the initial time-delay detection result includes: after establishing the anomaly detection model based on the isolated forest algorithm, inputting the N time-delay sampling data into the anomaly detection model to obtain the trained anomaly detection model and the initial time-delay detection result, where N is greater than a preset value.

[0056] In some embodiments, the initial latency detection result is verified based on the task states corresponding to the M2 first streaming tasks to obtain the latency detection result, including: for each of the M2 first streaming tasks, if the task state corresponding to the first streaming task is running, it is determined that the first streaming task has a latency anomaly; or, if the task state corresponding to the first streaming task is stopped, it is determined that the first streaming task does not have a latency anomaly.

[0057] In some embodiments, the initial delay detection result is stored in the detection database.

[0058] In some embodiments, verifying the initial latency detection result based on the task status corresponding to each of the M2 first streaming tasks to obtain the latency detection result includes: establishing an association between the initial latency detection result in the detection database and the metadata of the M2 first streaming tasks in the metadata database through a dimension table association module, so as to determine the task status corresponding to each of the M2 first streaming tasks, wherein the metadata includes the task status; and verifying the initial latency detection result based on the task status corresponding to each of the M2 first streaming tasks to obtain the latency detection result.

[0059] In some embodiments, obtaining a latency detection result based on N latency sampling data corresponding to the N first streaming tasks and an anomaly detection model corresponding to the first task scenario includes: clustering the multiple streaming tasks according to the task identifiers corresponding to the multiple streaming tasks to obtain at least one streaming task set; one streaming task set in the at least one streaming task set includes the N first streaming tasks, and the task identifiers corresponding to the multiple first streaming tasks all belong to the first task scenario; inputting the N latency sampling data corresponding to the N first streaming tasks in the at least one streaming task set into the anomaly detection model corresponding to the first task scenario to obtain the initial latency detection result and the trained anomaly detection model.

[0060] In some embodiments, the method further includes: determining the operator identifiers corresponding to the M1 first streaming tasks respectively.

[0061] In some embodiments, the method further includes: storing the initial latency detection result in the detection database; and establishing an association between the initial latency detection result in the detection database and the operator identifiers of the M1 first streaming tasks in the metadata database through a dimension table association module, so as to determine the operator identifiers corresponding to the M1 first streaming tasks respectively.

[0062] In some embodiments, the method further includes: pushing anomaly indication information of the M1 first streaming tasks to the operator's client based on the operator identifier.

[0063] In some embodiments, obtaining latency sampling data corresponding to multiple streaming tasks includes: obtaining task identifiers and latency sampling data corresponding to the multiple streaming tasks, and sending the task identifiers and latency sampling data corresponding to the multiple streaming tasks as multiple latency parameters corresponding to the multiple streaming tasks to the distributed messaging system, wherein the latency sampling data is related to the sampling time and the start time of the corresponding streaming task; and consuming the latency parameters of the first streaming task through a message consumer.

[0064] In some embodiments, the method further includes storing the metadata corresponding to the plurality of streaming tasks in the metadata database, wherein the metadata includes at least one of the task status, task identifier, and operator identifier of the corresponding streaming task.

[0065] The latency detection method for streaming data provided in this application acquires latency sampling data corresponding to multiple streaming tasks, and obtains latency detection results based on the N latency sampling data corresponding to the N first streaming tasks and the anomaly detection model corresponding to the first task scenario. The latency detection results are used to indicate M1 first streaming tasks with latency anomalies among the N first streaming tasks. This method can accurately detect latency anomalies of streaming tasks in the first task scenario, avoiding the problem of low detection accuracy caused by always using the same detection standard to detect latency anomalies of streaming tasks in different scenarios.

[0066] It should be noted that the execution subject of each step in the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 310 to 320 can be device A; or the execution subject of step 310 can be device A, and the execution subject of step 320 can be device B; and so on.

[0067] This application also provides an electronic device, see [link to relevant documentation] Figure 4 The embodiments in this application are only used as examples. Figure 4 This is used as an example for illustration and does not mean that this application is limited to this.

[0068] Figure 4 This is a schematic structural diagram of an electronic device 400 provided in an embodiment of this application. Figure 4 The electronic device 400 shown can be a terminal device or a server. When the electronic device 400 is deployed as a server in the cloud, it can be implemented as a cloud server. The electronic device 400 includes a processor 410, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0069] Optionally, such as Figure 4As shown, the electronic device 400 may further include a memory 430. The processor 410 can retrieve and run computer programs from the memory 430 to implement the methods described in the embodiments of this application.

[0070] The memory 430 can be a separate device independent of the processor 410, or it can be integrated into the processor 410.

[0071] Optionally, such as Figure 4 As shown, the electronic device 400 may also include a transceiver 420, which the processor 410 can control to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices.

[0072] The transceiver 420 may include a transmitter and a receiver. The transceiver 420 may further include an antenna, which may be one or more.

[0073] Optionally, the electronic device 400 can implement the corresponding process of the image generation system in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0074] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0075] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0076] It should be understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0077] This application also provides a computer-readable storage medium for storing computer programs.

[0078] Optionally, the computer-readable storage medium can be applied to the electronic device in the embodiments of this application, and the computer program causes the computer to execute the corresponding processes performed by the latency detection system of the streaming task in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0079] This application also provides a computer program product, including computer program instructions.

[0080] Optionally, the computer program product can be applied to the electronic device in the embodiments of this application, and the computer program instructions cause the computer to execute the corresponding process executed by the latency detection system of the streaming task in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0081] This application also provides a computer program.

[0082] Optionally, the computer program can be applied to the electronic device in the embodiments of this application. When the computer program is run on a computer, it causes the computer to execute the corresponding processes executed by the latency detection system of the streaming task in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0083] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0084] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0085] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A latency detection system for streaming tasks, characterized in that, include: A task data acquisition component is used to acquire latency sampling data corresponding to multiple streaming tasks. The latency sampling data is used to indicate the latency level of the corresponding streaming task. The multiple streaming tasks include streaming tasks under at least one task scenario. The at least one task scenario includes a first task scenario. The first task scenario includes N first streaming tasks. The N first streaming tasks include running streaming tasks and stopped streaming tasks. A latency anomaly handling component is provided, which is deployed with an anomaly detection model corresponding to each task scenario. The latency anomaly handling component is used to determine the anomaly detection model corresponding to the first task scenario, and obtain a latency detection result based on the N latency sampling data corresponding to the N first streaming tasks and the anomaly detection model corresponding to the first task scenario. The latency detection result is used to indicate that there are M1 first streaming tasks with latency anomalies among the N first streaming tasks. Where N>M1≥0; The delay anomaly processing component includes: a delay anomaly detection unit and a delay anomaly generation unit; The latency anomaly detection unit inputs the N latency sampling data into the anomaly detection model corresponding to the first task scenario to obtain an initial latency detection result. The initial latency detection result is used to indicate that there are M2 first streaming tasks with latency anomalies among the N first streaming tasks, where M2≥M1. The latency anomaly generation unit verifies the initial latency detection result based on the task status corresponding to the M2 first streaming tasks, and obtains the latency detection result. The delay anomaly generation unit is specifically used for: For each of the M2 first streaming tasks, if the task status corresponding to the first streaming task is running, it is determined that the first streaming task has a latency anomaly. When the task status corresponding to the first streaming task is stopped, it is determined that the first streaming task does not have any latency anomalies; The system further includes a detection database and a metadata database; the latency anomaly detection unit is also used to store the initial latency detection result in the detection database; the latency anomaly generation unit is specifically used for: The dimension table association module establishes an association between the initial latency detection results in the detection database and the metadata of the M2 first streaming tasks in the metadata database, so as to determine the task status corresponding to the M2 first streaming tasks respectively, and the metadata includes the task status. The initial latency detection result is verified based on the task status corresponding to the M2 first streaming tasks to obtain the latency detection result.

2. The system according to claim 1, characterized in that, The anomaly detection model is trained based on the isolated forest algorithm, and the latency anomaly detection unit is specifically used for: After establishing the anomaly detection model based on the isolated forest algorithm, the N time-delay sampling data are input into the anomaly detection model to obtain the trained anomaly detection model and the initial time-delay detection result, where N is greater than a preset value.

3. The system according to claim 1 or 2, characterized in that, The delay anomaly handling component is specifically used for: Based on the task identifiers corresponding to the multiple streaming tasks, the multiple streaming tasks are clustered to obtain at least one streaming task set; a streaming task set in the at least one streaming task set includes the N first streaming tasks, and the task identifiers corresponding to the multiple first streaming tasks all belong to the first task scenario. The N latency sampling data corresponding to the N first streaming tasks in the at least one streaming task set are input into the anomaly detection model corresponding to the first task scenario to obtain the initial latency detection result and the trained anomaly detection model.

4. The system according to claim 1 or 2, characterized in that, The delay anomaly handling component is also used for: Determine the operator identifiers corresponding to the M1 first streaming tasks.

5. The system according to claim 4, characterized in that, The system also includes a detection database and a metadata database; The delay anomaly detection unit is also used to store the initial delay detection result in the detection database; The delay anomaly generation unit is specifically used for: By using the dimension table association module, an association relationship is established between the initial latency detection results in the detection database and the operator identifiers of the M1 first streaming tasks in the metadata database, so as to determine the operator identifiers corresponding to the M1 first streaming tasks respectively.

6. The system according to claim 4, characterized in that, The system also includes: an alarm unit; The alarm unit receives a latency detection result from the latency anomaly processing component, and the latency detection result carries the operator identifier. Based on the operator identifier indicated in the latency detection result, anomaly indication information for the M1 first streaming tasks is pushed to the operator's client.

7. The system according to claim 1 or 2, characterized in that, The task data acquisition component includes: a task data acquisition unit and a distributed message unit; The task data acquisition unit obtains the task identifiers and latency sampling data corresponding to the multiple streaming tasks respectively, and sends the task identifiers and latency sampling data corresponding to the multiple streaming tasks as multiple latency parameters corresponding to the multiple streaming tasks to the distributed message unit. The latency sampling data is related to the sampling time and the start time of the corresponding streaming task. The distributed message unit sends the latency parameters of the first streaming task to the latency exception handling component through the message consumer.

8. The system according to claim 7, characterized in that, The system also includes: a time-series database; The time-series database receives multiple latency parameters corresponding to the multiple streaming tasks from the task data acquisition unit, and sends the multiple latency parameters to the distributed message unit.

9. The system according to claim 1 or 2, characterized in that, The system also includes a metadata database; The task data acquisition component stores the metadata corresponding to the multiple streaming tasks in the metadata database. The metadata includes at least one of the task status, task identifier, and operator identifier of the corresponding streaming task.

10. A delay detection method for streaming tasks, characterized in that, include: Acquire latency sampling data corresponding to multiple streaming tasks respectively. The latency sampling data is used to indicate the latency level of the corresponding streaming task. The multiple streaming tasks include streaming tasks under at least one task scenario. The at least one task scenario includes a first task scenario. The first task scenario includes N first streaming tasks. The N first streaming tasks all belong to the first task scenario. The N first streaming tasks include running streaming tasks and stopped streaming tasks. Determine the anomaly detection model corresponding to the first task scenario; Based on the N latency sampling data corresponding to the N first streaming tasks and the anomaly detection model corresponding to the first task scenario, a latency detection result is obtained. The latency detection result is used to indicate that there are M1 first streaming tasks with latency anomalies among the N first streaming tasks. Where N>M1≥0; The step of obtaining latency detection results based on the N latency sampling data corresponding to the N first streaming tasks and the anomaly detection model corresponding to the first task scenario includes: The N latency sampling data are input into the anomaly detection model corresponding to the first task scenario to obtain the initial latency detection result. The initial latency detection result is used to indicate that there are M2 first streaming tasks with latency anomalies among the N first streaming tasks, where M2≥M1. Based on the task status corresponding to the M2 first streaming tasks, the initial latency detection result is verified to obtain the latency detection result. The step of verifying the initial latency detection result based on the task states corresponding to the M2 first streaming tasks to obtain the latency detection result includes: For each of the M2 first streaming tasks, if the task status corresponding to the first streaming task is running, it is determined that the first streaming task has a latency anomaly. When the task status corresponding to the first streaming task is stopped, it is determined that the first streaming task does not have any latency anomalies; The method further includes a detection database and a metadata database; storing the initial latency detection result in the detection database; and verifying the initial latency detection result according to the task status corresponding to the M2 first streaming tasks to obtain the latency detection result, including: The dimension table association module establishes an association between the initial latency detection results in the detection database and the metadata of the M2 first streaming tasks in the metadata database, so as to determine the task status corresponding to the M2 first streaming tasks respectively, and the metadata includes the task status. The initial latency detection result is verified based on the task status corresponding to the M2 first streaming tasks to obtain the latency detection result.

Images