Data detection method and device, electronic equipment and storage medium

Abstract

The invention relates to a data detection method and device, electronic equipment and a storage medium, and the method comprises the steps: dividing a current detection period into a first processing window and a second processing window when the current detection period is reached; detecting service data based on a first data threshold value corresponding to the current detection period in the current detection period; determining a training data set corresponding to the next detection period of the current detection period in the first processing window based on the acquired business data and the historical business data set, and training a target prediction model based on the training data set in the second processing window, the trained target prediction model outputs a second data threshold value corresponding to the next detection period, and data detection in the next detection period is completed based on the second data threshold value. According to the method, a prediction empty window period caused by model training and threshold calculation in a traditional real-time detection mode is eliminated, system resource contention in a high-concurrency scene is avoided, and the real-time performance of data detection is improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data detection method, apparatus, electronic device and storage medium. Background Technology

[0002] In data quality monitoring scenarios, real-time anomaly detection is required for periodically arriving business data (e.g., hourly user activity counts) to ensure system stability and business health. Currently, a detection method relying on real-time computation is commonly used. This involves training a predictive model using historical business data at the start of each detection period, calculating the corresponding normal data threshold for that period, and then using this threshold to determine whether newly arriving business data is abnormal. However, because model training and threshold calculation are performed within the current detection period, this method not only results in the inability to output detection results promptly during the window period following the start of the current period, but also easily leads to resource contention in high-concurrency scenarios, causing system response delays and failing to meet real-time requirements. Summary of the Invention

[0003] In view of this, in order to solve the above-mentioned technical problems or some of the technical problems, the embodiments of this application provide a data detection method, apparatus, electronic device and storage medium.

[0004] Firstly, this application provides a data detection method, including: When the current detection cycle is reached, the current detection cycle is divided into a first processing window and a second processing window; Within the first processing window and the second processing window, the acquired business data is detected based on the first data threshold corresponding to the current detection period; Within the first processing window, based on the acquired business data and the historical business dataset acquired before the current detection period, the training dataset corresponding to the first detection period is determined, and within the second processing window, a target prediction model is trained based on the training dataset, so that the trained target prediction model outputs a second data threshold corresponding to the first detection period, and the first detection period is the next detection period of the current detection period; Upon reaching the first detection period, the detection of the business data acquired within the first detection period is completed based on the second data threshold.

[0005] In an optional implementation, the step of detecting the acquired business data based on a first data threshold corresponding to the current detection period includes: The first channel detects the acquired business data based on the first data threshold corresponding to the current detection period; The step of determining the training dataset corresponding to the first detection period based on the acquired business data and the historical business dataset acquired before the current detection period includes: The training dataset corresponding to the first detection period is determined by the second channel based on the acquired business data and the historical business dataset acquired before the current detection period; The step of training a target prediction model based on the training dataset, so that the trained target prediction model outputs a second data threshold corresponding to the first detection period, includes: The target prediction model is trained using the second channel based on the training dataset, so that the trained target prediction model outputs the second data threshold corresponding to the first detection period.

[0006] In an optional implementation, within the first processing window, the proportion of system resources allocated to the first channel is greater than the proportion of system resources allocated to the second channel. Within the second processing window, the proportion of system resources allocated to the first channel is less than the proportion of system resources allocated to the second channel.

[0007] In an optional implementation, determining the training dataset corresponding to the first detection period through the second channel based on the acquired business data and the historical business dataset acquired before the current detection period includes: The acquired business data is encapsulated into a first message through the first channel, and the first message is sent to the message queue. The first message is obtained from the message queue through the second channel; The training dataset corresponding to the first detection period is determined by the second channel based on the business data in the first message and the historical business dataset obtained before the current detection period.

[0008] In an optional implementation, training the target prediction model based on the training dataset through the second channel, so that the trained target prediction model outputs a second data threshold corresponding to the first detection period, includes: The training dataset is encapsulated into a second message through the second channel, and the second message is sent to the message queue. The second message is obtained from the message queue through the second channel; The target prediction model is trained using the second channel based on the training dataset in the second message, so that the trained target prediction model outputs the second data threshold corresponding to the first detection period.

[0009] In an optional implementation, the method further includes: After the trained target prediction model outputs the second data threshold corresponding to the first detection period, the correspondence between the first detection period and the second data threshold is stored in the target database. The step of detecting the business data acquired within the first detection period based on the second data threshold upon reaching the first detection period includes: Upon reaching the first detection cycle, the second data threshold corresponding to the first detection cycle is obtained from the target database based on the first detection cycle. The detection of business data acquired during the first detection period is completed based on the second data threshold.

[0010] In an optional implementation, training a target prediction model based on the training dataset, such that the trained target prediction model outputs a second data threshold corresponding to the first detection period, includes: The historical prediction model corresponding to the second detection period is obtained from the model storage pool. The model storage pool stores the historical prediction models corresponding to different detection periods. The second detection period is the previous detection period of the current detection period. The historical prediction model corresponding to the second detection period is a model trained in the second detection period and used to output the first data threshold. The historical prediction model corresponding to the second detection period is determined as the target prediction model; The target prediction model is incrementally trained based on the training dataset so that the trained target prediction model outputs the second data threshold corresponding to the first detection period.

[0011] Secondly, this application provides a data detection method, including: The partitioning module is used to divide the current detection cycle into a first processing window and a second processing window when the current detection cycle is reached. The detection module is used to detect the acquired business data within the first processing window and the second processing window, based on a first data threshold corresponding to the current detection period. The prediction module is used to determine the training dataset corresponding to the first detection period based on the acquired business data and the historical business dataset acquired before the current detection period within the first processing window, and to train the target prediction model based on the training dataset within the second processing window, so that the trained target prediction model outputs the second data threshold corresponding to the first detection period, wherein the first detection period is the next detection period of the current detection period. The detection module is further configured to, upon reaching the first detection period, perform detection of the business data acquired within the first detection period based on the second data threshold.

[0012] Thirdly, this application provides an electronic device, including a processor and a memory, wherein the processor is configured to execute a data detection program stored in the memory to implement the data detection method described above.

[0013] Fourthly, this application also provides a storage medium storing one or more programs that can be executed by one or more processors to implement the data detection method described above.

[0014] Compared with the prior art, the technical solution provided in this application has the following advantages. The method provided in this application includes: when the current detection period is reached, dividing the current detection period into a first processing window and a second processing window; within the first processing window and the second processing window, detecting the acquired business data based on a first data threshold corresponding to the current detection period; within the first processing window, determining the training dataset corresponding to the first detection period based on the acquired business data and the historical business dataset acquired before the current detection period; and within the second processing window, training a target prediction model based on the training dataset, so that the trained target prediction model outputs a second data threshold corresponding to the first detection period, the first detection period being the next detection period of the current detection period; and upon reaching the first detection period, completing the detection of the business data acquired within the first detection period based on the second data threshold. By dividing the current detection cycle into a first processing window and a second processing window, and performing real-time detection based on a first data threshold pre-stored in the previous detection cycle within the first processing window, while preparing a training dataset for the next detection cycle, the second processing window, in addition to performing real-time detection based on the first data threshold pre-stored in the previous detection cycle, also asynchronously completes the training of the prediction model and the calculation of the data threshold for the next cycle. This achieves temporal decoupling between detection and prediction, allowing the data threshold for the next detection cycle to be pre-calculated within the current detection cycle. Thus, when the next detection cycle arrives, detection can be completed immediately based on the calculated data threshold, eliminating the prediction window caused by model training and threshold calculation in traditional real-time detection methods, avoiding system resource contention in high-concurrency scenarios, and improving the real-time performance of data detection. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0018] Figure 1 A flowchart illustrating a data detection method provided in an embodiment of this application; Figure 2 A flowchart illustrating another data detection method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a data detection device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0021] refer to Figure 1 , Figure 1 This is a flowchart illustrating a data detection method provided in an embodiment of this application. The data detection method provided in this embodiment includes the following steps: S101: When the current detection cycle is reached, the current detection cycle is divided into a first processing window and a second processing window.

[0022] In this embodiment, the current detection period is a periodic time unit in which data detection is being performed. The detection period is a data detection time segment divided by fixed time intervals, such as hourly or daily. Its function is to establish a regular detection rhythm, which is the basis for data time-series analysis. The first processing window and the second processing window are obtained by dividing the current detection period into time segments. The first processing window is mainly used to handle tasks with the highest real-time requirements, while the second processing window is mainly used for computationally intensive processing. The time segment corresponding to the first processing window is earlier than the time segment corresponding to the second processing window, and the total duration of the first processing window is greater than the total duration of the second processing window to prioritize real-time detection resources. By dividing the first and second processing windows, task decoupling and resource peak shifting in the time dimension are achieved.

[0023] When the current detection cycle starts, it can be divided into two consecutive processing windows, namely the first processing window and the second processing window, according to the preset window ratio rules based on business needs. For example, if the time period corresponding to the current detection cycle is 14:00~15:00, and the preset window ratio rule is that the proportion of the first processing window is 60% and the proportion of the second processing window is 40%, then the time period corresponding to the first processing window is 14:00~14:36, and the time period corresponding to the second processing window is 14:36~15:00.

[0024] S102: Within the first processing window and the second processing window, the acquired business data is detected based on the first data threshold corresponding to the current detection cycle.

[0025] In this embodiment, the first data threshold is actually a pre-calculated and stored normal data value range within the second detection period, used for anomaly detection of business data acquired within the current detection period. If the business data exceeds the normal data value range, it is determined to be abnormal; conversely, if the business data is within the normal data value range, it is determined to be normal. The second detection period is the previous detection period. For example, if the current detection period is 14:00~15:00, then the second detection period is 13:00~14:00.

[0026] To ensure data detection, the acquired business data is detected both within the first and second processing windows (i.e., throughout the entire current detection period). During the detection of acquired business data, if data is obtained, a query is performed on the target database storing data thresholds based on the current detection period. The first data threshold corresponding to the current detection period is then retrieved from the target database. This first data threshold is then used to perform real-time detection of the business data acquired within the current detection period within both the first and second processing windows.

[0027] S103: Within the first processing window, based on the business data obtained within the first processing window and the historical business dataset obtained before the current detection period, determine the training dataset corresponding to the first detection period, and within the second processing window, train the target prediction model based on the training dataset so that the trained target prediction model outputs the second data threshold corresponding to the first detection period.

[0028] In this embodiment, the first detection period is the next detection period after the current detection period. For example, if the current detection period is 14:00~15:00, then the first detection period is 15:00~16:00. The second data threshold is actually the normal range of data values ​​pre-calculated and stored within the current detection period, used for anomaly detection of the business data acquired within the first detection period. Within the first processing window, if business data is acquired, it can be temporarily stored in the data pool. Simultaneously, the historical business dataset acquired before the current detection period is called and merged with the business data temporarily stored in the data pool within the current detection period to obtain the training dataset. At the end of the first processing window, the obtained training dataset is determined as the final training dataset corresponding to the first detection period.

[0029] After obtaining the training dataset corresponding to the first detection period, the historical prediction model trained and stored in the first detection period is loaded as the target prediction model. Within the second processing window, the target prediction model is trained based on the training dataset so that the trained target prediction model outputs the second data threshold corresponding to the first detection period.

[0030] After obtaining the second data threshold corresponding to the first detection period, the second data threshold can be stored so that it can be retrieved when the first detection period is reached, thereby completing the detection of the business data obtained in the first detection period based on the second data threshold.

[0031] In the above process, at the end of the first processing window, the total number of data in the training dataset is counted. If the total number exceeds a preset threshold, indicating sufficient training data, a second processing window is initiated. Within the second processing window, the acquired business data is detected based on the first data threshold corresponding to the current detection period, and a target prediction model is trained based on the training dataset. This trained target prediction model outputs the second data threshold corresponding to the first detection period. If the total number does not exceed the preset threshold, indicating insufficient training data, the first processing window is extended and the second processing window is shortened to update the first and second processing windows. Based on the updated first and second processing windows, the process returns to the first step of counting the total number of data in the training dataset at the end of the first processing window.

[0032] It should be noted that at the end of the first processing window, if the duration of the updated second processing window is less than a preset duration threshold and the total number still does not exceed a preset number threshold, then the updating of the first and second processing windows stops, and the second processing window is started to complete the data detection and model training tasks within the second processing window. Through this method, this embodiment not only solves the problem of low pre-calculation accuracy caused by insufficient data accumulation, but also ensures that the pre-calculation of the data threshold corresponding to the next detection cycle is completed within the current detection cycle, completely avoiding the window period.

[0033] S104: Upon reaching the first detection cycle, the detection of the business data acquired within the first detection cycle is completed based on the second data threshold.

[0034] In this embodiment, when the first detection cycle is started, the target database storing data thresholds is queried based on the first detection cycle to obtain the second data threshold corresponding to the first detection cycle from the target database. The first detection cycle is then used as the current detection cycle and the second data threshold is used as the first data threshold to execute the above-mentioned S101 step. Based on the divided processing window, the acquired business data is detected in real time and the data threshold of the next detection cycle is calculated to achieve dynamic relay prediction.

[0035] This embodiment provides a data detection method that divides the current detection period into a first processing window and a second processing window. Within the first processing window, real-time detection is performed based on a first data threshold pre-stored in the previous detection period, while simultaneously preparing a training dataset for the next detection period. Then, within the second processing window, in addition to performing real-time detection based on the first data threshold pre-stored in the previous detection period, the training of the prediction model and the calculation of the data threshold for the next period are asynchronously completed. This achieves temporal decoupling between detection and prediction, allowing the data threshold for the next detection period to be pre-calculated within the current detection period. Thus, when the next detection period arrives, detection can be completed immediately based on the calculated data threshold. This eliminates the prediction window caused by model training and threshold calculation in traditional real-time detection methods, avoids system resource contention in high-concurrency scenarios, and improves the real-time performance of data detection.

[0036] refer to Figure 2 , Figure 2 This is a flowchart illustrating another data detection method provided in an embodiment of this application. The data detection method provided in this embodiment includes the following steps: S201: When the current detection cycle is reached, the current detection cycle is divided into a first processing window and a second processing window.

[0037] In this embodiment, step S201 is the same as step S101 described above. For details, please refer to step S101 described above. This embodiment will not repeat the details here.

[0038] S202: Within the first processing window and the second processing window, the acquired business data is detected through the first channel based on the first data threshold corresponding to the current detection cycle.

[0039] S203: Within the first processing window, the training dataset corresponding to the first detection period is determined through the second channel based on the acquired business data and the historical business dataset acquired before the current detection period.

[0040] S204: Within the second processing window, a target prediction model is trained based on the training dataset through the second channel, so that the trained target prediction model outputs the second data threshold corresponding to the first detection period.

[0041] Regarding steps S202 to S204 above, the first channel is responsible for executing the real-time detection task (real-time detection of the acquired business data). Its role is to ensure that periodically arriving business data can be immediately identified as abnormal, which is crucial for ensuring real-time response capabilities. The second channel is responsible for executing the pre-computation task (including training dataset preparation, model training, and data threshold determination). Its role is to decouple it from the real-time detection task, so as to complete the computationally intensive work asynchronously and non-blockingly, thus preparing in advance for future detection cycles.

[0042] Within the first processing window, newly arriving business data for the current detection period is allocated to the first channel. This first channel performs millimeter-level detection on each arriving data piece based on a first data threshold corresponding to the current detection period, outputting anomaly detection results to ensure that the acquired business data is processed in real time. Simultaneously, the task of determining the training dataset corresponding to the first detection period is submitted to the second channel. This allows the second channel to acquire the business data obtained within the first processing window and retrieve historical business datasets acquired before the current detection period from historical storage. The second channel merges these two data sets to package and generate a training dataset, preparing for subsequent model training. This process is asynchronous and does not block the real-time response of the first channel.

[0043] Within the second processing window, the two channels continue to collaborate, but the focus shifts. In the second processing window, the first channel continues to perform anomaly detection on the business data acquired within the second processing window, based on the first data threshold corresponding to the current detection period, to ensure the continuity of data detection throughout the current detection period. Simultaneously, the second channel first loads the target prediction model trained in the previous detection period (i.e., the second detection period), and then trains this model using the training dataset corresponding to the second detection period prepared in the first processing window. This allows the trained target prediction model to output the second data threshold for the next detection period (i.e., the first detection period), preparing for data detection within the first detection period.

[0044] Through the above methods, this embodiment introduces logically independent first and second channels, further achieving spatial decoupling of the task flow on the basis of temporal decoupling. It binds real-time detection tasks and pre-computation tasks to different processing channels, allowing them to execute in parallel without interference. This ensures that the response time of real-time detection does not increase due to model training, while the pre-computation task receives dedicated resource guarantees and is not overwhelmed by sudden surges in real-time traffic, thus stably meeting both real-time requirements and complex prediction computation needs simultaneously.

[0045] In this embodiment, within the first processing window, the proportion of system resources allocated to the first channel is greater than the proportion of system resources allocated to the second channel; within the second processing window, the proportion of system resources allocated to the first channel is less than the proportion of system resources allocated to the second channel.

[0046] System resources refer to the available computing resources in the server cluster, mainly including CPU computing cores, memory capacity, network bandwidth and I / O throughput, which are the foundation for the system to carry all processing tasks.

[0047] Upon entering the first processing window of the current detection cycle, the real-time detection task executed by the first channel is elevated to a high-priority task, and most available system resources are allocated to the first channel to ensure that the real-time detection task has sufficient computing power to process bursts of data with millimeter-level latency, thus guaranteeing real-time response in data detection. Meanwhile, the second channel is allocated only a small amount of system resources necessary for basic operation, sufficient to execute lightweight tasks on a defined training dataset, but without initiating any potentially resource-intensive model training computations, thereby avoiding resource contention and interference on the critical path of real-time detection.

[0048] When switching from the first processing window to the second processing window, resources are dynamically rebalanced. At this time, the pre-computation tasks performed by the second channel are elevated to high priority, ensuring that the second channel receives most of the available system resources and can execute computationally intensive tasks such as model training and threshold calculation at full speed. The proportion of system resources allocated to the first channel is correspondingly reduced. Since its main task in the second processing window is processing the long tail of the data stream, this proportion of system resources is sufficient to maintain normal detection throughput and will not affect overall real-time performance.

[0049] It should be noted that the system resource allocation ratio in the first processing window and the second processing window can be set according to actual needs, and this embodiment does not impose any specific limitations on it.

[0050] By implementing a dynamic and variable resource allocation strategy for the dual channels within different processing windows, this embodiment solves the problem of drastic, sawtooth-like fluctuations in resource utilization during traditional periodic data detection (i.e., all tasks simultaneously compete for resources at the beginning of the detection period, leading to a computational storm, followed by resource idleness). This ensures that resource supply closely matches the temporal changes in task requirements, guaranteeing extremely low latency for detection within the first processing window and timely completion of model training within the second processing window, thus eliminating the risk of mutual blocking between tasks from a resource perspective.

[0051] In this embodiment, step S203 above determines the training dataset corresponding to the first detection period through the second channel based on the acquired business data and the historical business dataset acquired before the current detection period, specifically including: The acquired business data is encapsulated into a first message through the first channel, and the first message is sent to the message queue. The first message is retrieved from the message queue via the second channel; The training dataset corresponding to the second detection period is determined by using the second channel based on the business data in the first message and the historical business dataset obtained before the current detection period.

[0052] The first message, generated by the first channel, is a data packet containing business data acquired within the first processing window. It serves as the data carrier connecting the real-time detection task and the pre-computation task. The message queue is a middleware system following a producer-consumer model (e.g., Kafka). It acts as an asynchronous communication bus and buffer between the first and second channels, decoupling the two: the producer (first channel) can return after sending a message without waiting for the consumer to process it, and the consumer (second channel) can consume messages at its own pace; the two are completely independent in terms of processing capabilities.

[0053] Specifically, within the first processing window, the first channel, while performing real-time data detection, encapsulates each piece of business data it acquires. For example, it calls a message serialization component to package the business data content, the precise time point corresponding to the business data, the data source identifier, and other metadata into a structured data packet to form the first message. Subsequently, the first channel sends this first message to the first topic in the message queue. At this point, the first channel's task is complete, and it can immediately continue monitoring subsequently acquired business data without waiting for any response from the second channel.

[0054] Once the first message enters the message queue, the queue sets its status to "pending consumption," creating a buffer pool ordered chronologically. During this time, the message will not be lost regardless of whether the second channel is busy or restarts. The message queue is responsible for maintaining the message delivery state, ensuring that messages are eventually delivered.

[0055] The first consumer group configured in the second channel continuously listens for or retrieves new messages from the first topic in the consumption queue. Upon receiving the first message, the first consumer group reconstructs the original business data and metadata of the first message. Then, based on the timestamp and source of the business data, it queries the relevant historical business dataset from a historical storage system (such as a data warehouse or database). Finally, the business data obtained within the first processing window is fused with the retrieved historical business dataset to obtain the training dataset for training the model predicting the first detection period.

[0056] Through the above methods, this embodiment introduces a message queue to transform the training data generated in the first channel from synchronous transmission to asynchronous publish / subscribe, achieving complete decoupling of real-time data detection and pre-computation tasks. The responsibility of the first channel is simplified to detection and publishing, completely eliminating the blocking risk of waiting for the processing results of the second channel and ensuring extremely low latency and high throughput for real-time detection. At the same time, the second channel gains complete autonomous scheduling rights, and can consume business data from the message queue on demand or in batches for training preparation according to its own resource conditions and processing capabilities. This not only smoothly copes with data surges, but also facilitates load balancing and horizontal scaling, enhancing the reliability and robustness of the entire data pre-preparation process.

[0057] In this embodiment, step S204 above involves training a target prediction model based on a training dataset using a second channel, so that the trained target prediction model outputs a second data threshold corresponding to the first detection period. Specifically, this includes: The training dataset is encapsulated into a second message through the second channel, and the second message is sent to the message queue. The second message is retrieved from the message queue via the second channel; The target prediction model is trained using the training dataset in the second message through the second channel, so that the trained target prediction model outputs the second data threshold corresponding to the first detection period.

[0058] The second message, generated after the training dataset is prepared by the second channel, is a command package that triggers the model training task and contains the training dataset. It is a task instruction carrier that connects the two sub-stages of data preparation completion and model training start.

[0059] Specifically, after the second channel generates the training dataset based on the first message and the historical business dataset, it does not immediately invoke the model training algorithm. Instead, it encapsulates this training dataset to create a second message. This second message contains all the information needed to initiate a model training session. Subsequently, the second channel sends this second message to the second topic in the message queue.

[0060] Once the second message enters the message queue, it is persisted like the first message. This message queue acts as a list of tasks to be processed. The second consumer group configured for the second channel can retrieve the second message from the second topic of the message queue.

[0061] The model training consumer deployed in the second channel retrieves the second message from the message queue, parses the message content, loads the training dataset and the target prediction model, and performs a computationally intensive model training process based on the training dataset and the target prediction model. After training is complete, this computation instance drives the trained target prediction model to run and calculate the second data threshold corresponding to the first detection period.

[0062] Through the above methods, this embodiment further introduces a message queue-based task triggering mechanism within the second channel, achieving asynchronous management of computationally intensive model training tasks. It also decouples the training data preparation and model training sub-steps, allowing tasks to be submitted and resources released immediately after training data preparation. The model training task then enters a queue and is asynchronously consumed and processed by a dedicated computing resource pool. This achieves load balancing and elastic scheduling of pre-computation tasks, improving throughput efficiency and overall reliability for handling complex tasks.

[0063] In this embodiment, step S204 involves training a target prediction model based on a training dataset, so that the trained target prediction model outputs a second data threshold corresponding to the first detection period. Specifically, this includes: The historical prediction model corresponding to the second detection period is obtained from the model storage pool, which stores historical prediction models corresponding to different detection periods. The historical prediction model corresponding to the second detection cycle is determined as the target prediction model; The target prediction model is incrementally trained based on the training dataset so that the trained target prediction model outputs the second data threshold corresponding to the first detection period.

[0064] The model storage pool can be understood as a database that centrally stores and manages historical prediction models corresponding to the detection period. The second detection period is the previous detection period of the current detection period. The historical prediction model corresponding to the second detection period is the model trained in the second detection period and used to output the first data threshold. In fact, the historical prediction model corresponding to the second detection period is the model trained in the second detection period and successfully used to output the first data threshold used in the current detection period.

[0065] When model training needs to be started in the second processing window, a query request is first sent to the model storage pool. Based on the identifier of the second detection period carried in the query request, the model storage pool retrieves the historical prediction model corresponding to the second detection period. This retrieved historical prediction model is used as the target prediction model for the current training process. Using the training dataset prepared in the first processing window, the target prediction model is incrementally trained. The training process focuses on fine-tuning the model parameters with new data, allowing it to absorb the latest data features while maintaining its understanding of the original cyclical patterns. After training, the trained target prediction model has evolved into a new version adapted to the latest data distribution. Subsequently, the trained model is driven to perform predictions, outputting a second data threshold specifically for the first detection period.

[0066] Through the above methods, this embodiment introduces an incremental training mechanism based on a model storage pool, transforming model training for each detection cycle from independent reconstruction to iterative evolution. It utilizes a proven predictive model from the previous detection cycle as the starting point for optimization in the current training round, rapidly absorbing features from new data through incremental training, thereby significantly reducing the time and computational resources required for predictive model convergence. This method improves the efficiency of the pre-computation process, ensures that complex model updates can be completed within a limited second processing window, and maintains the consistency and stability of predictive knowledge through continuous iteration of the predictive model. It avoids performance fluctuations that may result from complete retraining of the predictive model, achieving robust and efficient continuous evolution of predictive capabilities.

[0067] The data detection method provided in this embodiment, after the trained target prediction model outputs the second data threshold corresponding to the first detection period, further includes the following steps: The correspondence between the first detection cycle and the second data threshold is stored in the target database; Upon reaching the first detection period, the business data acquired within the first detection period is detected based on the second data threshold, including: Upon reaching the first detection cycle, based on the first detection cycle, the second data threshold corresponding to the first detection cycle is obtained from the target database; The detection of business data acquired in the first detection cycle is completed based on the second data threshold.

[0068] The target database can be considered as a structured data management system, such as a relational database MySQL, for persistent storage. It stores the mapping relationship between each detection cycle and its pre-calculated data thresholds for a long time, ensuring that the data is not lost and can be queried efficiently and consistently.

[0069] Specifically, after the second channel completes model training and outputs a second data threshold for data detection in the first detection period, a structured record is generated. This record typically contains at least two core fields: the detection period timestamp (e.g., 2024-01-01 11:00:00) and the threshold data. This record is then inserted into a specific table in the target database. After the first detection period begins, the second data threshold corresponding to the first detection period is retrieved from the target database, and thus, data detection is performed using the second data threshold.

[0070] S205: Upon reaching the first detection cycle, based on the first detection cycle, query the target database to obtain the second data threshold corresponding to the first detection cycle.

[0071] S206: Detect the business data acquired in the first detection period based on the second data threshold.

[0072] Regarding steps S205 and S206 above, when the first detection cycle starts, the real-time data detection process is triggered. At this time, the first channel no longer performs any calculations, but instead sends a query request to the target database. The query request includes the timestamp of the first detection cycle. Based on the timestamp of the first detection cycle, the target database quickly locates and returns the second data threshold corresponding to the first detection cycle through an index. After receiving the second data threshold, the first channel immediately applies it to perform real-time detection of the business data acquired within the first detection cycle, and the entire process is completed within milliseconds. In this way, this embodiment eliminates the need for any calculations at the beginning of the detection cycle by persistently storing the pre-calculated data threshold corresponding to the future detection cycle in the database and directly querying and using it when the detection cycle arrives, thereby ensuring zero latency in the detection response.

[0073] This embodiment provides a data detection method that divides the current detection period into a first processing window and a second processing window. Within the first processing window, real-time detection is performed based on a first data threshold pre-stored in the previous detection period, while simultaneously preparing a training dataset for the next detection period. Then, within the second processing window, in addition to performing real-time detection based on the first data threshold pre-stored in the previous detection period, the training of the prediction model and the calculation of the data threshold for the next period are asynchronously completed. This achieves temporal decoupling between detection and prediction, allowing the data threshold for the next detection period to be pre-calculated within the current detection period. Thus, when the next detection period arrives, detection can be completed immediately based on the calculated data threshold. This eliminates the prediction window caused by model training and threshold calculation in traditional real-time detection methods, avoids system resource contention in high-concurrency scenarios, and improves the real-time performance of data detection.

[0074] refer to Figure 3 , Figure 3 This is a schematic diagram of a data detection device provided in an embodiment of this application. The data detection device provided in this embodiment includes: a partitioning module 10, a detection module 20, and a prediction module 30. The partitioning module 10 is used to divide the current detection period into a first processing window and a second processing window when the current detection period is reached. The detection module 20 is used to detect the acquired business data within the first and second processing windows based on a first data threshold corresponding to the current detection period. The prediction module 30 is used to determine a training dataset corresponding to the first detection period within the first processing window based on the acquired business data and historical business datasets acquired before the current detection period, and to train a target prediction model based on the training dataset within the second processing window, so that the trained target prediction model outputs a second data threshold corresponding to the first detection period, where the first detection period is the next detection period of the current detection period. The detection module 20 is also used to complete the detection of the business data acquired within the first detection period based on the second data threshold when the first detection period is reached.

[0075] In this embodiment, the detection module 20 is further configured to: The acquired business data is detected through the first channel based on the first data threshold corresponding to the current detection period.

[0076] In this embodiment, the prediction module 30 is further configured to: The training dataset corresponding to the first detection period is determined by the second channel based on the acquired business data and the historical business dataset acquired before the current detection period; The target prediction model is trained using the second channel based on the training dataset, so that the trained target prediction model outputs the second data threshold corresponding to the first detection period.

[0077] In this embodiment, within the first processing window, the proportion of system resources allocated to the first channel is greater than the proportion of system resources allocated to the second channel; Within the second processing window, the proportion of system resources allocated to the first channel is less than the proportion of system resources allocated to the second channel.

[0078] In this embodiment, the prediction module 30 is further configured to: The acquired business data is encapsulated into a first message through the first channel, and the first message is sent to the message queue. The first message is obtained from the message queue through the second channel; The training dataset corresponding to the first detection period is determined by the second channel based on the business data in the first message and the historical business dataset obtained before the current detection period.

[0079] In this embodiment, the prediction module 30 is further configured to: The training dataset is encapsulated into a second message through the second channel, and the second message is sent to the message queue. The second message is obtained from the message queue through the second channel; The target prediction model is trained using the second channel based on the training dataset in the second message, so that the trained target prediction model outputs the second data threshold corresponding to the first detection period.

[0080] The data detection device provided in this embodiment further includes a storage module, which is used for: After the trained target prediction model outputs the second data threshold corresponding to the first detection period, the correspondence between the first detection period and the second data threshold is stored in the target database.

[0081] In this embodiment, the detection module 20 is further configured to: Upon reaching the first detection cycle, the second data threshold corresponding to the first detection cycle is obtained from the target database based on the first detection cycle. The detection of business data acquired during the first detection period is completed based on the second data threshold.

[0082] In this embodiment, the prediction module 30 is further configured to: The historical prediction model corresponding to the second detection period is obtained from the model storage pool. The model storage pool stores the historical prediction models corresponding to different detection periods. The second detection period is the previous detection period of the current detection period. The historical prediction model corresponding to the second detection period is a model trained in the second detection period and used to output the first data threshold. The historical prediction model corresponding to the second detection period is determined as the target prediction model; The target prediction model is incrementally trained based on the training dataset so that the trained target prediction model outputs the second data threshold corresponding to the first detection period.

[0083] This embodiment provides a data detection device that divides the current detection cycle into a first processing window and a second processing window. Within the first processing window, real-time detection is performed based on a first data threshold pre-stored in the previous detection cycle, while simultaneously preparing a training dataset for the next detection cycle. Then, within the second processing window, in addition to performing real-time detection based on the first data threshold pre-stored in the previous detection cycle, the training of the prediction model and the calculation of the data threshold for the next cycle are asynchronously completed. This achieves temporal decoupling between detection and prediction, allowing the data threshold for the next detection cycle to be pre-calculated within the current detection cycle. Thus, when the next detection cycle arrives, detection can be completed immediately based on the calculated data threshold. This eliminates the prediction window caused by model training and threshold calculation in traditional real-time detection methods, avoids system resource contention in high-concurrency scenarios, and improves the real-time performance of data detection.

[0084] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 The illustrated electronic device 400 includes at least one processor 401, a memory 402, at least one network interface 404, and other user interfaces 403. The various components in the electronic device 400 are coupled together via a bus system 405. It is understood that the bus system 405 is used to implement communication between these components. In addition to a data bus, the bus system 405 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 4 The general designated all buses as Bus System 405.

[0085] The user interface 403 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0086] It is understood that the memory 402 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 402 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0087] In some implementations, memory 402 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 4021 and application program 4022.

[0088] The operating system 4021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 4022 includes various applications, such as a media player and a browser, used to implement various application functions. Programs implementing the methods of this application embodiment can be included in application program 4022.

[0089] In this embodiment of the application, the processor 401 executes the method steps provided by each method embodiment by calling the program or instructions stored in the memory 402, specifically the program or instructions stored in the application program 4022.

[0090] The methods disclosed in the embodiments of this application can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 402. Processor 401 reads the information in memory 402 and, in conjunction with its hardware, completes the steps of the above method.

[0091] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0092] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0093] The electronic device provided in this embodiment may be as follows: Figure 4 The electronic device shown can perform the following: Figure 1 and Figure 2 All steps of the data detection method in the middle, thereby achieving Figure 1 and Figure 2 For details on the technical effectiveness of the data detection method shown, please refer to [link / reference]. Figure 1 and Figure 2 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0094] This application also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; it may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and it may also include combinations of the above types of memory.

[0095] One or more programs in the storage medium can be executed by one or more processors to implement the data detection method described above that is executed on the data detection device side.

[0096] The processor is used to execute a data detection program stored in the memory to implement the following steps of a data detection method executed on the data detection device side.

[0097] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0098] It should be noted that the terms "one implementation," "embodiment," "exemplary embodiment," and "some embodiments" used in the specification indicate that the described embodiment may include a specific feature, structure, or characteristic, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Moreover, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in conjunction with other embodiments, whether explicitly described or not, is within the knowledge scope of those skilled in the art.

[0099] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data detection method, characterized in that, include: When the current detection cycle is reached, the current detection cycle is divided into a first processing window and a second processing window; Within the first processing window and the second processing window, the acquired business data is detected based on the first data threshold corresponding to the current detection period; Within the first processing window, based on the acquired business data and the historical business dataset acquired before the current detection period, the training dataset corresponding to the first detection period is determined, and within the second processing window, a target prediction model is trained based on the training dataset, so that the trained target prediction model outputs a second data threshold corresponding to the first detection period, and the first detection period is the next detection period of the current detection period; Upon reaching the first detection period, the detection of the business data acquired within the first detection period is completed based on the second data threshold.

2. The method according to claim 1, characterized in that, The step of detecting the acquired business data based on the first data threshold corresponding to the current detection period includes: The first channel detects the acquired business data based on the first data threshold corresponding to the current detection period; The step of determining the training dataset corresponding to the first detection period based on the acquired business data and the historical business dataset acquired before the current detection period includes: The training dataset corresponding to the first detection period is determined by the second channel based on the acquired business data and the historical business dataset acquired before the current detection period; The step of training a target prediction model based on the training dataset, so that the trained target prediction model outputs a second data threshold corresponding to the first detection period, includes: The target prediction model is trained using the second channel based on the training dataset, so that the trained target prediction model outputs the second data threshold corresponding to the first detection period.

3. The method according to claim 2, characterized in that, Within the first processing window, the proportion of system resources allocated to the first channel is greater than the proportion of system resources allocated to the second channel; Within the second processing window, the proportion of system resources allocated to the first channel is less than the proportion of system resources allocated to the second channel.

4. The method according to claim 2, characterized in that, The step of determining the training dataset corresponding to the first detection period through the second channel based on the acquired business data and the historical business dataset acquired before the current detection period includes: The acquired business data is encapsulated into a first message through the first channel, and the first message is sent to the message queue. The first message is obtained from the message queue through the second channel; The training dataset corresponding to the first detection period is determined by the second channel based on the business data in the first message and the historical business dataset obtained before the current detection period.

5. The method according to claim 2, characterized in that, The step of training a target prediction model based on the training dataset through the second channel, so that the trained target prediction model outputs a second data threshold corresponding to the first detection period, includes: The training dataset is encapsulated into a second message through the second channel, and the second message is sent to the message queue. The second message is obtained from the message queue through the second channel; The target prediction model is trained using the second channel based on the training dataset in the second message, so that the trained target prediction model outputs the second data threshold corresponding to the first detection period.

6. The method according to claim 1, characterized in that, The method further includes: After the trained target prediction model outputs the second data threshold corresponding to the first detection period, the correspondence between the first detection period and the second data threshold is stored in the target database. The step of detecting the business data acquired within the first detection period based on the second data threshold upon reaching the first detection period includes: Upon reaching the first detection cycle, the second data threshold corresponding to the first detection cycle is obtained from the target database based on the first detection cycle. The detection of business data acquired during the first detection period is completed based on the second data threshold.

7. The method according to claim 1, characterized in that, The step of training a target prediction model based on the training dataset, so that the trained target prediction model outputs a second data threshold corresponding to the first detection period, includes: The historical prediction model corresponding to the second detection period is obtained from the model storage pool. The model storage pool stores the historical prediction models corresponding to different detection periods. The second detection period is the previous detection period of the current detection period. The historical prediction model corresponding to the second detection period is a model trained in the second detection period and used to output the first data threshold. The historical prediction model corresponding to the second detection period is determined as the target prediction model; The target prediction model is incrementally trained based on the training dataset so that the trained target prediction model outputs the second data threshold corresponding to the first detection period.

8. A data detection method, characterized in that, include: The partitioning module is used to divide the current detection cycle into a first processing window and a second processing window when the current detection cycle is reached. The detection module is used to detect the acquired business data within the first processing window and the second processing window, based on a first data threshold corresponding to the current detection period. The prediction module is used to determine the training dataset corresponding to the first detection period based on the acquired business data and the historical business dataset acquired before the current detection period within the first processing window, and to train the target prediction model based on the training dataset within the second processing window, so that the trained target prediction model outputs the second data threshold corresponding to the first detection period, wherein the first detection period is the next detection period of the current detection period. The detection module is further configured to, upon reaching the first detection period, perform detection of the business data acquired within the first detection period based on the second data threshold.

9. An electronic device, characterized in that, include: A processor and a memory, the processor being configured to execute a data detection program stored in the memory to implement the data detection method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the data detection method according to any one of claims 1 to 7.

Images