Information flow processing method and device, computer equipment and storage medium

Abstract

The invention provides an information stream processing method and device, computer equipment and a storage medium, and the method comprises the steps: receiving a log data stream, and capturing target data from the log data stream; performing unit time grouping on the target data; respectively inputting the target data in each group into at least two different preset marking models, and respectively marking each piece of target data by utilizing the different preset marking models; and intercepting or releasing each piece of target data according to the marking result. The target data is captured from the log data flow and grouped in unit time, so that the system can receive a large amount of log data flow and process the target data in real time; and for each group of target data, each piece of target data is judged and marked by at least two different preset marking models, and the target data is intercepted or released according to the marking result, so that the system can intercept mass data efficiently and flexibly in real time.

Description

Technical Field

[0001] This invention relates to the field of big data, and in particular to an information flow processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the deep penetration of digitalization into fields such as the internet, finance, and industry, servers, terminal devices, and business systems continuously generate massive log information streams containing device status, operation records, and data interactions. These logs are crucial for troubleshooting, business optimization, and security auditing. For example, in monitoring systems, the log data stream contains a large number of root cause alarms and redundant derivative alarms. The system processes the data stream to intercept redundant derivative alarm data, enabling faster identification of system problems.

[0003] However, the current processing of massive log information streams faces significant bottlenecks. The core issue is the difficulty in flexibly, efficiently, and in real-time intercepting and processing massive amounts of target data. On the one hand, log sources are complex and formats are inconsistent. Existing processing solutions mostly rely on fixed rules, lacking flexibility in the face of dynamically changing log types and failing to adapt to the interception needs of different scenarios. On the other hand, the volume of log data is growing explosively. Traditional offline batch processing architectures are time-consuming in transmission, storage, and analysis, making it difficult to meet the needs of real-time interception and processing. At the same time, the processing efficiency is low under massive data. Redundant data that is not intercepted in time will occupy storage and computing resources, further affecting the processing speed of core data and restricting the rapid release of the value of log information streams. Summary of the Invention

[0004] This invention provides an information flow processing method, apparatus, computer equipment, and storage medium to address the problem that information flow processing systems struggle to flexibly, efficiently, and in real-time intercept and process massive amounts of data.

[0005] In a first aspect, embodiments of the present invention provide an information flow processing method, the method comprising: receiving a log data stream and extracting target data from the log data stream; grouping the target data into units of time; inputting the target data in each group into at least two different preset marking models, and marking each piece of target data using the different preset marking models; and intercepting or allowing each piece of target data according to the marking results.

[0006] Secondly, embodiments of the present invention also provide an information flow processing apparatus, which includes a unit for performing the above-described method.

[0007] Thirdly, embodiments of the present invention also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0008] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, can implement the above-described method.

[0009] This application provides an information stream processing method, apparatus, computer device, and storage medium. The method involves a data stream processing end that captures target data from a log data stream and groups the target data into units of time. It leverages Golang's advantage of parallel processing of large amounts of data per unit time using channels, enabling the system to receive large volumes of log data streams and process the target data in real time, reducing memory burden and thus improving processing speed. Furthermore, the model processing end simultaneously uses different preset labeling models to judge and label each piece of target data in each group, utilizing Python's ability to build and apply different machine learning models. The labeling results are returned to the data stream processing end for interception or release of the target data. This allows the system to intercept massive amounts of redundant, correlated, interfering, or system-required target data in real time, efficiently, and flexibly. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A schematic flowchart illustrating the information flow processing method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a sub-process of the information flow processing method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a sub-process of the information flow processing method provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a sub-process of the information flow processing method provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of a sub-process of the information flow processing method provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of a sub-process of the information flow processing method provided in an embodiment of the present invention; Figure 7 A schematic block diagram of an information flow processing device provided in an embodiment of the present invention; Figure 8 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0014] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0015] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] Please see Figure 1 This is a schematic flowchart of the information flow processing method provided in this embodiment of the invention. In this application, the information flow processing method can be applied to real-time monitoring in industrial scenarios such as cement and wind power generation, financial data flow processing systems, real-time transaction volume statistics, smart city and traffic monitoring scenarios, and data flow scenarios for large-scale online services and user experience. Through this information flow processing method, massive amounts of redundant, correlated, interfering, or target data required by the system can be intercepted in real time, efficiently, and flexibly.

[0017] This application provides an information flow processing method, apparatus, computer device, and storage medium. The method includes: receiving a log data stream and extracting target data from the log data stream; grouping the target data into units of time; inputting the target data in each group into at least two different preset tagging models, and tagging each piece of target data using the different preset tagging models; and intercepting or allowing each piece of target data according to the tagging results.

[0018] This application uses the data stream processing end to capture the target data from the log data stream and group the target data by unit time. It leverages the advantage of Golang in processing large amounts of data in parallel through channels within a unit time, enabling the system to receive large amounts of log data streams and process the target data in real time, reducing memory and other memory burdens, thereby improving processing speed. Furthermore, the model processing end uses different preset labeling models to judge and label each piece of target data in each group simultaneously, leveraging Python's ability to build and use different machine learning models. The labeling results are returned to the data stream processing end for interception or release of the target data, enabling the system to intercept massive amounts of redundant, correlated, interfering, or system-required target data in real time, efficiently, and flexibly.

[0019] Figure 1 This is a flowchart illustrating the information flow processing method provided in an embodiment of the present invention. Figure 1 As shown, the information flow processing method is applied to a monitoring platform, which includes a data flow processing end and a model processing end. The method includes the following steps S10-S40.

[0020] S10. Receive log data stream and extract target data from the log data stream; Specifically, the data stream processing end is the module in the system responsible for receiving, parsing, and initially processing log data and executing interception strategies, possessing the ability to efficiently process massive amounts of real-time data. The log data stream refers to a collection of log data continuously transmitted in a time sequence, generated by devices, systems, or business operations, including various types of raw data such as device status, operation records, and alarm information. The target data refers to specific data extracted from the log data stream, such as alarm data, transaction data, and operation data. The specific template data to be obtained can be determined based on the specific use case. For example, in a monitoring platform, if redundant alarm data needs to be intercepted, then the target data refers to the alarm data in the log data stream.

[0021] In the information flow processing method, receiving the log data stream and capturing the target data through the data flow processing terminal is the basic step of the entire process. Its core is to accurately extract the target data from massive real-time logs to provide data support for subsequent analysis, identification and interception.

[0022] In practical implementation, the system's data stream processing end establishes a connection with the monitored object through a preset interface, receiving all log data streams from devices, systems, or services in real time. In alarm interception scenarios, the system receives log data streams from physical machines, virtual machines, various software, and network devices in real time. The log data streams contain various information, including regular records reflecting normal operating status (such as service startup success prompts and periodic status inspection results) and alarm data recording abnormal situations (such as CPU utilization reaching a threshold and memory shortage warnings). The first half of each alarm data includes a trigger timestamp, followed by specific keywords marking the target data, such as "violatedRule." The target data contains a JSON data structure. While all target data structures are consistent, the alarm content recorded in the JSON may differ. Each JSON structure contains nested sub-JSON data structures, resulting in multi-level and multi-layered data information. Additionally, each JSON data structure contains an array containing all violated rules that triggered the alarm.

[0023] When the data stream processing terminal receives the log data stream, it starts a real-time parsing program to quickly identify each log entry in the log data stream according to preset rules and extract the target data. In this embodiment, the Go language is mainly used to determine whether the log data is the target data based on whether the log contains preset specific keywords (such as "violatedRule", "panic", "error", etc.).

[0024] If the log data is the target data, it will be captured and temporarily held to prepare for subsequent data interaction with the model processing end. Meanwhile, if the log data is not the target data, i.e., regular log data not identified as the target data, it will be allowed to pass through and flow along a preset path for daily operation record archiving, and will not enter the subsequent analysis process.

[0025] Therefore, this embodiment, by accurately identifying and separating the target data from regular log data, not only ensures the relevance of subsequent analysis but also avoids irrelevant information from occupying system resources, thus laying an efficient data foundation for intercepting the target data stream.

[0026] Meanwhile, this embodiment uses the Go language to process the log data stream, which can stably handle a large number of parallel processes and receive and efficiently process massive data streams within a unit of time.

[0027] S20. Group the target data by unit time. Specifically, the model processing end is a module used to analyze the target data, identify redundant information, and label it, typically integrating algorithms or machine learning models. The unit time refers to a preset time interval (generally 1 minute) used to control the data processing rhythm of the data stream processing end, reducing memory and other memory burdens, thereby further improving processing speed.

[0028] In the data stream interception process, the data stream processing end groups the target data per unit time and transmits each group of target data to the model processing end, which is a key link connecting data collection and intelligent analysis.

[0029] In practice, after capturing the target data, the data stream processing terminal first divides the target data into time windows according to a preset unit time (such as every minute). This ensures that all alarms generated during that period are aggregated and processed within a fixed cycle, avoiding low analysis efficiency due to fragmented data. For example, if the unit time is set to 1 minute, the system will group all target data generated between 8:00 and 8:01 into the same time window for centralized subsequent processing.

[0030] After the target data is grouped by unit time, the data stream processing end sends each group of target data to the model processing end in batches through the internal communication interface, so that the model processing end can judge and mark the grouped target data.

[0031] Therefore, in this embodiment, the target data is grouped by unit time, transforming the massive and scattered target data into structured group information. This reduces the analysis complexity of the model processing end and ensures the real-time performance of data processing, providing a clear and orderly data foundation for subsequent identification of redundant alarms and interception of target data streams. Simultaneously, the group transmission mechanism also avoids data congestion, ensuring the monitoring platform can operate stably under high load.

[0032] In one embodiment, such as Figure 2 As shown, steps S21-S22 precede step S20.

[0033] S21. Each piece of target data is split into a first array and a second array, wherein the first array stores the original log information of the target data and the second array stores the feature information of the target data; S22. Create an association identifier to bind the first array and the second array.

[0034] Specifically, the first array is a dataset storing the original records of the target data, retaining its complete log information. The original log information refers to all complete information about the target data, including timestamps, device identifiers, and specific anomaly descriptions. The second array is a dataset formed after extracting features from the target data. The feature information refers to the characteristic information in the target data, containing key information for rapid analysis, such as device type (e.g., server, virtual machine), indicator type (e.g., CPU, memory), system category (e.g., database, application service), and alarm level (e.g., prompt, warning, error). The association identifier is a unique identifier generated for each piece of target data, used to establish the correspondence between the first array and the second array, ensuring that the original information and feature information are mutually traceable.

[0035] Before grouping the target data by unit time type, it is necessary to first complete the splitting and association binding of each target data item. This is the foundation for achieving structured data processing. In specific implementation, after the data stream processing end captures the target data, it will perform a splitting operation on each target data item: storing the original log information containing the complete context into the first array, and extracting core features from the original log and storing them into the second array. For example, if an alarm log item is "2024-05-20 10:15:03, Server A (IP:192.168.1.10) CPU utilization reached 98%, triggering the warning threshold of 90%", the first array will completely save the entire content of this log item; the second array will extract key features from it, including alarm time (2024-05-20 10:15:03), involved object (Server A), indicator type (CPU utilization), outlier (98%), threshold (90%), etc., forming structured feature entries.

[0036] After the data is split, the system creates an association identifier for the target data. This association identifier is typically generated by combining a timestamp, a device identifier hash value, and a random sequence to ensure uniqueness. For example, for the alarm from server A mentioned above, the generated association identifier might be "20240520101503_7f3d2_9x5p", where "20240520101503" corresponds to the alarm time, "7f3d2" is the simplified hash value of server A's IP address, and "9x5p" is the random sequence. The system adds this association identifier to both the corresponding first and second arrays, forming a binding relationship of "original log information—feature information—association identifier".

[0037] For the incoming target data in batches, this process is executed automatically one by one: after each target data item enters the system, the parsing module first separates the original information and feature information, storing them in the first array and the second array respectively. Then, the identifier generation module creates the association identifier in real time and completes the binding. The two sets of data after binding are temporarily stored in the cache of the data stream processing end. For example, if 100 target data items are generated within 1 minute, the system will generate 100 sets of mutually bound first arrays and second arrays. Each set is established with a unique association identifier, ensuring that during subsequent grouping and analysis, the feature information of the second array can be quickly called for judgment and marking, and the original log details in the first array can be traced at any time through the association identifier to perform interception or release operations.

[0038] This embodiment achieves the separate storage of original information and feature information through data splitting, which not only preserves the complete original data for traceability, but also extracts structured features for rapid processing; while the binding of the association identifier ensures the accurate correspondence between the two types of data, providing a reliable data foundation for subsequent model analysis and feature recognition, and also facilitating information tracing during problem investigation.

[0039] More specifically, the first array is temporarily held within the data stream processing end, waiting for the JSON features (i.e., feature information) dynamically extracted from the original log data by the second array to be sent by Kafka to the model processing end for model processing and interception marking before being returned to guide the interception of the first array. The first array will continuously update and delete the marked information to ensure that the memory and CPU of the data stream processing end are not overloaded.

[0040] In one embodiment, such as Figure 3 As shown, step S21 includes steps S211-S212.

[0041] S211. Convert the target data into a JSON data structure; S212. Dynamically extract the feature information from the target data through a tree structure and store the feature information in the second array.

[0042] Specifically, the JSON data result is a lightweight data exchange format that organizes data in key-value pairs. It is easy to parse and cross-platform compatible, facilitating subsequent structured data processing. The tree structure is a hierarchical data organization form consisting of a root node and child nodes, which can clearly reflect the hierarchical relationship between data and is used to dynamically locate and extract specific information from the target data.

[0043] In this embodiment, during the process of splitting each piece of target data into the first array and the second array, the captured target data is first processed by format conversion. According to the JSON data structure specification, various types of information in the target data are mapped to corresponding key-value pairs. For example, the attribute name of the data is used as the "key" and the specific content corresponding to the attribute is used as the "value", forming structured JSON data to ensure that the hierarchy and relationship of the data are clearly distinguishable.

[0044] Next, a tree structure is constructed based on the converted JSON data, with the root key of the JSON data serving as the root node of the tree structure, and each child key corresponding to a child node of the tree structure, thus fully presenting the hierarchical structure of the target data. Subsequently, through the node traversal function of the tree structure, feature information is dynamically located and extracted from the target data according to preset feature information extraction rules (such as specifying the key name and data type to be extracted). During the extraction process, the extracted information is verified in real time to ensure that it conforms to the definition of feature information. After confirmation, this feature information is stored in the second array in an orderly manner.

[0045] In this embodiment, the feature information of the target data is extracted into the second array, which facilitates data transmission and subsequent model analysis and labeling steps, and avoids consuming too much CPU, computing power and memory.

[0046] More specifically, for the second array, the code is designed to dynamically extract data from the alarm JSON information based on a tree structure. The content of the tree structure and its hierarchical information must completely match the key names and hierarchical parent-child relationships in the alarm JSON; otherwise, dynamic extraction cannot be achieved. The number of features extracted into JSON is much smaller than the original target data, facilitating large-scale data transmission and model marking and interception. After this [alarm log - to - feature dataset] step, the data stream processing end can continue to process the corresponding log data stream in a loop, and the model processing end can promptly apply the model to intercept, judge, and mark the data after receiving the required feature data. The rapid transmission of large amounts of data through the Kafka middleware, with each component performing its specific function, improves efficiency.

[0047] In one embodiment, step S20 includes step S23.

[0048] S23. Enter the second array within the same unit of time into the same temporary data table.

[0049] Specifically, the temporary data table is a temporary data structure table created by the data stream processing end to temporarily store all the second arrays within a certain unit of time, which facilitates centralized analysis and marking operations of the model.

[0050] When grouping the target data by unit time, the data stream processing end operates according to a fixed process to ensure efficient and accurate grouping. Specifically, the system first uses a preset unit time (e.g., 1 minute) as a cycle and centrally records all second arrays generated within that time period into the same temporary data table. For example, if the unit time is set to 1 minute, the second arrays corresponding to the 50 pieces of target data generated between 8:30 and 8:31 will be uniformly stored in the temporary data table named "202406100830-0831". The temporary data table automatically assigns a row record to each second array and retains its association identifier for subsequent tracing of the first array.

[0051] Therefore, this embodiment achieves centralized management of alarm features within a unit of time through the temporary data table, which not only ensures the aggregation of data in the same time window, but also provides structured data support for subsequent model processing analysis and labeling, while avoiding data interference from different times and improving the processing efficiency of the entire monitoring system.

[0052] More specifically, the temporary data table is cleared after each processing mark is made, without archiving historical content. Simultaneously, the [detention-interception marking-destruction / release] loop for target data in the data stream is also completed within a unit of time. Thus, while utilizing the CPU to speed up processing as much as possible, the memory burden is reduced, thereby further improving processing speed.

[0053] S30. Input the target data in each group into at least two different preset labeling models, and use the different preset labeling models to label each piece of target data; Specifically, the model processing terminal is the core module of the system responsible for analyzing and labeling the target data, and it has the ability to run multiple algorithm models. The preset labeling model is an algorithm model pre-configured in the model processing terminal. Each model determines whether the target data needs to be intercepted for a specific type of data. The labeling result is a label generated by the analysis conclusion of the preset labeling model on the target data, including two types: "intercept and delete" and "allow".

[0054] In the data stream processing flow, the model processing end receives the target data, which is grouped and sent by the data stream processing end within a unit time period, through an internal communication channel.

[0055] After receiving the data, the model processing terminal inputs each target data in each group into at least two different preset labeling models simultaneously. For example, if the target data is divided into 3 groups, with 5 target data in each group, and there are 4 preset labeling models, then each group of target data is simultaneously input into the 4 preset labeling models so that the 4 preset labeling models label the 5 target data in each group.

[0056] More specifically, each of the preset labeling models is loaded with targeted analysis logic. For example, in the field of alarm suppression, tree-based alarm storm suppression methods and cluster information-based alarm storm suppression methods can be used; in financial data stream processing systems, risk labeling methods based on transaction chain tracing and anomaly identification methods based on fund pool correlation can be used; in real-time transaction volume statistics scenarios, fluctuation labeling methods based on time series decomposition and benchmark deviation methods based on merchant type clustering can be used; and in smart city and traffic monitoring scenarios, anomaly event labeling methods based on spatiotemporal density clustering and violation identification methods based on traffic rule base matching can be loaded.

[0057] Each target data item is simultaneously marked using different preset marking models. The system then compares the characteristic information within the target data (such as alarm frequency, range of affected devices, and correlation with historical root cause alarms) to determine whether the target data needs to be blocked. Furthermore, the preset marking models mark the target data according to the judgment results, and the marking of each target data item is limited to either "block" or "allow".

[0058] After all target data in each group is labeled, the model processing end returns the labeling results for each data item in batches to the data stream processing end. The data stream processing end then intercepts target data marked as "intercepted" based on the labeling results, and only pushes "allowed" target data to downstream systems (such as alarm notification modules).

[0059] This embodiment simultaneously inputs each group of data into the corresponding preset marking model for marking, so that multiple preset marking models can simultaneously judge and mark the target data in each group, thereby improving the accuracy and flexibility of intercepting the target data; the accurate judgment of the preset marking model realizes the effective identification of the target data, and the two clear marking results simplify the subsequent processing flow and improve the operating efficiency of the monitoring system.

[0060] In one embodiment, such as Figure 4 As shown, step S30 includes steps S31-S32.

[0061] S31. Input each of the temporary data tables into different preset labeling models; S32. The different preset marking models respectively mark each group of the second array in the temporary data table.

[0062] Specifically, after grouping the target data by unit time, the data stream processing terminal will record all the second arrays generated within the time period into the same temporary data table at a preset unit time (e.g., 1 minute).

[0063] The model processing terminal receives the temporary data table, where each record corresponds to one piece of target data, clearly presenting the content of the second array and ensuring a standardized and complete data structure. Subsequently, the model processing terminal uses a multi-threaded parallel transmission mechanism to simultaneously distribute each temporary data table to different preset labeling models. That is, a temporary data table is simultaneously input into different preset labeling models, maintaining data synchronization during transmission and avoiding excessive time differences in data reception between different models due to transmission delays. After receiving each temporary data table, each preset labeling model automatically identifies the second array fields in the table and, based on its own preset labeling rules (such as feature matching logic, attribute classification standards, etc.), simultaneously labels each group of the second array in the table.

[0064] The labeling operations of different models are independent of each other, and are carried out only on the feature information in the second array, without involving the content of the first array. After the labeling is completed, the labeling information corresponding to the second array is generated. After the entire set of data is processed, it waits to be summarized and returned to the data processing terminal.

[0065] In this embodiment, different preset labeling models are used to simultaneously label the second array in each of the temporary data tables, avoiding redundant operations of distributing single data. This allows the model processing end to synchronously input data into different preset labeling models in a "batch distribution" manner, thereby improving the efficiency and accuracy of data labeling. At the same time, the "parallel processing" mode of the model processing end and the preset labeling models decouples data transmission from the labeling task. If a preset labeling model experiences a brief failure, it only affects the labeling of the corresponding dimension and does not interrupt the processing flow of other models. This avoids the paralysis of the entire labeling process due to a single node failure, improves the system's fault tolerance and operational stability, and thus enhances the system's processing flexibility and stability.

[0066] In one embodiment, such as Figure 5 As shown, step S32 is followed by steps S321-S323.

[0067] S321. Record the marking results in the second array; S322. Associate the first array and the second array according to the association identifier; S323. Based on the marking result, the first array is intercepted or allowed to pass.

[0068] Specifically, after each set of data is input into the corresponding preset marking model, the preset marking model judges and marks each second array within the group. In specific implementation, different preset marking models will analyze and judge the second arrays received from different groups: each model evaluates each second array within the group according to built-in rules (such as the range of associated devices of the alarm, the correlation with historical root cause alarms, the frequency of occurrence, etc.), and obtains the marking result of "intercept" or "allow", and records the marking result directly in the second array. For example, in the model for handling server memory alarms, if a second array shows that the excessive memory usage is caused by the abnormal process of the same server, it will be marked "intercepted" in the second array; if the memory alarm occurs independently and continues to exceed the threshold, it will be marked "allowed".

[0069] After marking is completed, the system associates the first array with the corresponding second array that records the marking results using the association identifier. Since each second array is bound to the first array with the unique association identifier when it is generated, it is only necessary to match the association identifier to establish a correspondence between the first array storing the original logs and the second array with the marking results.

[0070] Finally, the system processes the first array according to the marking results in the second array: if the second array is marked as "intercepted", the associated first array (original alarm log) will be directly intercepted by the system and will not enter the downstream alarm notification process; if it is marked as "allowed", the first array will be allowed to continue to flow and will eventually be sent to the operation and maintenance personnel through the alarm push mechanism.

[0071] This embodiment achieves a precise correspondence between the first array and the marking results by directly recording the marking results in the second array and combining them with the associated identifier, ensuring the continuity of the processing logic. Simultaneously, by intercepting or allowing the first array based on the marking results, the effectiveness and reliability of the monitoring system are improved.

[0072] S40. The data stream processing terminal intercepts or allows each target data item based on the marking result; Specifically, in the target data stream interception process, the data stream processing end performs interception, deletion, or release operations on each piece of target data based on the marking results, which is a key step in ultimately achieving redundant information filtering. In specific implementation, the data stream processing end receives all the marking results returned by the model processing end, and each marking result is associated with the corresponding target data.

[0073] Subsequently, the data stream processing end traverses all the target data, reads the marking result of each target data, and performs the corresponding processing operations in batches. For all the target data marked as "intercepted", the data stream processing end triggers an interception mechanism: not only is it removed from the current processing queue, but the temporary cache associated with the data is also cleared to avoid consuming system resources.

[0074] For all the target data marked "released", the data stream processing terminal will allow it to continue to flow, that is, release the target data in batches and push the data to the downstream system according to the preset path.

[0075] In this embodiment, the data stream processing terminal strictly performs operations based on the marking results, which not only completely intercepts the data that needs to be intercepted, preventing the data stream from interfering with the system and personnel, but also ensures the smooth transmission of effective information, providing accurate information support for troubleshooting. At the same time, the data stream processing terminal can improve the system's processing speed by performing batch interception or release operations on the target data within a unit of time, so that the system can flexibly, efficiently, and in real time intercept massive amounts of data.

[0076] In one embodiment, such as Figure 6 As shown, step S40 includes steps S41-S43.

[0077] S41. Determine whether the marking result indicates interception or passage; S42. If the marking result is interception, then the target data is intercepted and deleted. S43. If the marking result is "release", then release the target data.

[0078] Specifically, in the information flow interception process, the data flow processing end intercepts, deletes, or allows each piece of target data based on the marking results, which is the final step in ensuring that the monitoring system outputs accurate information. In specific implementation, the data flow processing end first receives all target data and the corresponding marking results returned by the model processing end. These data have been bound through association identifiers, and each piece of target data is accompanied by a clear "intercept" or "allow" mark.

[0079] Subsequently, the data stream processing terminal initiates a judgment mechanism to determine the marking result of each target data item: the system reads the content in the marking result field to determine the operation type that should be performed on the current target data. If the marking result is "intercepted," it indicates that the target data is redundant derived information. The data stream processing terminal will immediately perform an interception and deletion operation, removing the target data from the processing queue and simultaneously clearing the relevant temporary storage cache to avoid redundant data consuming resources.

[0080] If the marking result is "allow," it indicates that the target data is valid information that requires attention. The data stream processing end will perform an allow operation, retain the complete information of the target data (including original logs and feature information), and push it to the downstream alarm notification module. This data will be presented to the operations and maintenance personnel in real time via SMS, email, or monitoring dashboard to ensure that critical anomalies are handled promptly.

[0081] In this embodiment, the data stream processing terminal strictly judges the labeling result of the target data, avoiding the flood of invalid information, effectively intercepting the data stream, and ensuring the smooth transmission of key information.

[0082] This application uses the data stream processing end to capture the target data from the log data stream and group the target data by unit time. It leverages the advantage of Golang in processing large amounts of data in parallel through channels within a unit time, enabling the system to receive large amounts of log data streams and process the target data in real time, reducing memory and other memory burdens, thereby improving processing speed. Furthermore, the model processing end uses different preset labeling models to judge and label each piece of target data in each group simultaneously, leveraging Python's ability to build and use different machine learning models. The labeling results are returned to the data stream processing end for interception or release of the target data, enabling the system to intercept massive amounts of redundant, correlated, interfering, or system-required target data in real time, efficiently, and flexibly.

[0083] Figure 7 This is a schematic block diagram of an information flow processing device 300 provided in an embodiment of the present invention. Figure 7 As shown, corresponding to the above information flow processing method, the present invention also provides an information flow processing apparatus 300. This information flow processing apparatus 300 includes a unit for performing the above information flow processing method, and the apparatus can be configured in a computer device. Specifically, please refer to... Figure 7 The information flow processing device 300 includes a processing unit 301 and a marking unit 302.

[0084] Processing unit 301 receives a log data stream and extracts target data from the log data stream; groups the target data into units of time; intercepts or allows each piece of target data according to the marking result; splits each piece of target data into a first array and a second array, the first array storing the original log information of the target data and the second array storing the feature information of the target data; creates an association identifier to bind the first array and the second array; converts the target data into a JSON data structure; dynamically extracts the feature information from the target data through a tree structure and stores the feature information in the second array; enters the second array within the same unit of time into the same temporary data table; records the marking result in the second array; associates the first array and the second array according to the association identifier; intercepts or allows the first array according to the marking result; inputs each temporary data table into different preset marking models; determines whether the marking result is interception or allowance; if the marking result is interception, the target data is intercepted and deleted; if the marking result is allowance, the target data is allowed. The marking unit 302 inputs the target data in each group into at least two different preset marking models, marks each piece of target data using the different preset marking models, and returns the marking results to the data stream processing end; the different preset marking models mark each group of the second array in the temporary data table.

[0085] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned information flow processing device and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0086] The aforementioned information flow processing device 300 can be implemented as a computer program, which can, for example... Figure 8 It runs on the computer device shown.

[0087] Please see Figure 8 , Figure 8 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a smartphone, tablet, laptop, desktop computer, personal digital assistant, or wearable device. The server can be a standalone server or a server cluster composed of multiple servers.

[0088] See Figure 8The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0089] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform an information flow processing method.

[0090] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0091] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute an information flow processing method.

[0092] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0093] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of the above-described information flow processing method.

[0094] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0095] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0096] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the steps of the information flow processing method described above.

[0097] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0098] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0099] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0100] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0101] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0102] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An information flow processing method, characterized in that, The method includes: Receive log data streams and extract target data from the log data streams; The target data is grouped by unit time. The target data in each group is input into at least two different preset labeling models, and each piece of target data is labeled using the different preset labeling models; Based on the marking results, each piece of target data is either intercepted or allowed to pass.

2. The method according to claim 1, characterized in that, Prior to the step of grouping the target data by unit time, the following steps are included: Each piece of target data is split into a first array and a second array. The first array stores the original log information of the target data, and the second array stores the feature information of the target data. Create an association identifier to bind the first array and the second array.

3. The method according to claim 2, characterized in that, The step of grouping the target data by unit time includes: The second array within the same unit of time is entered into the same temporary data table.

4. The method according to claim 3, characterized in that, The step of inputting the target data in each group into at least two different preset labeling models, and using the different preset labeling models to label each piece of target data, includes: Each of the temporary data tables is input into a different preset labeling model; The different preset labeling models respectively label each group of the second array in the temporary data table.

5. The method according to claim 4, characterized in that, After the step of marking each group of the second array in the temporary data table using the different preset marking models, the following is included: The marking results are recorded in the second array; The first array and the second array are associated according to the association identifier; The first array is intercepted or allowed based on the marking results.

6. The method according to claim 2, characterized in that, The step of splitting each piece of target data into a first array and a second array includes: Convert the target data into a JSON data structure; The feature information is dynamically extracted from the target data using a tree structure, and the feature information is stored in the second array.

7. The method according to claim 1, characterized in that, The step of intercepting or allowing each piece of target data based on the marking result includes: Determine whether the marking result indicates interception or passage. If the marking result is interception, then the target data is intercepted and deleted. If the marking result is "allow", then the target data is allowed to pass.

8. An information stream processing device, characterized in that, Includes a unit for performing the method as described in any one of claims 1-7.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, can implement the method as described in any one of claims 1-7.

Images