Method for realizing large data volume early warning based on memory cache subscription and release

Abstract

The invention relates to the technical field of big data early warning, and discloses a method for realizing large data volume early warning based on memory cache subscription publishing, and the method comprises the steps: arranging an early warning system in a to-be-monitored region, and starting the early warning system; setting an information monitor to continuously monitor the real-time information flow of the to-be-monitored area; the to-be-processed information forms key-value pairs according to the message identifier ID or according to the message identifier ID and the MD5 check code of the corresponding picture file, and the key-value pairs are cached in a memory in batches; judging whether the message to be analyzed forms early warning information or not according to an early warning rule, and further judging whether the early warning information needs to be stored persistently or not; and after the pre-warning information storage operation is completed, cleaning corresponding invalid or expired data in the cache according to the current pre-warning system time and the message identifier ID. According to the invention, the problems of response delay, false alarm, missing alarm and the like when a traditional system processes large-data-volume early warning information are effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of big data early warning technology, and in particular to a method for implementing large data volume early warning based on memory cache subscription and publishing. Background Art

[0002] In the development of smart cities and intelligent transportation, deploying tens of thousands of cameras for real-time monitoring and data collection has become a common practice. The parallel operation of these cameras generates massive amounts of warning data streams, which must be processed in real time to generate timely alarm information. Processing speed directly determines system efficiency; delayed responses will affect the accuracy of alarm information. Traditionally, after receiving data, systems often immediately store the information in relational or time-series databases, permanently writing the information to disk. However, when faced with tens of thousands of cameras, this approach often leads to false alarms and missed alarms.

[0003] Therefore, it is necessary to design a method based on memory cache subscription and publishing to achieve large data volume early warning to solve the problems existing in current technology. Summary of the Invention

[0004] In view of this, the present invention proposes a method for realizing large data volume early warning based on memory cache subscription and publishing, aiming to improve data processing speed and reduce the delay of data writing to the hard disk through memory cache technology, thereby ensuring the accuracy and timeliness of alarm information.

[0005] The present invention proposes a method for implementing large data volume early warning based on memory cache subscription and publishing, comprising:

[0006] Determine the area to be monitored, deploy the early warning system within the area to be monitored, and start the early warning system; in the initialization phase of the early warning system, pre-load and cache the public basic data, early warning rules, early warning information subscription quantity, and monitoring time used for information analysis and processing into the memory of the early warning system;

[0007] An information listener is set to continuously monitor the real-time information flow of the monitored area; when the access of the information to be processed is detected, the information to be processed is combined into a key-value pair according to the message identification ID or the MD5 checksum of the message identification ID and its corresponding image file, and is cached in batches into the memory;

[0008] Extracting the message to be analyzed from the memory, determining whether the message to be analyzed constitutes warning information according to the warning rule, and further determining whether the warning information needs to be persistently stored;

[0009] When it is determined that the message to be analyzed belongs to warning information, classifying and grouping the warning information according to the warning type corresponding to the warning information;

[0010] When it is determined that the warning information needs to be persisted, the warning information to be persisted is extracted in batches from the memory according to the warning type of the warning information and the corresponding warning information subscription quantity, and is saved in a relational database using an asynchronous write method;

[0011] After completing the warning information saving operation, the number of warning information subscriptions corresponding to the warning type of the warning information in the memory is updated to zero, and the monitoring time is updated to the current warning system time; according to the current warning system time and the message identification ID, the corresponding invalid or expired data in the cache is cleared;

[0012] A monitoring daemon service is set as a daemon thread of the message receiving service, and the change state of the monitoring time is periodically detected according to a preset time interval, and whether the running state of the message receiving service is normal is determined according to the change state.

[0013] Furthermore, when the public basic data, warning rules, warning information subscription quantity and monitoring time for information analysis and processing are pre-loaded and cached in the memory of the warning system, it includes:

[0014] Obtaining initial values ​​corresponding to the public basic data, warning rules, warning information subscription quantity and monitoring time by reading a configuration file or a database;

[0015] Loading the initial value into the memory cache area of ​​the early warning system and establishing a corresponding data structure; wherein the data structure is a key-value pair structure;

[0016] During the loading process, the validity of the initial value is verified;

[0017] If the verification fails, an error message is issued and the initialization process of the early warning system is terminated.

[0018] Furthermore, an information listener is set to continuously monitor the real-time information flow of the area to be monitored, including:

[0019] Receive the real-time information flow of the area to be monitored through the monitoring interface and parse the real-time information flow;

[0020] According to the analysis result, the valid data in the real-time information stream is encapsulated in a preset format to form the information to be processed;

[0021] The information to be processed is sent to the information listener for monitoring and processing.

[0022] Furthermore, the information to be processed is combined into a key-value pair according to the message identification ID or the message identification ID and the MD5 checksum of its corresponding image file, and cached in batches into the memory, including:

[0023] Uniquely identify the received information to be processed and generate a message identification ID;

[0024] If the information to be processed contains an image file, perform MD5 verification on the image file to generate a corresponding MD5 verification code;

[0025] Classify the information to be processed into corresponding batches according to the message identification ID or a key-value pair combination of the message identification ID and the MD5 checksum;

[0026] The classified information to be processed is cached in batches into the memory cache area of ​​the early warning system.

[0027] Furthermore, when judging whether the message to be analyzed constitutes warning information according to the warning rule, it includes:

[0028] Extracting key fields from the message to be analyzed and matching the key fields with warning rules;

[0029] If the match is successful, it is determined that the message to be analyzed constitutes warning information;

[0030] If the match is unsuccessful, the next message to be analyzed will be judged.

[0031] Furthermore, further determining whether the warning information needs to be persisted includes:

[0032] Determining a warning level corresponding to the warning information;

[0033] Preset persistent storage rules, and determine whether the warning information needs to be persistently stored according to the persistent storage rules;

[0034] If the warning level of the warning information meets the persistent storage rule, it is determined that the warning information needs to be persistently stored;

[0035] If the warning level of the warning information does not comply with the persistent storage rule, it is determined that the warning information does not need to be persistently stored.

[0036] Furthermore, according to the warning type of the warning information and the corresponding warning information subscription quantity, the warning information to be persisted is batch extracted from the memory and saved in the relational database in an asynchronous writing manner, including:

[0037] Determine the corresponding warning information subscription list based on the warning type of the warning information;

[0038] Traversing the warning information subscription list, and extracting the warning information to be persisted from the memory according to the number of warning information subscriptions;

[0039] Using asynchronous writing, the extracted warning information is written into the relational database in batches according to the warning type, and during the writing process, the writing status and writing time corresponding to each warning information are recorded.

[0040] Furthermore, when clearing the corresponding invalid or expired data in the cache according to the current warning system time and the message identification ID, it includes:

[0041] Obtaining the current warning system time, and determining whether the current warning system time exceeds a preset cache clearing time threshold;

[0042] If it exceeds, the cached data in the memory will be traversed, and invalid or expired warning information will be identified and cleared according to the warning time and message ID;

[0043] After the cleanup is completed, the corresponding memory space is released.

[0044] Furthermore, when periodically detecting the change state of the monitoring time according to a preset time interval, it includes:

[0045] Set the monitoring daemon service timer to trigger the detection task at the preset time interval;

[0046] Each time a detection task is triggered, the monitoring time of the current early warning system is read;

[0047] Compare the read monitoring time with the monitoring time recorded during the last detection to determine whether the monitoring time has changed;

[0048] If the monitoring time changes, calculate the difference between the current monitoring time and the monitoring time recorded during the last detection, and record it as the monitoring time difference, and determine whether the running status of the message receiving service is normal based on the monitoring time difference;

[0049] If the monitoring time does not change, the periodic detection will continue at the preset time interval.

[0050] Furthermore, it is characterized in that, when judging whether the running status of the message receiving service is normal according to the monitoring time difference, it includes:

[0051] Compare the monitoring time difference with the monitoring time difference threshold, and determine whether the running status of the message receiving service is normal based on the comparison result.

[0052] If the monitoring time difference is greater than or equal to the monitoring time difference threshold, it is determined that the running state of the message receiving service is abnormal, and an alarm prompt is issued;

[0053] If the monitoring time difference is less than the monitoring time difference threshold, it is determined that the running state of the message receiving service is normal, and periodic detection is continued at a preset time interval.

[0054] Compared with the prior art, the beneficial effect of the present invention is that the method provided by the present invention for realizing large-scale early warning based on memory cache subscription and publishing effectively solves the problems of response delay, false alarm and missed alarm when traditional systems process large-scale early warning information. Through memory cache technology, rapid processing and real-time analysis of data are achieved, which significantly improves the efficiency and accuracy of the early warning system. At the same time, the method also optimizes the persistent storage and memory management of data through strategies such as asynchronous writing and regular cache cleaning, further improving the stability and reliability of the system. In addition, the method also has good scalability and flexibility, can adapt to early warning needs of different scales and complexities, and provides strong technical support for the development of smart cities and intelligent transportation. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0056] Figure 1 This is a flowchart of a method for implementing large data volume early warning based on memory cache subscription and publishing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, unless there is a conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0058] See Figure 1 As shown, in some embodiments of the present application, this embodiment provides a method for implementing large data volume warning based on memory cache subscription and publishing, including the following steps:

[0059] S100: Determine an area to be monitored, deploy an early warning system within the area to be monitored, and start the early warning system; during the initialization phase of the early warning system, pre-load and cache public basic data, early warning rules, early warning information subscription quantity, and monitoring time used for information analysis and processing into the memory of the early warning system;

[0060] S200: Setting an information listener to continuously monitor the real-time information flow of the monitored area; when detecting access of information to be processed, forming a key-value pair of the information to be processed according to the message identification ID or the MD5 checksum of the message identification ID and its corresponding image file, and caching the key-value pair in batches into the memory;

[0061] S300: extracting a message to be analyzed from the memory, determining whether the message to be analyzed constitutes warning information according to a warning rule, and further determining whether the warning information needs to be persistently stored;

[0062] S400: When it is determined that the message to be analyzed is warning information, classify and group the warning information according to the warning type corresponding to the warning information;

[0063] S500: When it is determined that the warning information needs to be persisted, the warning information to be persisted is batch-extracted from the memory according to the warning type of the warning information and the corresponding warning information subscription quantity, and is saved in a relational database using an asynchronous write method;

[0064] S600: After completing the operation of saving the warning information, the number of subscriptions to the warning information corresponding to the warning type of the warning information in the memory is updated to zero, and the monitoring time is updated to the current warning system time; the corresponding invalid or expired data in the cache is cleared according to the current warning system time and the message identification ID;

[0065] S700: Setting a monitoring daemon service as a daemon thread of a message receiving service, periodically detecting a change in the monitoring time at a preset time interval, and determining whether the running state of the message receiving service is normal according to the change.

[0066] It is understandable that the method provided in this embodiment for implementing large-scale early warning based on memory cache subscription and publishing effectively solves the problems of response delay, false alarm and missed alarm when traditional systems process large-scale early warning information. Through memory caching technology, rapid processing and real-time analysis of data are achieved, which significantly improves the efficiency and accuracy of the early warning system. At the same time, the method also optimizes the persistent storage and memory management of data through strategies such as asynchronous writing and regular cache cleaning, further improving the stability and reliability of the system. In addition, the method also has good scalability and flexibility, can adapt to early warning needs of different scales and complexities, and provides strong technical support for the development of smart cities and intelligent transportation.

[0067] Specifically, when the public basic data, warning rules, warning information subscription quantity and monitoring time used for information analysis and processing are pre-loaded and cached in the memory of the warning system, it includes:

[0068] Obtaining initial values ​​corresponding to the public basic data, warning rules, warning information subscription quantity and monitoring time by reading a configuration file or a database;

[0069] Loading the initial value into the memory cache area of ​​the early warning system and establishing a corresponding data structure; wherein the data structure is a key-value pair structure;

[0070] During the loading process, the validity of the initial value is verified;

[0071] If the verification fails, an error message is issued and the initialization process of the early warning system is terminated.

[0072] Understandably, validating the initial values ​​is a critical step in ensuring the accurate and efficient operation of the early warning system. This validation allows for the timely discovery and correction of potential errors or inappropriate settings in the configuration files, thereby avoiding data errors, rule failures, or performance degradation during the operation of the early warning system. Specifically, validation may include checking the format, scope, and integrity of the data, as well as the logical consistency between rules. Only after all initial values ​​have passed validation will the early warning system proceed with subsequent initialization processes, such as starting the information listener and establishing the memory cache structure. This not only improves the robustness and reliability of the early warning system, but also provides users with a more user-friendly and convenient experience.

[0073] Specifically, when setting an information listener to continuously monitor the real-time information flow of the area to be monitored, it includes:

[0074] Receive the real-time information flow of the area to be monitored through the monitoring interface and parse the real-time information flow;

[0075] According to the analysis result, the valid data in the real-time information stream is encapsulated in a preset format to form the information to be processed;

[0076] The information to be processed is sent to the information listener for monitoring and processing.

[0077] It's understandable that setting up an information listener is a key step in the early warning system's ability to acquire and process information from the monitored area in real time. Through the monitoring interface, the system can continuously receive real-time information streams from the monitored area. These information streams include various sensor data, user behavior records, system logs, and more. To ensure the accuracy and efficiency of subsequent analysis and processing, the system parses the received real-time information stream, extracts valid data, and packages it according to a preset format to form the information to be processed. This information is then sent to the information listener for further monitoring and processing. This step not only ensures the real-time and integrity of the information but also provides a reliable data foundation for subsequent early warning analysis and processing.

[0078] Specifically, the information to be processed is formed into a key-value pair according to the message identification ID or the message identification ID and the MD5 checksum of its corresponding image file, and is cached in batches into the memory, including:

[0079] Uniquely identify the received information to be processed and generate a message identification ID;

[0080] If the information to be processed contains an image file, perform MD5 verification on the image file to generate a corresponding MD5 verification code;

[0081] Classify the information to be processed into corresponding batches according to the message identification ID or a key-value pair combination of the message identification ID and the MD5 checksum;

[0082] The classified information to be processed is cached in batches into the memory cache area of ​​the early warning system.

[0083] In this embodiment, the message ID is the key and the information to be processed is the value. This key-value pair combination can efficiently manage and access the information to be processed in memory. If an image file exists, the message ID and the MD5 code of the image file are used as the key to form a Key / Value pair.

[0084] It is understandable that the information to be processed is combined into a key-value pair based on the message identification ID or the MD5 checksum of the image file, and then cached in batches into memory. This step greatly improves the efficiency and accuracy of information processing. Through unique identification and MD5 verification, the system can ensure the uniqueness and integrity of each piece of information to be processed, avoiding repeated processing and erroneous analysis of information. At the same time, the batch caching method also optimizes the use of memory, allowing the system to process large amounts of data more efficiently. This not only improves the real-time and accuracy of the early warning system, but also provides users with a more reliable and stable early warning service.

[0085] Specifically, judging whether the message to be analyzed constitutes warning information according to the warning rules includes:

[0086] Extracting key fields from the message to be analyzed and matching the key fields with warning rules;

[0087] If the match is successful, it is determined that the message to be analyzed constitutes warning information;

[0088] If the match is unsuccessful, the next message to be analyzed will be judged.

[0089] It's understood that the messages to be analyzed are initially screened and collated from the messages to be processed, and are used for further analysis to determine whether they constitute warning information. The messages to be processed are raw data received and cached in real time by the warning system. This data may include various sensor data, user behavior records, system logs, and so on. The messages to be analyzed, based on this data and tailored to the needs of the warning system, extract key fields and information for subsequent warning analysis and processing.

[0090] It's understandable that determining whether the message being analyzed constitutes a warning based on warning rules is the core component of the early warning system's early warning functionality. By extracting key fields from the message being analyzed and matching them with pre-set warning rules, the system can accurately identify potential risks or anomalies. This rule-based matching approach not only improves the accuracy and pertinence of warnings, but also enables the system to flexibly adapt to diverse warning needs. When a match is successful, the system immediately determines that the message being analyzed constitutes a warning and triggers the corresponding warning process. If a match is unsuccessful, the system continues to determine whether the message being analyzed constitutes a warning until all messages being analyzed have been processed. This efficient and accurate processing method enables the early warning system to maintain stable performance and reliable early warning capabilities in large data environments.

[0091] Specifically, further determining whether the warning information needs to be persisted includes:

[0092] Determining a warning level corresponding to the warning information;

[0093] Preset persistent storage rules, and determine whether the warning information needs to be persistently stored according to the persistent storage rules;

[0094] If the warning level of the warning information meets the persistent storage rule, it is determined that the warning information needs to be persistently stored;

[0095] If the warning level of the warning information does not comply with the persistent storage rule, it is determined that the warning information does not need to be persistently stored.

[0096] Understandably, further determining whether warning information requires persistent storage is a crucial step in optimizing data storage and management within the early warning system. By determining the warning level of warning information and combining it with pre-set persistence rules, the system intelligently determines which warning information warrants long-term preservation for subsequent analysis and decision-making. This level-based determination not only improves the relevance and efficiency of data storage, but also avoids unnecessary data redundancy and waste of storage space.

[0097] Specifically, based on the warning type of the warning information and the corresponding number of subscriptions to the warning information, batch extracting the warning information to be persisted from the memory and saving it to the relational database in an asynchronous writing manner includes:

[0098] Determine the corresponding warning information subscription list based on the warning type of the warning information;

[0099] Traversing the warning information subscription list, and extracting the warning information to be persisted from the memory according to the number of warning information subscriptions;

[0100] Using asynchronous writing, the extracted warning information is written into the relational database in batches according to the warning type, and during the writing process, the writing status and writing time corresponding to each warning information are recorded.

[0101] It can be understood that according to the warning type of the warning information and the corresponding number of warning information subscriptions, the warning information that needs to be persisted is extracted in batches from the memory and saved to the relational database using asynchronous writing. This step realizes the efficient storage and management of the warning information. By determining the warning type, the system can accurately find the corresponding warning information subscription list, and then extract the warning information that needs to be persisted according to the number of subscriptions. This extraction method based on warning type and subscription number not only improves the accuracy and efficiency of data extraction, but also ensures that only warning information that really needs to be persisted will be saved, avoiding data redundancy and waste of storage space. At the same time, the warning information is written into the relational database in batches using asynchronous writing, which further improves the speed and reliability of data storage. During the writing process, the system also records the write status and write time corresponding to each warning information, providing strong support for subsequent data analysis and decision-making.

[0102] Specifically, when clearing the corresponding invalid or expired data in the cache according to the current warning system time and the message identification ID, it includes:

[0103] Obtaining the current warning system time, and determining whether the current warning system time exceeds a preset cache clearing time threshold;

[0104] If it exceeds, the cached data in the memory will be traversed, and invalid or expired warning information will be identified and cleared according to the warning time and message ID;

[0105] After the cleanup is completed, the corresponding memory space is released.

[0106] It is understandable that clearing invalid or expired data from the cache based on the current warning system time and message ID is an important means for the warning system to maintain memory space and data validity. By continuously monitoring the current system time and comparing it with the preset cache clearing time threshold, the system can promptly detect and process warning information that has lost its timeliness or is no longer needed. This step not only ensures the effective use of memory space, but also avoids the impact of invalid data on system performance. During the clearing process, the system will accurately identify and clear invalid or expired warning information based on the warning time and message ID. After the clearing is completed, the system will release the corresponding memory space, providing more sufficient and efficient resources for subsequent data processing and analysis. This cache clearing strategy based on time and message ID not only improves the stability and efficiency of the warning system, but also provides users with more reliable and real-time warning services.

[0107] Specifically, periodically detecting a change in the monitoring time according to a preset time interval includes:

[0108] Set the monitoring daemon service timer to trigger the detection task at the preset time interval;

[0109] Each time a detection task is triggered, the monitoring time of the current early warning system is read;

[0110] Compare the read monitoring time with the monitoring time recorded during the last detection to determine whether the monitoring time has changed;

[0111] If the monitoring time changes, calculate the difference between the current monitoring time and the monitoring time recorded during the last detection, and record it as the monitoring time difference, and determine whether the running status of the message receiving service is normal based on the monitoring time difference;

[0112] If the monitoring time does not change, the periodic detection will continue at the preset time interval.

[0113] It is understandable that setting up a listening daemon service as the daemon thread of the message receiving service is a key measure to ensure that the early warning system can operate continuously and stably. Through the listening daemon service timer, the system can periodically detect the changes in the listening time at preset time intervals. This step not only realizes real-time monitoring of the running status of the message receiving service, but also can promptly discover and handle possible problems when the listening time changes. After calculating the listening time difference, the system will determine whether the running status of the message receiving service is normal based on preset rules or thresholds. If the listening time difference exceeds the normal range, the system may trigger a corresponding alarm or recovery process to ensure the reliability and stability of the early warning system. This periodic detection strategy based on the listening daemon service not only improves the response speed and fault recovery capability of the early warning system, but also provides users with a more stable and reliable early warning service.

[0114] Specifically, it is characterized in that, when judging whether the running status of the message receiving service is normal according to the monitoring time difference, it includes:

[0115] Compare the monitoring time difference with the monitoring time difference threshold, and determine whether the running status of the message receiving service is normal based on the comparison result.

[0116] If the monitoring time difference is greater than or equal to the monitoring time difference threshold, it is determined that the running state of the message receiving service is abnormal, and an alarm prompt is issued;

[0117] If the monitoring time difference is less than the monitoring time difference threshold, it is determined that the running state of the message receiving service is normal, and periodic detection is continued at a preset time interval.

[0118] It is understandable that by setting a listening time difference threshold and comparing the actual listening time difference with the threshold, the system can intelligently judge the operating status of the message receiving service. When the listening time difference exceeds the normal range, this may mean that there is a delay, interruption or other abnormality in the message receiving service. At this time, the system will immediately issue an alarm so that the administrator can discover and deal with these problems in time. This judgment method based on the listening time difference not only improves the intelligence level of the early warning system, but also ensures the stability and reliability of the message receiving service. If the listening time difference is within the normal range, the system will continue to perform periodic detection at preset time intervals to maintain continuous monitoring of the operating status of the message receiving service. This step not only realizes real-time evaluation and optimization of the performance of the early warning system, but also provides users with a more stable and efficient early warning service.

[0119] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or a combination of software and hardware embodiments. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0121] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A method for implementing large data volume early warning based on memory cache subscription and publishing, characterized in that: include: Determine an area to be monitored, deploy an early warning system within the area to be monitored, and activate the early warning system; During the initialization phase of the early warning system, the common basic data, early warning rules, the number of subscriptions to early warning information, and the monitoring time for information analysis and processing are pre-loaded and cached in the memory of the early warning system; An information listener is set to continuously monitor the real-time information flow of the monitored area; when the access of the information to be processed is detected, the information to be processed is combined into a key-value pair according to the message identification ID or the MD5 checksum of the message identification ID and its corresponding image file, and is cached in batches into the memory; Extracting the message to be analyzed from the memory, determining whether the message to be analyzed constitutes warning information according to the warning rule, and further determining whether the warning information needs to be persistently stored; When it is determined that the message to be analyzed belongs to warning information, classifying and grouping the warning information according to the warning type corresponding to the warning information; When it is determined that the warning information needs to be persisted, the warning information to be persisted is extracted in batches from the memory according to the warning type of the warning information and the corresponding warning information subscription quantity, and is saved in a relational database using an asynchronous write method; After completing the warning information saving operation, the number of warning information subscriptions corresponding to the warning type of the warning information in the memory is updated to zero, and the monitoring time is updated to the current warning system time; according to the current warning system time and the message identification ID, the corresponding invalid or expired data in the cache is cleared; A monitoring daemon service is set as a daemon thread of the message receiving service, and the change state of the monitoring time is periodically detected according to a preset time interval, and whether the running state of the message receiving service is normal is determined according to the change state.

2. The method for realizing large data volume early warning based on memory cache subscription and publication according to claim 1 is characterized in that: When the public basic data, warning rules, warning information subscription quantity and monitoring time used for information analysis and processing are pre-loaded and cached in the memory of the warning system, it includes: Obtaining initial values ​​corresponding to the public basic data, warning rules, warning information subscription quantity and monitoring time by reading a configuration file or a database; Loading the initial value into the memory cache area of ​​the early warning system and establishing a corresponding data structure; wherein the data structure is a key-value pair structure; During the loading process, the validity of the initial value is verified; If the verification fails, an error message is issued and the initialization process of the early warning system is terminated.

3. The method for realizing large data volume early warning based on memory cache subscription and publication according to claim 2 is characterized in that: When setting up an information listener to continuously monitor the real-time information flow of the area to be monitored, it includes: Receive the real-time information flow of the area to be monitored through the monitoring interface and parse the real-time information flow; According to the analysis result, the valid data in the real-time information stream is encapsulated in a preset format to form the information to be processed; The information to be processed is sent to the information listener for monitoring and processing.

4. The method for realizing large data volume early warning based on memory cache subscription and publication according to claim 3 is characterized in that: The information to be processed is formed into a key-value pair according to the message identification ID or the MD5 checksum of the message identification ID and its corresponding image file, and is cached in batches into the memory, including: Uniquely identify the received information to be processed and generate a message identification ID; If the information to be processed contains an image file, perform MD5 verification on the image file to generate a corresponding MD5 verification code; Classify the information to be processed into corresponding batches according to the message identification ID or a key-value pair combination of the message identification ID and the MD5 checksum; The classified information to be processed is cached in batches into the memory cache area of ​​the early warning system.

5. The method for realizing large data volume early warning based on memory cache subscription and publication according to claim 4 is characterized in that: When judging whether the message to be analyzed constitutes warning information according to the warning rules, it includes: Extracting key fields from the message to be analyzed and matching the key fields with warning rules; If the match is successful, it is determined that the message to be analyzed constitutes warning information; If the match is unsuccessful, the next message to be analyzed will be judged.

6. The method for implementing large data volume early warning based on memory cache subscription and publishing according to claim 5 is characterized in that: Further determining whether the warning information needs to be persisted includes: Determining a warning level corresponding to the warning information; Preset persistent storage rules, and determine whether the warning information needs to be persistently stored according to the persistent storage rules; If the warning level of the warning information meets the persistent storage rule, it is determined that the warning information needs to be persistently stored; If the warning level of the warning information does not comply with the persistent storage rule, it is determined that the warning information does not need to be persistently stored.

7. The method for implementing large data volume early warning based on memory cache subscription and publishing according to claim 6 is characterized in that: Extracting the warning information to be persisted in batches from the memory according to the warning type of the warning information and the corresponding number of subscriptions to the warning information, and saving the information to the relational database in an asynchronous writing manner, including: Determine the corresponding warning information subscription list based on the warning type of the warning information; Traversing the warning information subscription list, and extracting the warning information to be persisted from the memory according to the number of warning information subscriptions; Using asynchronous writing, the extracted warning information is written into the relational database in batches according to the warning type, and during the writing process, the writing status and writing time corresponding to each warning information are recorded.

8. The method for implementing large data volume early warning based on memory cache subscription and publishing according to claim 7 is characterized in that: When clearing the corresponding invalid or expired data in the cache according to the current warning system time and the message identification ID, it includes: Obtaining the current warning system time, and determining whether the current warning system time exceeds a preset cache clearing time threshold; If it exceeds, the cached data in the memory will be traversed, and invalid or expired warning information will be identified and cleared according to the warning time and message ID; After the cleanup is completed, the corresponding memory space is released.

9. The method for implementing large data volume early warning based on memory cache subscription and publishing according to claim 8 is characterized in that: When periodically detecting a change state of the monitoring time according to a preset time interval, the method includes: Set the monitoring daemon service timer to trigger the detection task at the preset time interval; Each time a detection task is triggered, the monitoring time of the current early warning system is read; Compare the read monitoring time with the monitoring time recorded during the last detection to determine whether the monitoring time has changed; If the monitoring time changes, calculate the difference between the current monitoring time and the monitoring time recorded during the last detection, and record it as the monitoring time difference, and determine whether the running status of the message receiving service is normal based on the monitoring time difference; If the monitoring time does not change, the periodic detection will continue at the preset time interval.

10. The method for implementing large data volume early warning based on memory cache subscription and publishing according to claim 9, characterized in that: When judging whether the running status of the message receiving service is normal according to the monitoring time difference, it includes: Compare the monitoring time difference with the monitoring time difference threshold, and determine whether the running status of the message receiving service is normal based on the comparison result. If the monitoring time difference is greater than or equal to the monitoring time difference threshold, it is determined that the running state of the message receiving service is abnormal, and an alarm prompt is issued; If the monitoring time difference is less than the monitoring time difference threshold, it is determined that the running state of the message receiving service is normal, and periodic detection is continued at a preset time interval.

Images