Multi-source heterogeneous event data real-time processing method and system

Abstract

The invention discloses a multi-source heterogeneous event data real-time processing method and system, and relates to the technical field of data processing. Aiming at the problems of efficient convergence, intelligent cleaning and real-time accurate distribution of multi-source heterogeneous data, the adopted scheme comprises the following steps of: automatically matching an analysis protocol of a heterogeneous data source through a multi-source adapter dynamic loading technology, and selecting a corresponding conversion script to convert analyzed data into a uniform format and store the analyzed data so as to realize convergence of multi-source heterogeneous event data; making a lightweight intelligent cleaning strategy, performing primary filtering on the converged event data through a rule engine of a primary filtering layer, and performing semantic correction on the filtered event data through a lightweight AI model of a semantic correction layer; and for the event data after semantic correction, extracting a structured semantic tag, carrying out multi-dimensional matching on the structured semantic tag and a city right and liability list, realizing intelligent event distribution in combination with a dynamic routing and distribution mechanism, and optimizing the right and liability list based on a processing result. According to the invention, the event convergence and distribution efficiency can be improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for real-time processing of multi-source heterogeneous event data. Background Technology

[0002] Urban event data comes from diverse sources (such as traffic monitoring, social media, public safety systems, sensor networks, and government systems), and suffers from problems such as heterogeneous formats, semantic differences, and high real-time requirements, making comprehensive analysis and collaborative handling difficult. Traditional data aggregation methods struggle to balance high throughput, low latency, and dynamic scalability. Existing technologies lack sufficient intelligence in filtering abnormal data and merging redundant data, and the polling or timed push distribution mechanisms result in high latency, reducing the speed of emergency response. Summary of the Invention

[0003] This invention addresses the needs and shortcomings of current technological development by providing a method and system for real-time processing of multi-source heterogeneous event data. It solves the problems of efficient aggregation, intelligent cleaning, and real-time accurate distribution of multi-source heterogeneous data, thereby improving the decision-making and response efficiency of urban operational events.

[0004] Firstly, the present invention provides a method for real-time processing of multi-source heterogeneous event data, and the technical solution adopted to solve the above-mentioned technical problems is as follows:

[0005] A method for real-time processing of multi-source heterogeneous event data includes the following steps:

[0006] S1. Through multi-source adapter dynamic loading technology, the parsing protocol of heterogeneous data sources is automatically matched, the corresponding conversion script is selected, the parsed data is converted into a unified format and stored, and the aggregation of multi-source heterogeneous event data is realized.

[0007] S2. Develop a lightweight intelligent cleaning strategy. Use the rule engine of the primary filtering layer to perform primary filtering on the event data output in step S1. Use the lightweight AI model TinyBERT of the semantic correction layer to perform semantic correction on the event data after primary filtering. Then push it to the next process in a unified manner.

[0008] S3. For the event data output in step S2, extract structured semantic tags and match them with the city's list of powers and responsibilities in multiple dimensions. Combine dynamic routing and distribution mechanisms to achieve intelligent event distribution, and optimize the list of powers and responsibilities based on the processing results.

[0009] Optionally, step S1 specifically includes:

[0010] S1.1 Establish a connection based on the pre-configured or dynamically discovered address, access the multi-source raw data stream from traffic cameras, urban management event data and IoT devices, and maintain the long-term connection status of the source data;

[0011] S1.2 Determine the transmission protocol type of the access source data: a) If it is HTTP, MQTT or RTSP protocol, directly call the protocol parsing engine to parse the data according to the specification, and trigger the exception handling mechanism to discard invalid data packets and trigger retransmission; b) If it is an unknown protocol, load the corresponding protocol template that was developed in advance or dynamically uploaded from the plugin repository, and then call the protocol parsing engine to parse the data, and trigger the exception handling mechanism to discard invalid data packets and trigger retransmission.

[0012] S1.3 After completing the parsing of the raw data, further determine the format of the event data: a) If it is in JSON, XML or CSV format, convert it into a unified intermediate format as standardized data output; b) If it is in binary or hexadecimal format, call the user-defined script to parse and convert it into a unified intermediate format.

[0013] S1.4. Inject metadata into the unified intermediate format data and attach context information. The context information includes at least one of the following: data event source, protocol type, integrity score, credibility score, event classification, and event priority.

[0014] Further optionally, step S2 specifically includes:

[0015] S2.1 For the aggregated multi-source heterogeneous event data, the rule engine of the primary filtering layer first performs primary filtering, completes format validation, field integrity check and range check operations, discards null values ​​and illogical data that are out of range, marks data that lacks key information as low confidence data, and then judges whether the data after primary filtering is valid. If it is valid, it enters the next process; otherwise, it is discarded or marked as abnormal and recorded in the system log.

[0016] S2.2 Confirm that valid data enters the semantic correction layer. The lightweight AI model TinyBERT on the semantic correction layer standardizes the geographic location information in the event data into GeoHash format, converts the time information in the event data into standard UTC timestamps, and eliminates semantic ambiguity in the event data, distinguishing between real events and metaphorical expressions.

[0017] Further optionally, step S2 may also include:

[0018] S2.3 After semantic correction, the data enters the dynamic optimization layer, which first determines whether dynamic adjustment needs to be triggered:

[0019] a) If no dynamic processing is required, the event data will be directly pushed to subsequent modules.

[0020] b) If dynamic processing is required, the parameters of the lightweight AI model TinyBERT are first incrementally updated through the online learning module and the rule priority is adjusted. Then, the updated event data is fed back to the storage module, and the event data is fed back to the primary filtering layer in a closed loop to re-enter the processing flow. The above process is repeated until no dynamic adjustment is required. Finally, the event data is output and uniformly pushed to the next process.

[0021] Further optionally, step S3 specifically includes:

[0022] S3.1. The event data output in step S2 is processed sequentially through data preprocessing, key information identification, label classification standardization, context association enhancement, and label storage index optimization to extract structured labels in order to achieve data standardization and provide basic support for subsequent matching of responsibility lists and intelligent decision-making.

[0023] S3.2. Multi-dimensional matching based on extracted semantic tags and a list of rights and responsibilities database: Match the department corresponding to the event type through Elasticsearch, verify the GeoHash prefix of the event's geographical jurisdiction, check the matching degree between the department's processing authority and the event priority, use the Neo4j graph database and rule engine to handle conflicts and trigger cross-departmental collaboration, and dynamically adjust the event priority based on real-time data.

[0024] S3.3 After matching is completed, a dynamic routing and distribution mechanism is triggered. Subscription rules are generated through the tag matching engine, bound to department topics, and high-priority events are inserted into dedicated partitions to ensure at least one delivery. Unresponsive events are transferred to the manual processing queue.

[0025] S3.4 After the department provides feedback on the processing results, follow-up processing will be carried out on the incident. If a certain type of incident is frequently transferred to manual processing or is mistakenly sent to a specific region, the rule self-learning algorithm will be triggered to optimize the list of responsibilities and powers, dynamically adjust the jurisdiction of GeoHash, and continuously improve the accuracy of distribution.

[0026] Secondly, this invention provides a real-time processing system for multi-source heterogeneous event data, and the technical solution adopted to solve the above-mentioned technical problems is as follows:

[0027] A real-time processing system for multi-source heterogeneous event data, comprising:

[0028] The multi-source adaptation dynamic loading module supports multiple data transmission protocols and formats, automatically matches the parsing protocol of the accessed heterogeneous data source, selects the corresponding conversion script to convert the parsed data into a unified format and stores it, thereby realizing the aggregation of multi-source heterogeneous event data;

[0029] The lightweight intelligent cleaning module is used to perform primary filtering on the aggregated event data through the rule engine of the primary filtering layer, and to perform semantic correction on the event data after primary filtering through the lightweight AI model TinyBERT of the semantic correction layer, and then push it to the downstream module in a unified manner.

[0030] The event intelligent distribution module receives event data output from the lightweight intelligent cleaning module, extracts structured semantic tags and performs multi-dimensional matching with the city's list of powers and responsibilities, combines dynamic routing and distribution mechanisms to achieve intelligent event distribution, and optimizes the list of powers and responsibilities based on the processing results.

[0031] Optionally, the multi-source adaptation dynamic loading module involved specifically includes:

[0032] The data access unit is used to establish a connection based on a pre-configured or dynamically discovered address, access multi-source raw data streams from traffic cameras, urban management event data and IoT devices, and maintain a long-term connection state for the source data.

[0033] The protocol judgment unit is used to determine the transmission protocol type of the access source data: a) If it is HTTP, MQTT or RTSP protocol, the protocol parsing engine is directly called to parse the data according to the specification, and the exception handling mechanism is triggered at the same time to discard invalid data packets and trigger retransmission; b) If it is an unknown protocol, the corresponding protocol template developed in advance or dynamically uploaded is loaded from the plugin repository, and then the protocol parsing engine is called to parse the data, and the exception handling mechanism is triggered at the same time to discard invalid data packets and trigger retransmission.

[0034] The format judgment unit is used to further judge the event data format: a) If it is in JSON, XML or CSV format, it is converted into a unified intermediate format as standardized data output; b) If it is in binary or hexadecimal format, the user-defined script is called to parse and convert it into a unified intermediate format.

[0035] The information injection unit is used to inject metadata into data in a unified intermediate format and attach context information. The context information includes at least one of the following: data event source, protocol type, integrity score, credibility score, event classification, and event priority.

[0036] Further, optionally, the lightweight intelligent cleaning module involved specifically includes:

[0037] The primary filtering layer is used to perform primary filtering on aggregated multi-source heterogeneous event data. The rule engine first performs primary filtering, completes format validation, field integrity checks and range checks, discards null values ​​and illogical data that are out of range, and marks data that lacks key information as low-confidence data. Then it judges whether the data after primary filtering is valid. If it is valid, it enters the semantic correction layer; otherwise, it is discarded or marked as abnormal and recorded in the system log.

[0038] The semantic correction layer is used to standardize the geolocation information in the event data into GeoHash format using the lightweight AI model TinyBERT, convert the time information in the event data into standard UTC timestamps, and eliminate semantic ambiguity in the event data, distinguishing between real events and metaphorical expressions.

[0039] Further, optionally, the lightweight intelligent cleaning module also includes:

[0040] The dynamic optimization layer is used to first determine whether dynamic adjustments need to be triggered:

[0041] a) If no dynamic processing is required, the event data will be directly pushed to subsequent modules.

[0042] b) If dynamic processing is required, the parameters of the lightweight AI model TinyBERT are first incrementally updated through the online learning unit and the rule priority is adjusted. Then, the updated event data is fed back to the storage unit, and the event data is fed back to the primary filtering layer in a closed loop to re-enter the processing flow. The above process is repeated until no dynamic adjustment is required. Finally, the event data is output and uniformly pushed to the subsequent modules.

[0043] Optionally, the event intelligent distribution module involved specifically includes:

[0044] The data standardization unit is used to process the event data output by the lightweight intelligent cleaning module. It extracts structured tags to achieve data standardization through data preprocessing, key information identification, tag classification standardization, context association enhancement, and tag storage index optimization, providing basic support for subsequent matching of responsibility lists and intelligent decision-making.

[0045] The multi-dimensional matching unit is used to perform multi-dimensional matching based on the extracted semantic tags and the list of rights and responsibilities database: matching the department corresponding to the event type through Elasticsearch, verifying the GeoHash prefix of the geographical jurisdiction of the event, checking the matching degree between the department's processing authority and the event priority, handling conflicts and triggering cross-departmental collaboration with the help of Neo4j graph database and rule engine, and dynamically adjusting the event priority according to real-time data.

[0046] The matching trigger unit is used to trigger the dynamic routing and distribution mechanism after the matching is completed. It generates subscription rules through the tag matching engine, binds them to department topics, inserts high-priority events into dedicated partitions, ensures at least one delivery, and transfers unresponsive events to the manual processing queue.

[0047] The self-learning optimization unit is used to process events after the department provides feedback on the processing results. If a certain type of event is frequently transferred to manual processing or is mistakenly sent to a region, the self-learning algorithm is triggered to optimize the list of responsibilities and powers, dynamically adjust the jurisdiction of GeoHash, and continuously improve the accuracy of distribution.

[0048] The real-time processing method and system for multi-source heterogeneous event data of the present invention has the following advantages compared with the prior art:

[0049] 1. This invention achieves efficient aggregation, intelligent cleaning, and real-time distribution of multi-source heterogeneous data through dynamic adapters, intelligent cleaning based on rule engines and lightweight AI models, and event-driven distribution technology based on event semantic tags and urban power and responsibility lists. This significantly improves the city's event monitoring and emergency response capabilities, solves the problems of low efficiency and delayed response in traditional urban management models, improves the efficiency of urban emergency response and handling, provides core support for sustainable urban development, promotes the intelligent transformation of urban management, and meets citizens' pursuit of high-quality urban services.

[0050] 2. This invention constructs a closed-loop system for urban event data through three core technologies: multi-source dynamic access, lightweight real-time processing, and precise intelligent distribution. This system achieves a leapfrog upgrade from data fragmentation to real-time collaboration, providing a standardized and scalable solution for event monitoring, emergency response, and long-term governance in smart cities.

[0051] 3. This invention can meet the access needs of multi-source heterogeneous data, including government data, urban IoT, network data, and surveillance video. Through dynamic plug-in adapters, it supports hot loading of protocol templates, enabling non-stop expansion of data sources, greatly shortening the adaptation time for new data sources, and solving the problem of long development cycles for custom data sources when using hard-coded adapters. Through scripts, it dynamically extends the original data protocol parsing logic, supporting the parsing of multiple file formats, including binary and private protocols, solving the problem that traditional modes only support custom protocols and require additional customization for private protocols.

[0052] 4. This invention employs a rule engine and the lightweight AI model TinyBERT, which can meet the requirements of low latency, intelligent cleaning, and adaptive optimization of cleaning strategies for event data. It solves the problem that relying solely on rule engines in the traditional case cannot handle semantic ambiguity, greatly improving the intelligence of cleaning and the credibility of the cleaned data. The application of the online learning mechanism also enables the cleaning strategy to continuously iterate, optimize, and upgrade as the event data distribution changes, solving the pain point that the cleaning strategy can only be upgraded manually in the traditional model, making the cleaning strategy more in line with the actual situation.

[0053] 5. This invention, through semantic tagging and responsibility list matching, can meet the needs of intelligent routing and accurate distribution of event data, solving the problem of high misclassification rate in traditional models based on manual sorting or simple rule distribution; it uses an event-driven distribution mode to replace the traditional polling or timed push mode, shortening the push delay and greatly improving the speed of emergency response. Attached Figure Description

[0054] Appendix Figure 1 This is a flowchart of the method according to Embodiment 1 of the present invention;

[0055] Appendix Figure 2 This is a flowchart of step S1 in Embodiment 1 of the present invention;

[0056] Appendix Figure 3 This is a flowchart of step S2 in Embodiment 1 of the present invention;

[0057] Appendix Figure 4 This is a flowchart of step S3 in Embodiment 1 of the present invention;

[0058] Appendix Figure 5 This is a module connection block diagram of Embodiment 2 of the present invention. Detailed Implementation

[0059] To make the technical solution, the technical problem solved, and the technical effect of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments.

[0060] Example 1:

[0061] Combined with appendix Figure 1 This embodiment proposes a real-time processing method for multi-source heterogeneous event data, which includes the following steps:

[0062] S1. Through multi-source adapter dynamic loading technology, the parsing protocol of heterogeneous data sources is automatically matched, the corresponding conversion script is selected, the parsed data is converted into a unified format and stored, and the aggregation of multi-source heterogeneous event data is realized.

[0063] Combined with appendix Figure 2 This process specifically includes:

[0064] S1.1 Establish a connection based on a pre-configured or dynamically discovered address (such as IP / URL), access multi-source raw data streams from traffic cameras, urban management event data and IoT devices, and maintain a long-term connection state for the source data;

[0065] S1.2 Determine the transmission protocol type of the access source data: a) If it is HTTP, MQTT or RTSP protocol, directly call the protocol parsing engine to parse the data according to the specification, and trigger the exception handling mechanism to discard invalid data packets and trigger retransmission; b) If it is an unknown protocol, load the corresponding protocol template that was developed in advance or dynamically uploaded from the plugin repository, and then call the protocol parsing engine to parse the data, and trigger the exception handling mechanism to discard invalid data packets and trigger retransmission.

[0066] S1.3 After completing the parsing of the raw data, further determine the format of the event data: a) If it is in JSON, XML or CSV format, convert it into a unified intermediate format as standardized data output; b) If it is in binary or hexadecimal format, call the user-defined script to parse and convert it into a unified intermediate format.

[0067] S1.4. Inject metadata into the unified intermediate format data and attach context information. The context information includes at least one of the following: data event source, protocol type, integrity score, credibility score, event classification, and event priority. This can enhance the interpretability, traceability, and efficiency of subsequent processing of the data.

[0068] S2. Develop a lightweight intelligent cleaning strategy. Use the rule engine of the primary filtering layer to perform primary filtering on the event data output in step S1. Use the lightweight AI model TinyBERT of the semantic correction layer to perform semantic correction on the event data after primary filtering. Then push the data to the next process in a unified manner.

[0069] Combined with appendix Figure 3 This process specifically includes:

[0070] S2.1 For the aggregated multi-source heterogeneous event data, the rule engine of the primary filtering layer first performs primary filtering, completes format validation, field integrity check and range check operations, discards null values ​​and illogical data that are out of range (such as latitude and longitude exceeding the city boundary), and marks data that lacks key information as low confidence data. Then it is determined whether the data after primary filtering is valid. If it is valid, it enters the next process. Otherwise, it is discarded or marked as abnormal and recorded in the system log.

[0071] S2.2 Confirm that valid data enters the semantic correction layer. The lightweight AI model TinyBERT on the semantic correction layer standardizes the geographic location information in the event data into GeoHash format, converts the time information in the event data into standard UTC timestamps, and eliminates semantic ambiguity in the event data, distinguishing between real events and metaphorical expressions.

[0072] S2.3 After semantic correction, the data enters the dynamic optimization layer, which first determines whether dynamic adjustment needs to be triggered:

[0073] a) If no dynamic processing is required, the event data will be directly pushed to subsequent modules.

[0074] b) If dynamic processing is required, the parameters of the lightweight AI model TinyBERT are first incrementally updated through the online learning module and the rule priority is adjusted. Then, the updated event data is fed back to the storage module, and the event data is fed back to the primary filtering layer in a closed loop to re-enter the processing flow. The above process is repeated until no dynamic adjustment is required. Finally, the event data is output and uniformly pushed to the next process.

[0075] S3. For the event data output in step S2, extract structured semantic tags and match them with the city's list of powers and responsibilities in multiple dimensions. Combine dynamic routing and distribution mechanisms to achieve intelligent event distribution, and optimize the list of powers and responsibilities based on the processing results.

[0076] Combined with appendix Figure 4 This process specifically includes:

[0077] S3.1. The event data output in step S2 is processed sequentially through data preprocessing, key information identification, label classification standardization, context association enhancement, and label storage index optimization to extract structured labels to achieve data standardization, providing basic support for subsequent matching of responsibility lists and intelligent decision-making.

[0078] S3.2. Perform multi-dimensional matching based on the extracted semantic tags and the rights and responsibilities list database:

[0079] Event type matching: Traverse the "Department-Event Type" list of the responsibility list and use Elasticsearch to match the department corresponding to the event type;

[0080] Geographic jurisdiction verification: The GeoHash of the event location is compared with the GeoHash prefix of the department's jurisdiction. This process supports multi-level matching at the municipal level, district level, and town / street level to ensure accurate matching.

[0081] Priority and permission verification: Real-time query of the department status table to check whether the event priority is within the department's processing authority. For example, "low priority" events may only be handled by district-level departments. Verify whether the number of events currently being processed by the department exceeds the threshold.

[0082] Conflict resolution and multi-departmental collaboration: Leveraging Neo4j graph database queries and rule engine decision tables, the system enables the confirmation of responsible departments and the triggering of cross-departmental collaborative tasks.

[0083] It should be noted that, considering that the priority of events can be affected by real-time data, a dynamic priority adjustment strategy has been added during the matching process of the list of responsibilities. The priority of events is dynamically increased based on real-time data, and higher-level regulatory departments are automatically added to high-priority events.

[0084] S3.3 After matching is completed, a dynamic routing and distribution mechanism is triggered. Subscription rules are generated through the tag matching engine, bound to department topics, and high-priority events are inserted into dedicated partitions to ensure at least one delivery. Unresponsive events are transferred to the manual processing queue.

[0085] S3.4 After the department completes the consumption, it provides feedback on the processing results and conducts follow-up processing on the event. If a certain type of event is frequently transferred to manual processing or is mistakenly sent to a specific region, the rule self-learning algorithm is triggered to optimize the list of responsibilities and duties, dynamically adjust the jurisdiction of GeoHash, and continuously improve the accuracy of distribution.

[0086] Example 2:

[0087] Combined with appendix Figure 5 This embodiment proposes a real-time processing system for multi-source heterogeneous event data, which includes:

[0088] The multi-source adaptation dynamic loading module supports multiple data transmission protocols and formats, automatically matches the parsing protocol of the accessed heterogeneous data source, selects the corresponding conversion script to convert the parsed data into a unified format and stores it, thereby realizing the aggregation of multi-source heterogeneous event data;

[0089] The lightweight intelligent cleaning module is used to perform primary filtering on the aggregated event data through the rule engine of the primary filtering layer, and to perform semantic correction on the event data after primary filtering through the lightweight AI model TinyBERT of the semantic correction layer, and then push it to the downstream module in a unified manner.

[0090] The event intelligent distribution module receives event data output from the lightweight intelligent cleaning module, extracts structured semantic tags and performs multi-dimensional matching with the city's list of powers and responsibilities, combines dynamic routing and distribution mechanisms to achieve intelligent event distribution, and optimizes the list of powers and responsibilities based on the processing results.

[0091] In this embodiment, the multi-source adaptation dynamic loading module specifically includes:

[0092] The data access unit is used to establish a connection based on a pre-configured or dynamically discovered address, access multi-source raw data streams from traffic cameras, urban management event data and IoT devices, and maintain a long-term connection state for the source data.

[0093] The protocol judgment unit is used to determine the transmission protocol type of the access source data: a) If it is HTTP, MQTT or RTSP protocol, the protocol parsing engine is directly called to parse the data according to the specification, and the exception handling mechanism is triggered at the same time to discard invalid data packets and trigger retransmission; b) If it is an unknown protocol, the corresponding protocol template developed in advance or dynamically uploaded is loaded from the plugin repository, and then the protocol parsing engine is called to parse the data, and the exception handling mechanism is triggered at the same time to discard invalid data packets and trigger retransmission.

[0094] The format judgment unit is used to further judge the event data format: a) If it is in JSON, XML or CSV format, it is converted into a unified intermediate format as standardized data output; b) If it is in binary or hexadecimal format, the user-defined script is called to parse and convert it into a unified intermediate format.

[0095] The information injection unit is used to inject metadata into data in a unified intermediate format and attach context information. The context information includes at least one of the following: data event source, protocol type, integrity score, credibility score, event classification, and event priority.

[0096] In this embodiment, the lightweight intelligent cleaning module specifically includes:

[0097] The primary filtering layer is used to perform primary filtering on aggregated multi-source heterogeneous event data. The rule engine first performs primary filtering, completes format validation, field integrity checks and range checks, discards null values ​​and illogical data that are out of range, and marks data that lacks key information as low-confidence data. Then it judges whether the data after primary filtering is valid. If it is valid, it enters the semantic correction layer; otherwise, it is discarded or marked as abnormal and recorded in the system log.

[0098] The semantic correction layer is used to standardize the geolocation information in the event data into GeoHash format using the lightweight AI model TinyBERT, convert the time information in the event data into standard UTC timestamps, and eliminate semantic ambiguity in the event data, distinguishing between real events and metaphorical expressions.

[0099] The dynamic optimization layer is used to first determine whether dynamic adjustments need to be triggered:

[0100] a) If no dynamic processing is required, the event data will be directly pushed to subsequent modules.

[0101] b) If dynamic processing is required, the parameters of the lightweight AI model TinyBERT are first incrementally updated through the online learning unit and the rule priority is adjusted. Then, the updated event data is fed back to the storage unit, and the event data is fed back to the primary filtering layer in a closed loop to re-enter the processing flow. The above process is repeated until no dynamic adjustment is required. Finally, the event data is output and uniformly pushed to the subsequent modules.

[0102] In this embodiment, the event intelligent distribution module specifically includes:

[0103] The data standardization unit is used to process the event data output by the lightweight intelligent cleaning module. It extracts structured tags to achieve data standardization through data preprocessing, key information identification, tag classification standardization, context association enhancement, and tag storage index optimization, providing basic support for subsequent matching of responsibility lists and intelligent decision-making.

[0104] The multi-dimensional matching unit is used to perform multi-dimensional matching based on the extracted semantic tags and the list of rights and responsibilities database: matching the department corresponding to the event type through Elasticsearch, verifying the GeoHash prefix of the geographical jurisdiction of the event, checking the matching degree between the department's processing authority and the event priority, handling conflicts and triggering cross-departmental collaboration with the help of Neo4j graph database and rule engine, and dynamically adjusting the event priority according to real-time data.

[0105] The matching trigger unit is used to trigger the dynamic routing and distribution mechanism after the matching is completed. It generates subscription rules through the tag matching engine, binds them to department topics, inserts high-priority events into dedicated partitions, ensures at least one delivery, and transfers unresponsive events to the manual processing queue.

[0106] The self-learning optimization unit is used to process events after the department provides feedback on the processing results. If a certain type of event is frequently transferred to manual processing or is mistakenly sent to a region, the self-learning algorithm is triggered to optimize the list of responsibilities and powers, dynamically adjust the jurisdiction of GeoHash, and continuously improve the accuracy of distribution.

[0107] In summary, the multi-source heterogeneous event data real-time processing method and system of the present invention can solve the problems of data format standardization, dynamic cleaning, efficient fusion and real-time distribution, and improve the efficiency of event aggregation and distribution.

[0108] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for real-time processing of multi-source heterogeneous event data, characterized in that, Includes the following steps: S1. Through multi-source adapter dynamic loading technology, the parsing protocol of heterogeneous data sources is automatically matched, the corresponding conversion script is selected, the parsed data is converted into a unified format and stored, and the aggregation of multi-source heterogeneous event data is realized. S2. Develop a lightweight intelligent cleaning strategy. Use the rule engine of the primary filtering layer to perform primary filtering on the event data output in step S1. Use the lightweight AI model TinyBERT of the semantic correction layer to perform semantic correction on the event data after primary filtering. Then push it to the next process in a unified manner. S3. For the event data output in step S2, extract structured semantic tags and match them with the city's list of powers and responsibilities in multiple dimensions. Combine dynamic routing and distribution mechanisms to achieve intelligent event distribution, and optimize the list of powers and responsibilities based on the processing results.

2. The method for real-time processing of multi-source heterogeneous event data according to claim 1, characterized in that, Step S1 specifically includes: S1.1 Establish a connection based on the pre-configured or dynamically discovered address, access the multi-source raw data stream from traffic cameras, urban management event data and IoT devices, and maintain the long-term connection status of the source data; S1.2 Determine the transmission protocol type of the access source data: a) If it is HTTP, MQTT or RTSP protocol, directly call the protocol parsing engine to parse the data according to the specification, and trigger the exception handling mechanism to discard invalid data packets and trigger retransmission; b) If it is an unknown protocol, load the corresponding protocol template that was developed in advance or dynamically uploaded from the plugin repository, and then call the protocol parsing engine to parse the data, and trigger the exception handling mechanism to discard invalid data packets and trigger retransmission. S1.3 After completing the parsing of the raw data, further determine the format of the event data: a) If it is in JSON, XML or CSV format, convert it into a unified intermediate format as standardized data output; b) If it is in binary or hexadecimal format, call the user-defined script to parse and convert it into a unified intermediate format. S1.

4. Inject metadata into the unified intermediate format data and attach context information. The context information includes at least one of the following: data event source, protocol type, integrity score, credibility score, event classification, and event priority.

3. The method for real-time processing of multi-source heterogeneous event data according to claim 2, characterized in that, Step S2 specifically includes: S2.1 For the aggregated multi-source heterogeneous event data, the rule engine of the primary filtering layer first performs primary filtering, completes format validation, field integrity check and range check operations, discards null values ​​and illogical data that are out of range, marks data that lacks key information as low confidence data, and then judges whether the data after primary filtering is valid. If it is valid, it enters the next process; otherwise, it is discarded or marked as abnormal and recorded in the system log. S2.2 Confirm that valid data enters the semantic correction layer. The lightweight AI model TinyBERT on the semantic correction layer standardizes the geographic location information in the event data into GeoHash format, converts the time information in the event data into standard UTC timestamps, and eliminates semantic ambiguity in the event data, distinguishing between real events and metaphorical expressions.

4. The method for real-time processing of multi-source heterogeneous event data according to claim 3, characterized in that, Step S2 further includes: S2.3 After semantic correction, the data enters the dynamic optimization layer, which first determines whether dynamic adjustment needs to be triggered: a) If no dynamic processing is required, the event data will be directly pushed to subsequent modules. b) If dynamic processing is required, the parameters of the lightweight AI model TinyBERT are first incrementally updated through the online learning module and the rule priority is adjusted. Then, the updated event data is fed back to the storage module, and the event data is fed back to the primary filtering layer in a closed loop to re-enter the processing flow. The above process is repeated until no dynamic adjustment is required. Finally, the event data is output and uniformly pushed to the next process.

5. The method for real-time processing of multi-source heterogeneous event data according to claim 4, characterized in that, Step S3 specifically includes: S3.

1. The event data output in step S2 is processed sequentially through data preprocessing, key information identification, label classification standardization, context association enhancement, and label storage index optimization to extract structured labels in order to achieve data standardization and provide basic support for subsequent matching of responsibility lists and intelligent decision-making. S3.

2. Multi-dimensional matching based on extracted semantic tags and a list of rights and responsibilities database: Match the department corresponding to the event type through Elasticsearch, verify the GeoHash prefix of the event's geographical jurisdiction, check the matching degree between the department's processing authority and the event priority, use the Neo4j graph database and rule engine to handle conflicts and trigger cross-departmental collaboration, and dynamically adjust the event priority based on real-time data. S3.3 After matching is completed, a dynamic routing and distribution mechanism is triggered. Subscription rules are generated through the tag matching engine, bound to department topics, and high-priority events are inserted into dedicated partitions to ensure at least one delivery. Unresponsive events are transferred to the manual processing queue. S3.4 After the department provides feedback on the processing results, follow-up processing will be carried out on the incident. If a certain type of incident is frequently transferred to manual processing or is mistakenly sent to a specific region, the rule self-learning algorithm will be triggered to optimize the list of responsibilities and powers, dynamically adjust the jurisdiction of GeoHash, and continuously improve the accuracy of distribution.

6. A real-time processing system for multi-source heterogeneous event data, characterized in that, It includes: The multi-source adaptation dynamic loading module supports multiple data transmission protocols and formats, automatically matches the parsing protocol of the accessed heterogeneous data source, selects the corresponding conversion script to convert the parsed data into a unified format and stores it, thereby realizing the aggregation of multi-source heterogeneous event data; The lightweight intelligent cleaning module is used to perform primary filtering on the aggregated event data through the rule engine of the primary filtering layer, and to perform semantic correction on the event data after primary filtering through the lightweight AI model TinyBERT of the semantic correction layer, and then push it to the downstream module in a unified manner. The event intelligent distribution module receives event data output from the lightweight intelligent cleaning module, extracts structured semantic tags and performs multi-dimensional matching with the city's list of powers and responsibilities, combines dynamic routing and distribution mechanisms to achieve intelligent event distribution, and optimizes the list of powers and responsibilities based on the processing results.

7. A real-time processing system for multi-source heterogeneous event data according to claim 6, characterized in that, The multi-source adaptation dynamic loading module specifically includes: The data access unit is used to establish a connection based on a pre-configured or dynamically discovered address, access multi-source raw data streams from traffic cameras, urban management event data and IoT devices, and maintain a long-term connection state for the source data. The protocol judgment unit is used to determine the transmission protocol type of the access source data: a) If it is HTTP, MQTT or RTSP protocol, the protocol parsing engine is directly called to parse the data according to the specification, and the exception handling mechanism is triggered at the same time to discard invalid data packets and trigger retransmission; b) If it is an unknown protocol, the corresponding protocol template developed in advance or dynamically uploaded is loaded from the plugin repository, and then the protocol parsing engine is called to parse the data, and the exception handling mechanism is triggered at the same time to discard invalid data packets and trigger retransmission. The format judgment unit is used to further judge the event data format: a) If it is in JSON, XML or CSV format, it is converted into a unified intermediate format as standardized data output; b) If it is in binary or hexadecimal format, the user-defined script is called to parse and convert it into a unified intermediate format. The information injection unit is used to inject metadata into data in a unified intermediate format and attach context information. The context information includes at least one of the following: data event source, protocol type, integrity score, credibility score, event classification, and event priority.

8. A real-time processing system for multi-source heterogeneous event data according to claim 7, characterized in that, The lightweight intelligent cleaning module specifically includes: The primary filtering layer is used to perform primary filtering on aggregated multi-source heterogeneous event data. The rule engine first performs primary filtering, completes format validation, field integrity checks and range checks, discards null values ​​and illogical data that are out of range, and marks data that lacks key information as low-confidence data. Then it judges whether the data after primary filtering is valid. If it is valid, it enters the semantic correction layer; otherwise, it is discarded or marked as abnormal and recorded in the system log. The semantic correction layer is used to standardize the geolocation information in the event data into GeoHash format using the lightweight AI model TinyBERT, convert the time information in the event data into standard UTC timestamps, and eliminate semantic ambiguity in the event data, distinguishing between real events and metaphorical expressions.

9. A real-time processing system for multi-source heterogeneous event data according to claim 8, characterized in that, The lightweight intelligent cleaning module also includes: The dynamic optimization layer is used to first determine whether dynamic adjustments need to be triggered: a) If no dynamic processing is required, the event data will be directly pushed to subsequent modules. b) If dynamic processing is required, the parameters of the lightweight AI model TinyBERT are first incrementally updated through the online learning unit and the rule priority is adjusted. Then, the updated event data is fed back to the storage unit, and the event data is fed back to the primary filtering layer in a closed loop to re-enter the processing flow. The above process is repeated until no dynamic adjustment is required. Finally, the event data is output and uniformly pushed to the subsequent modules.

10. A real-time processing system for multi-source heterogeneous event data according to claim 9, characterized in that, The event intelligent distribution module specifically includes: The data standardization unit is used to process the event data output by the lightweight intelligent cleaning module. It extracts structured tags to achieve data standardization through data preprocessing, key information identification, tag classification standardization, context association enhancement, and tag storage index optimization, providing basic support for subsequent matching of responsibility lists and intelligent decision-making. The multi-dimensional matching unit is used to perform multi-dimensional matching based on the extracted semantic tags and the list of rights and responsibilities database: matching the department corresponding to the event type through Elasticsearch, verifying the GeoHash prefix of the geographical jurisdiction of the event, checking the matching degree between the department's processing authority and the event priority, handling conflicts and triggering cross-departmental collaboration with the help of Neo4j graph database and rule engine, and dynamically adjusting the event priority according to real-time data. The matching trigger unit is used to trigger the dynamic routing and distribution mechanism after the matching is completed. It generates subscription rules through the tag matching engine, binds them to department topics, inserts high-priority events into dedicated partitions, ensures at least one delivery, and transfers unresponsive events to the manual processing queue. The self-learning optimization unit is used to process events after the department provides feedback on the processing results. If a certain type of event is frequently transferred to manual processing or is mistakenly sent to a region, the self-learning algorithm is triggered to optimize the list of responsibilities and powers, dynamically adjust the jurisdiction of GeoHash, and continuously improve the accuracy of distribution.

Images