Multi-algorithm integrated data analysis and push system
The data parsing and push system integrating multiple algorithms enables intelligent parsing and efficient push of multi-source data, solving the problems of complexity and resource waste in the integration of multiple algorithms in existing technologies, and improving data processing efficiency and algorithm task stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING GUOJI SCI & TECH RES INST CO LTD
- Filing Date
- 2025-12-31
- Publication Date
- 2026-04-17
AI Technical Summary
Existing systems lack effective methods for integrating and calling multiple algorithms, leading to increased complexity, wasted resources, and low processing efficiency, making it difficult to achieve accurate parsing and delivery of multi-source data.
A multi-algorithm integrated data parsing and push system was designed, including an algorithm management module, a service gateway module, a task creation module, a data parsing module, and an event deduplication module. Through standardized algorithm package management, heartbeat signal monitoring, traffic governance, and event deduplication mechanisms, it achieves intelligent parsing and efficient push of multi-source data.
It improves algorithm compatibility and resource utilization, ensures stable operation of algorithm tasks, supports high-concurrency and high-real-time data parsing, eliminates duplicate events, reduces redundant processing, and improves data processing efficiency.
Smart Images
Figure CN121879870A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a data parsing and push system integrating multiple algorithms, belonging to the field of multi-source data processing and algorithm integration technology. Background Technology
[0002] In recent years, the rapid development of information technology has driven the widespread application of algorithms across various industries, especially in fields such as data analysis. However, with the increasing number and diversity of algorithms, existing systems often lack effective methods to adapt and manage them when calling them. This complicates algorithm integration and invocation, limiting their performance in practical applications. Furthermore, the continuous growth in data volume often leads to problems such as data redundancy, resource waste, and low processing efficiency. These issues not only affect data processing efficiency but also reduce the actual effectiveness of algorithm applications. Therefore, how to efficiently integrate and invoke multiple algorithms to achieve accurate parsing and delivery of multi-source data has become an urgent problem to be solved. Summary of the Invention
[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide a data parsing and push system integrating multiple algorithms to overcome the shortcomings of the existing technology. This system can integrate algorithms from multiple sources to achieve intelligent parsing of multi-source data.
[0004] To address the aforementioned technical problems, this invention discloses a multi-algorithm integrated data parsing and push system, comprising:
[0005] Algorithm management module 1 is used to manage various access algorithms, parse algorithms from different sources, and store them in the algorithm library in the form of standardized algorithm packages;
[0006] Service gateway module 2 is used to receive user requests, parse the requests, determine the type of request and the required data; it is also used to return responses.
[0007] Task creation module 3 is used to select algorithm tags from the algorithm library according to user requests, and determine the data, time and period for algorithm implementation to create algorithm tasks;
[0008] Data parsing module 4 is used to receive data collected by video, images or IoT sensors, bind it to the algorithm task, obtain the algorithm events generated by the algorithm task, and generate structured algorithm parsing results.
[0009] Furthermore, after the task creation module 3 creates the algorithm task, it adds the newly created algorithm task to the algorithm container, and the algorithm container periodically sends a heartbeat signal to the algorithm management module 1; the algorithm management module detects the running status of the algorithm task by monitoring the heartbeat signal.
[0010] Furthermore, the service gateway module 2 includes a request receiving module 21 and a request processing module 22;
[0011] The request receiving module 21 is used to receive user requests, and after determining that the request is a legitimate request, it parses the request to determine the type of request and the required data.
[0012] The request processing module 22 is used to obtain the algorithm parsing results and encapsulate them into response data to return to the user.
[0013] Furthermore, the service gateway module 2 also includes a traffic management component 23, which is used to implement traffic control and record the source IP, operation type, and response status of the received user requests.
[0014] Furthermore, the task creation module also includes drawing a recognition area; the recognition area is the range of data collected by the algorithm.
[0015] Furthermore, the data parsing module 4 performs structured processing on the algorithm tags, algorithm events, event occurrence times, and event data links to obtain the algorithm parsing results.
[0016] Furthermore, it also includes an event deduplication module 5, which performs deduplication processing on the algorithm parsing results. The deduplication processing steps are as follows:
[0017] Compare the event occurrence times in algorithm parsing results where the algorithm tags, algorithm events, and event data links are all the same. If the difference in event occurrence times is less than the preset push interval, they are determined to be the same event; otherwise, they are determined to be different events. Select one of the multiple algorithm parsing results determined to be the same event and store it, and the rest are determined to be duplicate events.
[0018] Furthermore, among the multiple algorithmic analysis results that are determined to be the same event, the one with the latest event occurrence time is stored, and the rest are determined to be duplicate events.
[0019] Furthermore, it also includes a result push module 6; the result push module runs on a timer, and for algorithm parsing results with a preset push time interval, if the difference between the event occurrence time and the current time exceeds the preset push time interval, the corresponding algorithm event is marked as ended.
[0020] Furthermore, the result push module adds the failed algorithm parsing results to the message queue and uses an exponential backoff strategy to retry. When the number of retries reaches a preset retry threshold, the failed algorithm parsing results are persisted to the exception queue.
[0021] Beneficial effects: The multi-algorithm integrated data parsing and push system disclosed in this invention has the following advantages:
[0022] 1. It can achieve unified access and standardized management of algorithms from multiple vendors, improving algorithm compatibility and resource utilization; it monitors the status of algorithm tasks through heartbeat signals and sets up an automatic restart mechanism to ensure the stable operation of algorithm tasks, which is suitable for high-concurrency and high-real-time data parsing scenarios such as intelligent transportation and public security.
[0023] 2. Supports automated parsing of video, image, and IoT data, accurately associating algorithm events with multi-source data, and generating structured event data packets based on the results of algorithm task execution;
[0024] 3. The event deduplication module adopts a time-interval driven deduplication mechanism to eliminate duplicate events and reduce redundant processing; it also promptly pushes the status of completed events to the user. Attached Figure Description
[0025] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.
[0026] Figure 1 This is a schematic diagram of the data parsing and push system with multi-algorithm integration disclosed in Embodiment 1 of the present invention;
[0027] Figure 2 This is a schematic diagram of the data parsing and push system with multi-algorithm integration disclosed in Embodiment 2 of the present invention;
[0028] Figure 3 This is a schematic diagram of the algorithm task in Example 2. Detailed Implementation
[0029] This invention discloses a data parsing and push system integrating multiple algorithms, such as... Figure 1 As shown, it includes:
[0030] Algorithm management module 1 is used to manage various access algorithms, parse algorithms from different sources, and store them in the algorithm library in the form of standardized algorithm packages;
[0031] Service gateway module 2 is used to receive user requests, parse the requests, determine the type of request and the required data; it is also used to return responses.
[0032] The service gateway module includes a request receiving module 21 and a request processing module 22;
[0033] The request receiving module 21 is used to receive user requests, and after determining that the request is a legitimate request, it parses the request to determine the type of request and the required data.
[0034] The request processing module 22 is used to obtain the algorithm parsing results and encapsulate them into response data to return to the user.
[0035] Task creation module 3 is used to select algorithm tags from the algorithm library according to user requests, and determine the data, time and period for algorithm implementation to create algorithm tasks;
[0036] After creating an algorithm task, the task creation module 3 adds the newly created task to the algorithm container. The algorithm container periodically sends a heartbeat signal to the algorithm management module. The algorithm management module monitors the heartbeat signal to detect the running status of the algorithm tasks. The algorithm management module marks algorithm tasks that fail to send a heartbeat signal within a timeout as abnormal, and abnormal algorithm tasks will be automatically restarted.
[0037] It also includes drawing a recognition area; the recognition area is the range of data collected by the algorithm.
[0038] Data parsing module 4 is used to receive data collected by video, images or IoT sensors, bind it to the algorithm task, obtain the algorithm events generated by the algorithm task, and generate structured algorithm parsing results.
[0039] The algorithm parsing results are obtained by structuring the algorithm tags, algorithm events, event occurrence times, and event data links; the algorithm parsing results are stored in the parsing result management module.
[0040] The service gateway module 2 also includes a traffic management component 23, which is used to implement traffic control and record the source IP, operation type, and response status of received user requests.
[0041] Example 1:
[0042] This example uses dynamic pedestrian flow statistics to illustrate the working principle of the multi-algorithm integrated data parsing and push system. Multiple vendors' algorithm server tools include dynamic pedestrian flow statistics algorithms. When these algorithms are integrated into the algorithm management module, the same algorithm parsing rules are set to ensure that algorithms from different sources can operate according to the same standards. Then, these algorithms are tagged and stored in a unified algorithm library as standardized algorithm packages. The algorithm library stores the algorithm's configuration information and records its metadata, such as algorithm identifiers and descriptions.
[0043] The request receiving module in the service gateway module receives user requests. User requests can be accessed via protocols such as HTTP / HTTPS and WebSocket. After determining that the request is legitimate, it is parsed to determine the request type and the required data. The multi-algorithm integrated data parsing and push system receives two user requests: dynamic pedestrian flow statistics at point A on the marathon route, and dynamic pedestrian flow statistics at point B at the stadium entrance. After parsing, the types of the two user requests and the required data are determined. The task creation module analyzes the algorithm server's usage and ultimately selects the currently available algorithms A and B. Algorithm Task A creates Algorithm Task A and Algorithm Task B. Algorithm Task A uses video data collected by a surveillance camera at point A, from 7:30 AM to 12:30 PM on [Date], with a 10-minute interval. The recognition area is the field of view of the surveillance camera at point A; that is, the number of people in the video image at point A every 10 minutes represents the dynamic pedestrian flow at point A. Algorithm Task B uses video data collected by a surveillance camera at point B, from 8:30 AM to 5:30 PM on [Date], with a 30-minute interval. The recognition area is the field of view of the surveillance camera at point B; that is, the number of people in the video image at point B every 30 minutes represents the dynamic pedestrian flow at point B. After successful creation, Algorithm Tasks A and B are added to an algorithm container. The algorithm container periodically sends heartbeat signals to the algorithm management module to report the running status of Algorithm Tasks A and B. The algorithm management module marks algorithm tasks that fail to send a heartbeat signal within a timeout as abnormal, and abnormal algorithm tasks will be automatically restarted.
[0044] The data parsing module receives video footage from the surveillance camera at point A and binds it to algorithm task A; it also receives video footage from the surveillance camera at point B and binds it to algorithm task B. Algorithm task A acquires dynamic pedestrian traffic data at point A every 10 minutes during its execution time. It then performs structured processing on the algorithm tags (dynamic pedestrian traffic statistics), algorithm events (dynamic pedestrian traffic statistics at point A), event occurrence time (current event), and event data links (current video data links) to obtain algorithm parsing result A. Algorithm parsing result A is encapsulated into response data by request processing module 22 and returned via a secure channel to the user who requested "Dynamic pedestrian traffic statistics at point A on the marathon route."
[0045] Similarly, every 30 minutes, the data parsing module generates algorithm parsing result B, which is then encapsulated as response data by the request processing module 22 and returned via a secure channel to the user corresponding to the "Dynamic Pedestrian Flow Statistics at Point B at the Stadium Entrance" request. For ease of post-event management, algorithm parsing results A and B are stored in the parsing result management module 41.
[0046] The number of concurrent user requests may be large. In order to prevent sudden traffic from overwhelming the backend service, the service gateway module 2 uses the traffic management component 23 to implement traffic control and records the source IP, operation type, and response status of the received user requests.
[0047] Example 2:
[0048] To avoid redundant processing of duplicate events, this embodiment adds an event deduplication module 5 to the multi-algorithm integrated data parsing and push system to deduplicat the algorithm parsing results, such as... Figure 2 As shown. The deduplication process is as follows:
[0049] The algorithm parsing results for algorithms with identical algorithm tags, algorithm events, and event data links compare the event occurrence times. If the difference in event occurrence times is less than a preset push interval, they are determined to be the same event; otherwise, they are determined to be different events. Among multiple algorithm parsing results determined to be the same event, one is selected and stored, and the rest are determined to be duplicate events. In this embodiment, the algorithm parsing results for the same event select the one with the most recent event occurrence time for storage, and the rest are determined to be duplicate events. Duplicate events are ignored and not returned to the user.
[0050] This embodiment uses the example of preventing motor vehicles from entering a protected waterway at night. The algorithm runs once per minute to detect whether any motor vehicles have entered the protected area via surveillance video. Figure 3 The preset push interval is 2 minutes; if the algorithm analysis results are as shown in the table below:
[0051] Algorithm analysis result sequence number Algorithm Tags Algorithm events Time of the incident 1 Motor vehicle area intrusion Vehicle intrusion 2025.11.21 00:01:00 2 Motor vehicle area intrusion Vehicle intrusion 2025.11.21 00:02:00 3 Motor vehicle area intrusion No vehicle intrusion 2025.11.21 00:03:00 4 Motor vehicle area intrusion No vehicle intrusion 2025.11.21 00:04:00 5 Motor vehicle area intrusion Vehicle intrusion 2025.11.21 00:05:00
[0052] According to the above deduplication process, if Algorithm Result 1 and Algorithm Result 2 are the same event, Algorithm Result 2 will be retained, while Algorithm Result 1 will be determined as a duplicate event and ignored, and will not be returned to the user; if the time difference between Algorithm Result 5 and Algorithm Result 2 is 3 minutes, which is greater than the preset push interval, then they are different events.
[0053] To determine the status of algorithm events, such as whether they have ended, this embodiment adds a result push module 6 to the multi-algorithm integrated data parsing and push system. The result push module runs periodically. For algorithm parsing results with a preset push time interval, if the difference between the event occurrence time and the current time exceeds the preset push time interval, the corresponding algorithm event is marked as ended. At 00:05:00 on November 21, 2025, the event corresponding to algorithm parsing result 2 was marked as ended. The status of the algorithm event is pushed by the result push module to the request processing module in the service gateway module, which then returns it to the user.
[0054] The result push module adds the failed algorithm parsing results to the message queue and uses an exponential backoff strategy for retries. When the number of retries reaches a preset retry threshold, the algorithm parsing results that still fail to push are persisted to the exception queue. In this embodiment, the retry threshold is set to 5.
[0055] This invention provides a method and approach for data parsing and push using multi-algorithm integration. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.
Claims
1. A data parsing and push system integrating multiple algorithms, characterized in that, include: The algorithm management module (1) is used to manage various access algorithms, parse algorithms from different sources, and store them in the algorithm library in the form of standardized algorithm packages; The service gateway module (2) is used to receive user requests, parse the requests, determine the type of the request and the required data; it is also used to return a response. The task creation module (3) is used to select algorithm tags from the algorithm library according to user requests, and determine the data, time and period of algorithm implementation to create algorithm tasks; The data parsing module (4) is used to receive data collected by video, image or IoT sensors, bind it to the algorithm task, obtain the algorithm events generated by the algorithm task, and generate structured algorithm parsing results.
2. The multi-algorithm integrated data parsing and push system according to claim 1, characterized in that, After the task creation module (3) creates the algorithm task, it adds the newly created algorithm task to the algorithm container. The algorithm container periodically sends a heartbeat signal to the algorithm management module (1). The algorithm management module detects the running status of the algorithm task by monitoring the heartbeat signal.
3. The multi-algorithm integrated data parsing and push system according to claim 1, characterized in that, The service gateway module (2) includes a request receiving module (21) and a request processing module (22). The request receiving module (21) is used to receive user requests, and after determining that the request is a legitimate request, it parses the request to determine the type of request and the required data. The request processing module (22) is used to obtain the algorithm parsing result and encapsulate it into response data to return to the user.
4. The multi-algorithm integrated data parsing and push system according to claim 3, characterized in that, The service gateway module (2) also includes a traffic management component (23), which is used to implement traffic control and record the source IP, operation type, and response status of the received user request.
5. The data parsing and push system integrating multiple algorithms according to claim 1, characterized in that, The task creation module also includes drawing a recognition area; the recognition area is the range of data collected by the algorithm.
6. The multi-algorithm integrated data parsing and push system according to claim 1, characterized in that, The data parsing module (4) performs structured processing on the algorithm tags, algorithm events, event occurrence time, and event data links to obtain the algorithm parsing results.
7. The multi-algorithm integrated data parsing and push system according to claim 1, characterized in that, It also includes an event deduplication module (5), which performs deduplication processing on the algorithm parsing results. The deduplication processing steps are as follows: Compare the event occurrence times in the algorithm parsing results where the algorithm tags, algorithm events, and event data links are all the same. If the difference in event occurrence times is less than the preset push interval, they are determined to be the same event; otherwise, they are determined to be different events. Select one of the multiple algorithm analysis results that are determined to be the same event and store it; the rest are determined to be duplicate events.
8. The multi-algorithm integrated data parsing and push system according to claim 7, characterized in that, Among multiple algorithm analysis results that are determined to be the same event, the one with the latest event occurrence time is stored, and the rest are determined to be duplicate events.
9. The multi-algorithm integrated data parsing and push system according to claim 7, characterized in that, It also includes a result push module (6); the result push module runs on a timer. For algorithm parsing results with a preset push time interval, if the difference between the event occurrence time and the current time exceeds the preset push time interval, the corresponding algorithm event is marked as ended.
10. The multi-algorithm integrated data parsing and push system according to claim 9, characterized in that, The result push module adds the failed algorithm parsing results to the message queue and uses an exponential backoff strategy to retry. When the number of retries reaches the preset retry threshold, the failed algorithm parsing results are persisted to the exception queue.