Massive conversation data aggregation analysis method, system, device and medium
By splitting the query time into multiple time slices and processing with the task management module, the problem of the existing technology being unable to support massive session data analysis within a long time range is solved, and efficient session data processing and real-time interactive improvement is achieved.
Patent Information
- Application Number
- CN202211684219.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-27
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-12-27
AI Technical Summary
The existing technology cannot effectively support the analysis of massive session data over a long period of time, resulting in system resource limitations, the analysis process time is too long, the interactivity is poor, and the long connections in the network cannot be fully restored.
By splitting the query time range into multiple fixed time slice subtasks, using message queue and task management modules for sharding processing, monitoring task status in real time, merging sharded data, and realizing group aggregation and integrity verification of session data , supports long-term large-scale session data analysis.
It realizes a complete analysis of massive conversation data over a long period of time, feedbacks intermediate results in real time, improves interactivity, and supports efficient conversation data processing.
Smart Images

Figure CN115987812B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a method, system, device and medium for aggregate analysis of massive conversation data. Background Art
[0002] In the field of network performance management, it is a common demand to aggregate time series data within a certain time range into session data in order to gain insight into and analyze sessions in the network. The processing power of current network performance management equipment is increasing day by day, with the number of sessions per minute reaching millions, and time series data reaching tens of millions per minute. At this time, users need to aggregate session data from the massive time series data within the time range where the problem occurs, and locate the session with the problem. The general implementation method is to aggregate the time series data within the entire query time period into session data during the query. This method requires limiting the query time range, such as limiting the range to one minute, due to the large amount of data and the inability of computer memory to meet the demand. This makes the troubleshooting inefficient, and due to the short time range, it is impossible to effectively restore long connections in the network. In addition, due to the large amount of time series data, aggregation takes a long time, users have to wait for results for a long time, and the interactivity is poor.
[0003] In summary, the prior art has the following defects: 1) Due to system resource limitations, it is impossible to support long-term analysis of massive conversation data, resulting in incomplete conversations; 2) The analysis process takes too long, and the results can only be fed back after the entire query is completed, resulting in poor interactivity. Summary of the invention
[0004] In view of the defects in the prior art, the present invention provides a method, system, device and medium for aggregating and analyzing massive conversation data.
[0005] According to a method, system, device and medium for aggregate analysis of massive conversation data provided by the present invention, the scheme is as follows:
[0006] In a first aspect, a method for aggregating and analyzing massive conversation data is provided, the method comprising:
[0007] Step S1: Send a query request through the front end, the input module receives and verifies the request parameters, and returns a verification error when necessary parameters are missing. After verification, the request parameters are sent to the session analysis management module;
[0008] Step S2: receiving request parameters through the session analysis management module, managing the entire query life cycle, slicing the task, and sending the slicing task to the session task management module;
[0009] Step S3: The session data task management module receives the subtask through the message queue, executes the subtask after receiving it, and correctly marks the subtask status;
[0010] Step S4: When the session data task is completed, the session analysis management module schedules the session table query module to generate segmented session data, merge multiple segments, and generate a preview or final result.
[0011] Preferably, the step S2 comprises:
[0012] Step S2.1: Split the subtasks, in ascending order of time, split the time range into multiple session data generation subtasks with fixed time slices, and set the first time slice to 10 seconds;
[0013] Step S2.2: After splitting the subtasks, the subtasks are sent to the session data task management module in sequence. After sending, the subtask status is polled regularly. At this time, the subtask status is divided into scheduling, running, completed, and abnormally ended. When the task status is marked as scheduling or running, the next polling cycle continues to poll until the task status is marked as completed or abnormally ended;
[0014] Step S2.3: The session table query module queries the session data of the subtask and performs filtering, sorting and paging.
[0015] Preferably, in step S2.2, for tasks that are abnormally terminated, there is no need to trigger a preview; for tasks that are completed, a preview action is triggered, and a request for a subtask preview is sent to the session table query module.
[0016] Preferably, step S2.3 further includes: if the previous sharding has been completed, merging the current page data and the previous page data, sorting and paging, and then outputting the preview result; if the current page is already the last page, the query ends.
[0017] Preferably, step S3 comprises:
[0018] Step S3.1: Determine whether the concurrency limit is exceeded. If so, insert the task to the front of the cache queue, sleep for a certain period of time, and schedule it first next time. If the concurrency limit is not exceeded, start scheduling the task.
[0019] Step S3.2: After receiving the subtask, the session data task management module calls the time series data integrity check module to perform data integrity check on the shard task;
[0020] Step S3.3: Before executing data generation, the session data generation module calls the task status management module, marks the task as running, and then starts to generate data. For session data, in addition to the five-tuple in the network, a session id is selected for grouping and aggregation, and the session indicators of multiple shards are accumulated in buckets. After aggregation, they are output to the global session table.
[0021] Preferably, the data integrity check of the shard task in step S3.2 includes: if the data is incomplete, marking the shard as incomplete and skipping the aggregation of session data of the shard; if the shard is complete, calling the task status management module to check the subtask status.
[0022] Preferably, checking the subtask status includes:
[0023] If the subtask has been completed normally, mark the subtask as completed and return;
[0024] If the subtask is running, mark the subtask as running and return;
[0025] If the subtask status is abnormal, query the session table. If there is no time-sliced task in the session table, re-mark the subtask as being in cloud operation and execute the session data aggregation task. If there is data, mark it as completed and return. If the subtask is unknown, execute the session data aggregation task.
[0026] In a second aspect, a system for aggregating and analyzing massive conversation data is provided, the system comprising:
[0027] Input module: provides an external interface, receives query requests, and verifies request parameters;
[0028] Conversation analysis management module: processes query requests, manages the entire query life cycle, and outputs intermediate preview results in a timely manner;
[0029] Session data task management module: maintains a task queue and manages shard query tasks;
[0030] Session data generation module: takes the sharding task in the task management module as input, obtains the task status, executes the sharding task if the task has not been executed, aggregates the sharding time series data into session data, and updates the sharding status;
[0031] Time series data integrity check module: check the integrity of time series data;
[0032] Task status management module: maintains the task status table and records the status of shard tasks;
[0033] Session table query module: queries the session table according to the query requirements and returns the intermediate preview results. When the query is completed, the final query results are returned.
[0034] According to a third aspect, a computer-readable storage medium storing a computer program is provided, and when the computer program is executed by a processor, the steps in the method for aggregating and analyzing massive conversation data are implemented.
[0035] In a fourth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps in the method for aggregating and analyzing massive conversation data.
[0036] Compared with the prior art, the present invention has the following beneficial effects:
[0037] 1. The present invention can support the analysis of massive initial session data over a long period of time;
[0038] 2. The present invention can provide real-time feedback of intermediate processing results during the analysis process, thereby improving interactivity.
[0039] Other beneficial effects of the present invention will be explained in the specific implementation manner through the introduction of specific technical features and technical solutions. Through the introduction of these technical features and technical solutions, those skilled in the art should be able to understand the beneficial technical effects brought about by the technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0041] Figure 1 Provide an overall flow chart for conversation analysis;
[0042] Figure 2 It is the system module diagram;
[0043] Figure 3 Schedule flow chart for session sharding tasks;
[0044] Figure 4 This is a preview image of the sharding task;
[0045] Figure 5 Bucketing algorithm and query graph for session data. DETAILED DESCRIPTION
[0046] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.
[0047] The embodiment of the present invention provides a system for aggregate analysis of massive conversation data, referring to Figure 1 and Figure 2 As shown, the system specifically includes the following contents:
[0048] Input module: provides an external interface for receiving query requests and verifying request parameters.
[0049] Conversation analysis management module: processes query requests, manages the entire query life cycle, and outputs intermediate preview results in a timely manner.
[0050] Session data task management module: This module maintains a task queue for managing shard query tasks
[0051] Session data generation module: takes the sharding task in the task management module as input, obtains the task status, and executes the sharding task if the task has not been executed, aggregates the sharding time series data into session data, and updates the sharding status.
[0052] Time series data integrity verification module: used to verify the integrity of time series data.
[0053] Task status management module: maintains the task status table to record the status of shard tasks.
[0054] Session table query module: queries the session table according to the query requirements and returns the intermediate preview results. When the query is completed, the final query results are returned.
[0055] Reference Figure 1 As shown, the present invention also provides a method for aggregating and analyzing massive conversation data, which specifically includes:
[0056] Step S1: The input module receives the query request sent by the front end and verifies the request parameters. When the necessary parameters are missing, a verification error is returned. The necessary parameters include the time range, the session five-tuple column, the indicator column and the number of pages. The time range consists of the start time and the end time, indicating that you want to query the active sessions within that period of time. The number of pages refers to which page data needs to be obtained. The session five-tuple includes the source IP address, the source port number, the destination IP address, the destination port number and the network protocol. The indicator column includes but is not limited to the packet length, the number of packets, the bit rate, the byte rate, the packet rate, the number of sessions, the average packet length, the number of broadcast packets, the group Number of broadcast packets, number of unicast packets, packet length distribution, number of connection no responses, connection no response rate, number of connection resets, connection reset rate, SYN, SYN&ACK, FIN, RST, number of zero windows, average window size, number of retransmissions, retransmission rate, number of packet losses, packet loss rate, number of out-of-order packets, out-of-order rate, number of duplicate packets, duplication rate, number of connection establishment, number of successful connections, connection success rate, connection failure rate, three-way handshake time, maximum three-way handshake time, minimum three-way handshake time, Apdex index, network delay, response time, maximum response time, minimum response time, data transmission time, and ACK time.
[0057] After verification, the request parameters are sent to the session analysis management module.
[0058] Step S2: Receive request parameters through the session analysis management module, manage the entire query life cycle, slice the task, and send the sliced task to the session task management module.
[0059] Reference Figure 3 and Figure 4 As shown, step S2 specifically includes:
[0060] Step S2.1: Split subtasks. Due to the large amount of data, if the data of the entire query time range is aggregated, the memory will be seriously insufficient. At this time, it is necessary to split the time range into multiple fixed time slices of session data generation subtasks in ascending order. In order to improve the response time of the home page, refer to Figure 4 As shown, the first time slice needs to be set to 10 seconds.
[0061] Step S2.2: After splitting the subtasks, the subtasks are sent to the session data task management module in turn. After sending, the subtask status is polled regularly. At this time, the subtask status is divided into scheduling, running, completed, and abnormally ended. When the task status is marked as scheduling or running, the next polling cycle continues to poll until the task status is marked as completed or abnormally ended. For abnormally ended tasks, there is no need to trigger a preview. For completed status, refer to Figure 4 As shown, the preview behavior is triggered, and the request for subtask preview is sent to the session table query module.
[0062] Step S2.3: Reference Figure 5 As shown, the session table query module queries the session data of the subtask, performs filtering, sorting and paging. The filtering conditions include time filtering and five-tuple filtering, such as filtering ip.src=="192.168.1.1"&&port.dest==80&&starttime>="2022-12-12 10:00:00"&&endtime<"2022-12-1216:00:00". Sorting is to sort in ascending or descending order according to one or more indicators, such as sorting in descending order according to the number of packets. When paging, assuming that the number of sessions per page is N, if the query condition takes page 3, all records with offsets from 30 to 40 are returned. At this time, if the previous sharding has been completed, it is necessary to merge the current page data and the previous page data, and then do sorting and paging, and then output the preview result. If the current page is already the last page, this query ends.
[0063] Step S3: The session data task management module receives the subtask through the message queue, executes the subtask after receiving it, and correctly marks the subtask status.
[0064] The step S3 specifically includes the following contents:
[0065] Step S3.1: Determine whether the concurrency is exceeded. If so, insert the task to the front of the cache queue, sleep for a certain period of time, and schedule it first next time. If the concurrency is not exceeded, start scheduling the task.
[0066] Step S3.2: After the module receives the subtask, it calls the time series data integrity verification module to perform data integrity verification on the shard task. If the data is incomplete, the shard is marked as incomplete and the session data aggregation of this shard is skipped. If the shard is complete, the task status management module is called to check the subtask status. If the subtask has ended normally, the subtask is marked as ended and returned. If the subtask is running, the subtask is marked as running and returned. If the subtask status is abnormal, the session table is queried. If there is no time shard task in the session table, the subtask is re-marked as cloud running and the session data aggregation task is executed. If there is data, it is marked as completed and returned. If the subtask is unknown, the session data aggregation task is executed.
[0067] Step S3.3: Before executing data generation, the session data generation module calls the task status management module, marks the task as running, and then starts to generate data. For session data, in addition to the 5-tuple in the network (source IP address, source port number, destination IP address, destination port number and network protocol), since the same quintuple may be different sessions, a session id is required at this time. This method uses the start time of the session StartTime as the session ID, that is, grouping and aggregation are performed according to (session start time, source IP address, source port number, destination IP address, destination port number and network protocol), and bucketing and accumulating session indicators of multiple shards, refer to Figure 5 As shown in the figure, after aggregation, the data is output to the global session table.
[0068] Step S3.4: The task status management module maintains a shard task session status table. After the session data is generated, if there is an exception, the shard status is marked as abnormal; if it ends normally, it is marked as completed.
[0069] Step S4: When the session data task is completed, the session analysis management module schedules the session table query module to generate segmented session data, merge multiple segments, and generate a preview or final result.
[0070] The embodiments of the present invention provide a method, system, device and medium for aggregate analysis of massive conversation data, which are highly interactive and can support analysis of massive initial conversation data over a long period of time.
[0071] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.
[0072] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.
Claims
1. A method for aggregate analysis of massive conversation data. It is characterized in that include: Step S1: Send a query request through the front end, the input module receives and verifies the request parameters. When the necessary parameters in the request parameters are missing, a verification error is returned. After verification, the request parameters are sent to the session analysis management module; Step S2: receiving request parameters through the session analysis management module, managing the entire query life cycle, slicing the task, and sending the slicing task to the session task management module; Step S3: The session data task management module receives the subtask through the message queue, executes the subtask after receiving it, and correctly marks the subtask status; Step S4: When the session data task is completed, the session analysis management module schedules the session table query module to generate segmented session data, merge multiple segments, and generate a preview or final result; The step S2 comprises: Step S2.1: Split the subtasks, in ascending order of time, split the time range into multiple session data generation subtasks with fixed time slices, and set the first time slice to 10 seconds; Step S2.2: After splitting the subtasks, the subtasks are sent to the session data task management module in sequence. After sending, the subtask status is polled regularly. At this time, the subtask status is divided into scheduling, running, completed, and abnormally ended. When the task status is marked as scheduling or running, the next polling cycle continues to poll until the task status is marked as completed or abnormally ended; Step S2.3: The session table query module queries the session data of the subtask and performs filtering, sorting and paging.
2. The method for aggregate analysis of massive conversation data according to claim 1, It is characterized in that In the step S2.2, for tasks that are abnormally terminated, there is no need to trigger a preview; for tasks that are completed, a preview action is triggered, and a request for a subtask preview is sent to the session table query module.
3. The method for aggregate analysis of massive conversation data according to claim 1, It is characterized in that The step S2.3 also includes: if the previous sharding has been completed, merging the current page data with the previous page data, sorting and paging, and then outputting the preview result; if the current page is already the last page, the query ends.
4. The method for aggregating and analyzing massive conversation data according to claim 1, It is characterized in that The step S3 comprises: Step S3.1: Determine whether the concurrency limit is exceeded. If so, insert the task to the front of the cache queue, sleep for a certain period of time, and schedule it first next time. If the concurrency limit is not exceeded, start scheduling the task. Step S3.2: After receiving the subtask, the session data task management module calls the time series data integrity check module to perform data integrity check on the shard task; Step S3.3: Before executing data generation, the session data generation module calls the task status management module to mark the task as running, and then aggregates the session data from the time series session table according to the five-tuple and the session start time. After aggregation, it is output to the global session table.
5. The method for aggregate analysis of massive conversation data according to claim 4, It is characterized in that The data integrity check of the shard task in step S3.2 includes: if the data is incomplete, marking the shard as incomplete and skipping the aggregation of session data of the shard; if the shard is complete, calling the task status management module to check the subtask status.
6. The method for aggregate analysis of massive conversation data according to claim 5, It is characterized in that The checking of subtask status includes: If the subtask has been completed normally, mark the subtask as completed and return; If the subtask is running, mark the subtask as running and return; If the subtask status is abnormal, query the session table. If there is no time-sliced task in the session table, re-mark the subtask as being in cloud operation and execute the session data aggregation task. If there is data, mark it as completed and return. If the subtask is unknown, execute the session data aggregation task.
7. A massive conversation data aggregation and analysis system, based on the massive conversation data aggregation and analysis method according to any one of claims 1 to 6, It is characterized in that include: Input module: provides an external interface, receives query requests, and verifies request parameters; Conversation analysis management module: processes query requests, manages the entire query life cycle, and outputs intermediate preview results in a timely manner; Session data task management module: maintains a task queue and manages shard query tasks; Session data generation module: takes the sharding task in the task management module as input, obtains the task status, executes the sharding task if the task has not been executed, aggregates the sharding time series data into session data, and updates the sharding status; Time series data integrity check module: check the integrity of time series data; Task status management module: maintains the task status table and records the status of shard tasks; Session table query module: queries the session table according to the query requirements and returns the intermediate preview results. When the query is completed, the final query results are returned.
8. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the steps in the method for aggregate analysis of massive conversation data according to any one of claims 1 to 6 are implemented.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the computer program is executed by a processor, the steps in the method for aggregate analysis of massive conversation data according to any one of claims 1 to 6 are implemented.
