Efficient short message data statistical method, system and device and storage medium

Abstract

The invention discloses an efficient short message data statistics method, system and device and a storage medium, belongs to the technical field of short message data statistics, and solves the technical problem that the delay of data statistics is increased by an existing serial processing mode. The method comprises the following steps of: 1, establishing a plurality of threads by adopting a multi-thread framework; 2, dividing the short message data of one day into a plurality of short message data tables according to time periods; 3, processing the short message data in the short message data table corresponding to a set time period at a first timing point, and storing the short message data in a first temporary statistical table; 4, processing the rest short message data in the short message data table at a second timing point, and storing the rest short message data in a second temporary statistical table; and 5, merging data in the first temporary statistical table and the second temporary statistical table into a final statistical table. The multi-thread technology is adopted, the processing efficiency and response speed of short message data statistics are improved, the temporary statistical table is introduced, real-time data summarization is allowed, and the overall statistical time is shortened.

Description

Technical Field

[0001] The present invention relates to the technical field of short message data statistics, and more specifically, to an efficient short message data statistics method, system, device and storage medium. Background Art

[0002] In current SMS sending platforms, data statistics usually rely on traditional batch processing methods, which often show obvious performance bottlenecks when the data volume is large, especially when the daily SMS data volume exceeds 300 million.

[0003] In the prior art, the scheduling of scheduled tasks usually adopts a single-thread or simple multi-threading method, which leads to low processing efficiency during the peak period of data statistics. In addition, in the data statistics process, the existing system often needs to wait for the previous task to be completely completed before starting the next task. Such a serial processing method increases the delay of data statistics and affects the ability of real-time data analysis. Therefore, the prior art has obvious deficiencies in high-concurrency and high-efficiency data processing, and a more efficient algorithm is urgently needed to improve the data processing speed and accuracy. Summary of the invention

[0004] The technical problem to be solved by the present invention is to address the above-mentioned deficiencies in the prior art. One purpose of the present invention is to provide an efficient method for counting short message data.

[0005] The second purpose of the present invention is to provide an efficient short message data statistics system.

[0006] A third object of the present invention is to provide a computer device.

[0007] A fourth objective of the present invention is to provide a computer storage medium.

[0008] In order to achieve the above objective 1, the present invention provides an efficient method for counting SMS data, comprising the following steps:

[0009] Step 1. Use a multi-threaded architecture to establish multiple threads, which are used to distribute data statistics tasks to multiple threads for parallel execution to reduce processing time;

[0010] Step 2. Divide the SMS data of the day into several SMS data tables according to time periods;

[0011] Step 3. Set the first scheduled task to process the SMS data in the SMS data table corresponding to the set time period at the first scheduled point. The specific steps are as follows:

[0012] Step 31. Divide the SMS data in the SMS data table corresponding to the set time period into a number of subsets according to the number of threads, the number of subsets is equal to the number of threads, and each subset corresponds to each thread one by one;

[0013] Step 32: Start multiple threads to perform statistical calculations on each subset respectively;

[0014] Step 33. After each thread completes the statistical calculation, the result is stored in the first temporary statistical table;

[0015] Step 4. Set the second scheduled task to process the remaining SMS data in the SMS data table at the second scheduled point. The specific steps are as follows:

[0016] Step 41. Divide the SMS data in the SMS data table corresponding to the remaining time period into a number of subsets according to the number of threads. The number of subsets is equal to the number of threads, and each subset corresponds to each thread one by one.

[0017] Step 42: Start multiple threads to perform statistical calculations on each subset respectively;

[0018] Step 43. After completing the statistical calculation, each thread stores the result in the second temporary statistical table;

[0019] Step 5. Combine the data in the first temporary statistical table and the second temporary statistical table into a final statistical table.

[0020] As a further improvement, in step 2, the SMS data of one day is divided into 24 SMS data tables by hour.

[0021] Furthermore, the first timing point is 22:00, which is triggered at 22:00 every day to extract the SMS data from the 0th to the 21st hour from the database, with a total of 22 SMS data tables;

[0022] The second timing point is 2:00, which is triggered at 2:00 every day to extract the SMS data of the 22nd and 23rd hours from the database, with a total of 2 SMS data tables.

[0023] Furthermore, in step 32 and step 42, the extracted data is cleaned to process missing values ​​and outliers.

[0024] Furthermore, the SMS data table is used to store all SMS records, which include fields for sending time, receiving number, and content;

[0025] The first temporary statistical table has the same structure as the second temporary statistical table, and is used to store the intermediate statistical results of each thread. The structure is also consistent with the final statistical table, which is convenient for subsequent data aggregation.

[0026] The final statistical table is used to store the final statistical results for subsequent data analysis and report generation.

[0027] Furthermore, the scheduling mechanism adopts a time-based scheduling strategy to ensure that the task is started at a predetermined time point, and uses a thread pool to manage the creation and destruction of threads to improve resource utilization efficiency.

[0028] Furthermore, in step 33, the first temporary statistical table is saved in the database and marked as a state to be merged;

[0029] In step 43, the second temporary statistical table is saved in the database and marked as a pending merge state;

[0030] In step 5, the final statistical table is saved in the database and marked as completed; at the same time, the data of the first temporary statistical table and the second temporary statistical table are cleared to prepare for the next statistical task;

[0031] Record the processing results and execution time of each step to ensure the traceability of data processing.

[0032] In order to achieve the above-mentioned second objective, the present invention provides an efficient SMS data statistics system, comprising:

[0033] The thread creation module is used to establish multiple threads using a multi-threaded architecture, and is used to distribute data statistics tasks to multiple threads for parallel execution to reduce processing time;

[0034] A partitioning module is used to divide the SMS data of a day into several SMS data tables according to time periods;

[0035] The first task module is used to set the first scheduled task, and process the SMS data in the SMS data table corresponding to the set time period at the first scheduled point, specifically as follows: divide the SMS data in the SMS data table corresponding to the set time period into a plurality of subsets according to the number of threads, the number of subsets is equal to the number of threads, and each subset corresponds to each thread one by one; start multiple threads, and perform statistical calculations on each subset respectively; after each thread completes the statistical calculations, the results are stored in the first temporary statistical table;

[0036] The second task module is used to set a second timing task to process the remaining SMS data in the SMS data table at the second timing point, specifically as follows: divide the SMS data in the SMS data table corresponding to the remaining time period into several subsets according to the number of threads, the number of subsets is equal to the number of threads, and each subset corresponds to each thread one by one; start multiple threads, and perform statistical calculations on each subset respectively; after each thread completes the statistical calculation, the result is stored in the second temporary statistical table;

[0037] The merging module is used to merge the data in the first temporary statistical table and the second temporary statistical table into a final statistical table.

[0038] In order to achieve the third objective above, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned efficient SMS data statistics method when executing the computer program.

[0039] In order to achieve the fourth objective above, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the above-mentioned efficient SMS data statistics method is implemented.

[0040] Beneficial Effects

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] 1. Improved processing efficiency: Multi-threaded parallel processing can process multiple data subsets at the same time, significantly shortening the time required for data statistics, especially in high-concurrency scenarios.

[0043] 2. Real-time data statistics: The introduction of a temporary statistical table mechanism allows each thread to summarize the results immediately after processing its own data, reducing waiting time and improving the real-time and response speed of statistics.

[0044] 3. Enhanced data accuracy: Through structured data storage and step-by-step aggregation, it ensures that data is not lost during the statistical process, reduces the error rate caused by concurrent processing, and improves the reliability of the final results.

[0045] 4. Flexible scheduling mechanism: Combined with the scheduling strategy of scheduled tasks, statistical tasks can be automatically started within the scheduled time, reducing the need for manual intervention and improving the automation level of the system.

[0046] 5. Resource utilization optimization: Use thread pool to manage thread resources, avoid frequent creation and destruction of threads, improve system resource utilization, and reduce system burden.

[0047] 6. Easy to expand and maintain: The modular design of this method makes it easy to expand and maintain the system in the future to adapt to more complex business needs.

[0048] In summary, through multi-threaded parallel processing and structured data management, the efficiency and accuracy of SMS data statistics are significantly improved, which has important practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of the present invention;

[0050] Figure 2 This is a flow chart of the first timing task in the present invention;

[0051] Figure 3 This is a flow chart of the second timing task in the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below with reference to the specific embodiments in the accompanying drawings.

[0053] See also Figure 1 to Figure 3 , an efficient SMS data statistics method, including the following steps 1 to 5:

[0054] Step 1. Use a multi-threaded architecture to establish multiple threads to distribute data statistics tasks to multiple threads for parallel execution to reduce processing time.

[0055] Step 2. Divide the SMS data of a day into several SMS data tables by time period. In this embodiment, the SMS data of a day is divided into 24 SMS data tables by hour, namely SMS data table 0 to SMS data table 23, which respectively represent the SMS data of each hour in a day.

[0056] Step 3. Set the first scheduled task, and process the SMS data in the SMS data table corresponding to the set time period at the first scheduled point. In this embodiment, the first scheduled point is 22:00, and it is triggered at 22:00 every day to extract the SMS data from the 0th to the 21st hour from the database, a total of 22 SMS data tables. The specific steps are as follows:

[0057] Step 31. Divide the SMS data in the SMS data table corresponding to the set time period (i.e. SMS data table 0 to SMS data table 21) into a number of subsets according to the number of threads, the number of subsets is equal to the number of threads, and each subset corresponds to each thread one by one;

[0058] Step 32: Start multiple threads to perform statistical calculations on each subset respectively;

[0059] Step 33. After each thread completes the statistical calculation, the result is stored in the first temporary statistical table;

[0060] Step 4. Set the second scheduled task to process the remaining SMS data in the SMS data table at the second scheduled point. In this embodiment, the second scheduled point is 2:00 (2 a.m.), which is triggered at 2:00 every day to extract the SMS data of the 22nd and 23rd hours from the database, for a total of 2 SMS data tables. The specific steps are as follows:

[0061] Step 41. Divide the SMS data in the SMS data table corresponding to the remaining time period (i.e. SMS data table 22 - SMS data table 23) into a number of subsets according to the number of threads, the number of subsets is equal to the number of threads, and each subset corresponds to each thread one by one;

[0062] Step 42: Start multiple threads to perform statistical calculations on each subset respectively;

[0063] Step 43. After completing the statistical calculation, each thread stores the result in the second temporary statistical table.

[0064] Step 5. Combine the data in the first temporary statistical table and the second temporary statistical table into a final statistical table to ensure data integrity and accuracy.

[0065] In step 32 and step 42, the extracted data is cleaned, missing values ​​and outliers are processed, and the impact of missing values ​​and outliers on the overall data is reduced.

[0066] In this embodiment, the SMS data table is used to store all SMS records, and the SMS records include fields for sending time, receiving number, and content.

[0067] The first temporary statistical table has the same structure as the second temporary statistical table and is used to store the intermediate statistical results of each thread. The structure is also consistent with the final statistical table to facilitate subsequent data aggregation.

[0068] The final statistical table is used to store the final statistical results for subsequent data analysis and report generation.

[0069] The multi-threaded scheduling mechanism adopts a time-based scheduling strategy to ensure that the task is started at the scheduled time point, and uses the thread pool to manage the creation and destruction of threads to improve resource utilization efficiency.

[0070] In step 33, it also includes saving the first temporary statistical table into the database and marking it as a state to be merged; in step 43, it also includes saving the second temporary statistical table into the database and marking it as a state to be merged; in step 5, it also includes saving the final statistical table into the database and marking it as a completed state; at the same time, the data of the first temporary statistical table and the second temporary statistical table are cleared to prepare for the next statistical task.

[0071] Record the processing results and execution time of each step to ensure the traceability of data processing.

[0072] Through the first scheduled task and the second scheduled task, the daily SMS data can be efficiently counted to ensure that the statistical results of the previous day can be viewed in time the next day.

[0073] This method can improve processing speed: through multi-threaded parallel processing, statistical tasks are distributed to multiple threads for execution, effectively shortening the task completion time. It can reduce delays: introducing temporary statistical tables to accumulate results during task execution, avoiding waiting for all data to be processed, and improving real-time performance. It can improve accuracy: through structured data storage and aggregation mechanisms, ensure that data is not lost during the statistical process and ensure the accuracy of the final results.

[0074] An efficient SMS data statistics system, comprising:

[0075] The thread creation module is used to establish multiple threads using a multi-threaded architecture, and is used to distribute data statistics tasks to multiple threads for parallel execution to reduce processing time;

[0076] A partitioning module is used to divide the SMS data of a day into several SMS data tables according to time periods;

[0077] The first task module is used to set the first scheduled task, and process the SMS data in the SMS data table corresponding to the set time period at the first scheduled point, specifically as follows: divide the SMS data in the SMS data table corresponding to the set time period into a plurality of subsets according to the number of threads, the number of subsets is equal to the number of threads, and each subset corresponds to each thread one by one; start multiple threads, and perform statistical calculations on each subset respectively; after each thread completes the statistical calculations, the results are stored in the first temporary statistical table;

[0078] The second task module is used to set a second timing task to process the remaining SMS data in the SMS data table at the second timing point, specifically as follows: divide the SMS data in the SMS data table corresponding to the remaining time period into several subsets according to the number of threads, the number of subsets is equal to the number of threads, and each subset corresponds to each thread one by one; start multiple threads, and perform statistical calculations on each subset respectively; after each thread completes the statistical calculation, the result is stored in the second temporary statistical table;

[0079] The merging module is used to merge the data in the first temporary statistical table and the second temporary statistical table into a final statistical table.

[0080] A computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the above-mentioned efficient method for counting short message data is implemented.

[0081] A computer-readable storage medium stores a computer program, which implements the above-mentioned efficient SMS data statistics method when executed by a processor.

[0082] The above are only preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the structure of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. An efficient method for counting SMS data, characterized in that: The following steps are involved: Step 1. Use a multi-threaded architecture to establish multiple threads, which are used to distribute data statistics tasks to multiple threads for parallel execution to reduce processing time; Step 2. Divide the SMS data of the day into several SMS data tables according to time periods; Step 3. Set the first scheduled task to process the SMS data in the SMS data table corresponding to the set time period at the first scheduled point. The specific steps are as follows: Step 31. Divide the SMS data in the SMS data table corresponding to the set time period into a number of subsets according to the number of threads, the number of subsets is equal to the number of threads, and each subset corresponds to each thread one by one; Step 32: Start multiple threads to perform statistical calculations on each subset respectively; Step 33. After each thread completes the statistical calculation, the result is stored in the first temporary statistical table; Step 4. Set the second scheduled task to process the remaining SMS data in the SMS data table at the second scheduled point. The specific steps are as follows: Step 41. Divide the SMS data in the SMS data table corresponding to the remaining time period into a number of subsets according to the number of threads. The number of subsets is equal to the number of threads, and each subset corresponds to each thread one by one. Step 42: Start multiple threads to perform statistical calculations on each subset respectively; Step 43. After completing the statistical calculation, each thread stores the result in the second temporary statistical table; Step 5. Combine the data in the first temporary statistical table and the second temporary statistical table into a final statistical table.

2. The efficient SMS data statistics method according to claim 1, characterized in that: In step 2, the SMS data of one day is divided into 24 SMS data tables by hour.

3. The efficient SMS data statistics method according to claim 1, characterized in that: The first timing point is 22:

00. It is triggered at 22:00 every day to extract the SMS data from the 0th to the 21st hour from the database, with a total of 22 SMS data tables. The second timing point is 2:00, which is triggered at 2:00 every day to extract the SMS data of the 22nd and 23rd hours from the database, with a total of 2 SMS data tables.

4. The efficient SMS data statistics method according to claim 1, characterized in that: In step 32 and step 42, the extracted data is cleaned to process missing values ​​and outliers.

5. The efficient SMS data statistics method according to claim 1, characterized in that: The SMS data table is used to store all SMS records, which include fields such as sending time, receiving number, and content; The first temporary statistical table has the same structure as the second temporary statistical table, and is used to store the intermediate statistical results of each thread. The structure is also consistent with the final statistical table, which is convenient for subsequent data aggregation. The final statistical table is used to store the final statistical results for subsequent data analysis and report generation.

6. The efficient SMS data statistics method according to claim 1, characterized in that: The scheduling mechanism adopts a time-based scheduling strategy to ensure that the task is started at the scheduled time, and uses the thread pool to manage the creation and destruction of threads to improve resource utilization efficiency.

7. The efficient SMS data statistics method according to claim 1, characterized in that: In step 33, the first temporary statistical table is saved in the database and marked as a pending merge state; In step 43, the second temporary statistical table is saved in the database and marked as a pending merge state; In step 5, the final statistical table is saved in the database and marked as completed; at the same time, the data of the first temporary statistical table and the second temporary statistical table are cleared to prepare for the next statistical task; Record the processing results and execution time of each step to ensure the traceability of data processing.

8. An efficient SMS data statistics system, characterized in that: include: The thread creation module is used to establish multiple threads using a multi-threaded architecture, and is used to distribute data statistics tasks to multiple threads for parallel execution to reduce processing time; A partitioning module is used to divide the SMS data of a day into several SMS data tables according to time periods; The first task module is used to set the first scheduled task, and process the SMS data in the SMS data table corresponding to the set time period at the first scheduled point, specifically as follows: the SMS data in the SMS data table corresponding to the set time period is divided into a number of subsets according to the number of threads, the number of subsets is equal to the number of threads, and each subset corresponds to each thread one by one; Start multiple threads to perform statistical calculations on each subset respectively; after each thread completes the statistical calculations, the result is stored in a first temporary statistical table; The second task module is used to set a second scheduled task to process the remaining SMS data in the SMS data table at the second scheduled point, specifically as follows: the SMS data in the SMS data table corresponding to the remaining time period is divided into a number of subsets according to the number of threads, the number of subsets is equal to the number of threads, and each subset corresponds to each thread one by one; Start multiple threads to perform statistical calculations on each subset respectively; after each thread completes the statistical calculations, the result is stored in the second temporary statistical table; The merging module is used to merge the data in the first temporary statistical table and the second temporary statistical table into a final statistical table.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, an efficient SMS data statistics method as described in any one of claims 1-7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, an efficient SMS data statistics method as described in any one of claims 1 to 7 is implemented.

Images