A data processing method and system in a high-concurrency scenario in a data center

By collecting the data to be processed in the data center, designing the database table structure, performing data partitioning and bottleneck identification, and combining load balancing optimization, the problem of long response time for data processing in high concurrency scenarios is solved, and fast response and efficient data processing are achieved.

CN118708593BActive Publication Date: 2024-11-26BEIJING YIDAZHAOXIANG TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410918704.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-26
Estimated Expiration
2044-07-10

AI Technical Summary

Technical Problem

In the prior art, the data processing response time is long in high concurrency scenarios, especially when there are a large number of users or flash killers, the average response time of the server is extended, resulting in low efficiency of high concurrent data processing in the data center.

Method used

By collecting the data to be processed within the preset time period of the data center, designing the database table structure and performing data partitioning, identifying bottleneck partitions, performing separation processing, and monitoring transmission delays in real time, obtaining high concurrent data evaluation index, and optimizing using load balancing methods Database table structure until a single server corresponds to a high concurrent user within a single preset time period.

Benefits of technology

It has achieved the shortening of the response time of data processing in high concurrency scenarios, improved the data processing efficiency of the data center, solved the problem of long response time of data processing, and numerically evaluated the data volume and performance bottlenecks of the database table structure, improving the evaluation Accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118708593B_ABST
    Figure CN118708593B_ABST
Patent Text Reader

Abstract

The present invention discloses a data processing method and system in a high-concurrency scenario of a data center. The data processing method in a high-concurrency scenario of a data center relates to the field of electronic digital data processing technology, and includes the following steps: database table structure design; performance bottleneck identification; high-concurrency data processing; structure optimization. The present invention collects the data to be processed within a preset time period of the data center and designs the database table structure according to business needs, and then partitions the database table structure in combination with the data to be processed to obtain a new partition structure and identify the bottleneck partition in the database table structure, and then separates and processes the data to be processed according to the bottleneck partition and obtains a high-concurrency data evaluation index, and finally optimizes the database table structure according to the high-concurrency data evaluation index, thereby achieving the effect of shortening the response time of data processing in a high-concurrency scenario of the data center, and solves the problem of long data processing response time in a high-concurrency scenario of the data center in the prior art.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a data processing method and system in a high-concurrency scenario of a data center. Background Art

[0002] With the rapid advancement of modern technology, high-concurrency scenarios are appearing more and more frequently in various fields. High concurrency refers to scenarios where the system needs to respond to and efficiently process massive user requests at the same time. In this scenario, the system must not only have excellent performance, but also maintain high stability and fast response speed. In life, it is not difficult to find such scenarios everywhere, such as busy e-commerce promotions, train ticket rush during the Spring Festival, hot topic discussions on social media, limited-time flash sales, peak hours of online games, automated control of factory production lines, rapid response of hospital registration systems, smooth operation of public transportation ticketing systems, real-time updates of information flow platforms, and real-time feedback of AI applications. The core challenge of these high-concurrency scenarios is whether the system can withstand and effectively process a large number of real-time requests. Message processing, as a key link in the flow of information in modern society, covers multiple aspects such as data collection, organization, in-depth analysis, secure storage, and efficient transmission.

[0003] Existing high-concurrency data processing systems use information classification and coding technology to improve the efficiency and accuracy of information processing, data compression technology to reduce storage and transmission costs, data encryption and decryption technology to ensure information security, and sequence analysis and pattern recognition technology to extract hidden value in data.

[0004] For example, the data processing method, device and storage medium under multi-blockchain interactive concurrency announced by the invention patent with announcement number: CN109376020B include: taking out an idle thread from the specified thread pool after receiving the transaction chain message; parsing the transaction chain message in the idle thread to obtain transaction information, taking out a Channel from the head of the specified two-way queue, and determining whether the Channel is the Channel corresponding to the blockchain to which the transaction information belongs; if the Channel is the Channel corresponding to the blockchain to which the transaction information belongs, then using the Channel to send the transaction information, and putting the Channel back to the head of the two-way queue after use; if the Channel is not the Channel corresponding to the blockchain to which the transaction information belongs, then putting the Channel at the end of the two-way queue.

[0005] For example, a data processing method and device announced in the invention patent with announcement number: CN104794172B includes: selecting a counting proxy entity corresponding to the type of data in the counting request instruction sent by the client; receiving the counting request instruction sent by the client through the counting proxy entity; using the counting proxy entity to perform a counting operation according to the count quantity of the data in the counting request instruction; judging whether the counting result meets the counting transfer condition through the counting proxy entity; and when the condition is met, using the counting transfer entity to send a counting increase instruction to the counting server, so that the counting server performs data processing according to the counting increase instruction.

[0006] However, in the process of implementing the technical solution of the invention in the embodiments of the present application, the present application found that the above technology has at least the following technical problems:

[0007] In the existing technology, traditional data processing in high-concurrency scenarios generally adopts methods such as Hypertext Markup Language (HTML) staticization, image server separation, clustering and load balancing. Most of these methods are based on the perspective of improving the performance of the entire WEB system. When the number of users is large or flash sales are used, high concurrency will lead to a long average server response time, and there is a problem of long data processing response time in high-concurrency scenarios in data centers. Summary of the invention

[0008] The embodiments of the present application solve the problem of long data processing response time in high-concurrency scenarios in data centers in the prior art by providing a data processing method and system in high-concurrency scenarios in data centers, thereby shortening the data processing response time in high-concurrency scenarios in data centers.

[0009] An embodiment of the present application provides a data processing method in a high-concurrency scenario in a data center, comprising the following steps: S1, collecting data to be processed within a preset time period of the data center and designing a database table structure according to business requirements, and creating a cache invalidation processing condition in the database table structure; S2, partitioning the database table structure in combination with the data to be processed to obtain a new partition structure, and recording the execution time of the data to be processed to identify the bottleneck partition in the database table structure; S3, separating the data to be processed in combination with the identified bottleneck partition, and monitoring the transmission delay data of the data to be processed in real time during the transmission process, and obtaining a high-concurrency data evaluation index, wherein the high-concurrency data evaluation index is used to describe the data processing efficiency of the database table structure in a high-concurrency scenario; S4, optimizing the database table structure according to the obtained high-concurrency data evaluation index and using a load balancing method until a single server corresponds to a high-concurrency user within a single preset time period.

[0010] Furthermore, the specific method for obtaining the data volume analysis index is as follows: extracting the field data differential of the data to be processed from the new partition structure of the database table structure and normalizing it; calculating the data volume analysis index of the data to be processed according to the obtained field data differential, and the data volume analysis index is calculated using the following formula:

[0011] ;

[0012] In the formula, e is a natural constant, f represents the number of the data to be processed, , F is the total amount of data to be processed, is the data volume analysis indicator for the fth data to be processed, is the total amount of data to be processed for the fth data, is the data growth rate of the fth data to be processed, is the data query frequency of the fth data to be processed.

[0013] Furthermore, after identifying the bottleneck partition in the database table structure, it is necessary to further analyze the cause of the performance bottleneck; the specific analysis process of the performance bottleneck cause is: obtaining data performance indicators of the database table structure within a preset time interval from the monitoring operation data; obtaining a data performance score based on the obtained data performance indicators, and the data performance score includes a data response time score, a data throughput score and a data concurrency score; obtaining a data performance bottleneck index of the database table structure based on the obtained data performance score, and identifying the cause of the performance bottleneck of the database table structure based on the data performance bottleneck index, and the data performance bottleneck index is used to describe the degree of performance bottleneck of the database table structure.

[0014] Furthermore, a specific method for obtaining the high concurrency data evaluation index is as follows: monitoring points are set at key nodes of the transmission path of the data to be processed and real-time monitoring is performed; a static delay threshold is set by preset professionals, and when it is monitored that the transmission delay exceeds the set static delay threshold, the transmission delay data of the data to be processed during the transmission process is obtained; based on the obtained transmission delay data, a transmission delay coefficient of the data to be processed is obtained, and the transmission delay coefficient includes a data sending delay coefficient, a data propagation delay coefficient, a data queuing delay coefficient and a data processing delay coefficient; based on the obtained transmission delay coefficient, a high concurrency data evaluation index of the data to be processed is obtained.

[0015] Furthermore, the high concurrency data evaluation index is calculated using the following formula:

[0016] ;

[0017] In the formula, d represents the number of the data to be processed, , D is the total amount of data to be processed, is the high concurrent data evaluation index of the dth data to be processed, is the data transmission delay coefficient of the dth data to be processed, is the data propagation delay coefficient of the dth data to be processed, is the data queuing delay coefficient of the dth data to be processed, is the data processing delay coefficient of the dth data to be processed.

[0018] The embodiment of the present application provides a data processing system in a high-concurrency scenario of a data center, characterized in that it includes: a database table structure design module, a performance bottleneck identification module, a high-concurrency data processing module and a structure optimization module; wherein the database table structure design module is used to collect data to be processed within a preset time period of the data center and design the database table structure according to business requirements, and at the same time create a cache invalidation processing condition in the database table structure; the performance bottleneck identification module is used to partition the database table structure in combination with the data to be processed to obtain a new partition structure, and record the execution time of the data to be processed to identify the bottleneck partition in the database table structure; the high-concurrency data processing module is used to separate and process the data to be processed in combination with the identified bottleneck partition, and monitor the transmission delay data of the data to be processed in real time during the transmission process, and at the same time obtain a high-concurrency data evaluation index, and the high-concurrency data evaluation index is used to describe the data processing efficiency of the database table structure in a high-concurrency scenario; the structure optimization module is used to optimize the database table structure according to the obtained high-concurrency data evaluation index and using a load balancing method until a single server corresponds to a high-concurrency user within a single preset time period.

[0019] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0020] 1. By collecting the data to be processed within a preset time period in the data center and designing the database table structure according to business needs, the database table structure is partitioned according to the data to be processed to obtain a new partition structure and identify the bottleneck partition in the database table structure, and then the data to be processed is separated according to the bottleneck partition and a high-concurrency data evaluation index is obtained. Finally, the database table structure is optimized according to the high-concurrency data evaluation index, thereby realizing rapid response processing of data in high-concurrency scenarios, thereby shortening the data processing response time in high-concurrency scenarios in data centers, and effectively solving the problem of long data processing response time in high-concurrency scenarios in data centers in the prior art.

[0021] 2. By extracting the field data differentials of the data to be processed from the new partition structure of the database table structure, and then calculating the data volume analysis index of the data to be processed based on the obtained field data differentials, and finally evaluating the data volume of the data to be processed in the database table structure through the calculated data volume analysis index, the data volume of the data to be processed in the database table structure is quantified, thereby achieving a more accurate evaluation of the data volume of the data to be processed in the database table structure.

[0022] 3. The transmission delay coefficient is obtained by obtaining the transmission delay data of the data to be processed during the transmission process, and then the high concurrency data evaluation index of the data to be processed is obtained according to the obtained transmission delay coefficient. Finally, the data processing efficiency of the database table structure in the high concurrency scenario is evaluated according to the obtained high concurrency data evaluation index, thereby realizing the quantification of the data processing efficiency of the database table structure in the high concurrency scenario, and then realizing a more accurate evaluation of the data processing efficiency of the database table structure in the high concurrency scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flow chart of a data processing method in a high-concurrency scenario in a data center provided by an embodiment of the present application;

[0024] Figure 2 A design flow chart of the database table structure provided in the embodiment of the present application;

[0025] Figure 3 A statistical chart showing changes in the data volume analysis indicators provided in the embodiments of the present application;

[0026] Figure 4 A structural diagram of a data processing system for a high-concurrency scenario in a data center provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The embodiment of the present application solves the problem of long data processing response time in high-concurrency scenarios in data centers in the prior art by providing a data processing method and system in high-concurrency scenarios in data centers. The method collects data to be processed within a preset time period in the data center and designs a database table structure according to business requirements. At the same time, a cache invalidation processing condition is created in the database table structure. Then, the database table structure is partitioned in combination with the data to be processed to obtain a new partition structure. The execution time of the data to be processed is recorded to identify the bottleneck partitions in the database table structure. Then, the data to be processed is separated and processed in combination with the identified bottleneck partitions. The transmission delay data of the data to be processed during the transmission process is monitored in real time. At the same time, a high-concurrency data evaluation index is obtained. Finally, the database table structure is optimized according to the obtained high-concurrency data evaluation index and by using a load balancing method until a single server corresponds to a high-concurrency user within a single preset time period. This achieves a reduction in data processing response time in high-concurrency scenarios in data centers.

[0028] The technical solution in the embodiment of the present application is to solve the problem of long data processing response time in the high concurrency scenario of the above data center. The overall idea is as follows:

[0029] By collecting the data to be processed within a preset time period in the data center and designing the database table structure according to business needs, the database table structure is partitioned based on the data to be processed to obtain a new partition structure and identify the bottleneck partitions in the database table structure. The data to be processed is then separated according to the bottleneck partitions and a high-concurrency data evaluation index is obtained. Finally, the database table structure is optimized based on the high-concurrency data evaluation index, thereby achieving the effect of shortening the data processing response time in high-concurrency scenarios in the data center.

[0030] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0031] like Figure 1As shown, it is a flow chart of a data processing method in a high-concurrency scenario of a data center provided by an embodiment of the present application, and the method includes the following steps: S1, collecting the data to be processed within a preset time period of the data center and designing the database table structure according to business needs, and creating a cache invalidation processing condition in the database table structure; S2, partitioning the database table structure in combination with the data to be processed to obtain a new partition structure, and recording the execution time of the data to be processed to identify the bottleneck partition in the database table structure; S3, separating the data to be processed in combination with the identified bottleneck partition, and monitoring the transmission delay data of the data to be processed in the transmission process in real time, and obtaining a high-concurrency data evaluation index, which is used to describe the data processing efficiency of the database table structure in a high-concurrency scenario; S4, optimizing the database table structure according to the obtained high-concurrency data evaluation index and using a load balancing method until a single server corresponds to a high-concurrency user within a single preset time period.

[0032] In this embodiment, the data to be processed represents the access data of the user within a preset time interval; the database table structure is used to process and cache the collected data to be processed; the new partition structure is a sub-area of ​​the database table structure, the bottleneck partition represents a new partition structure with a performance bottleneck, and the execution time represents the timestamp required to process the data to be processed; the separation processing is used to separate the bottleneck partition, and the high-concurrency user represents the user who accesses the data to be processed within the same time interval; the data processing response time is shortened in the high-concurrency scenario of the data center.

[0033] Further, such as Figure 2 As shown, it is a design flow chart of the database table structure provided in the embodiment of the present application. The specific steps for designing the database table structure according to business requirements are as follows: Step 1, according to business requirements, identify the data to be processed that needs to be stored and define attributes to obtain the definition data to be processed, the attribute definition means defining the data to be processed through data fields, and the definition data to be processed means the data to be processed after the attribute definition; Step 2, obtain the definition field table and set field constraints, and at the same time define the primary key for the definition field table, the definition field table is created by selecting the field type for the definition data to be processed according to the nature of the data, and the primary key is used to uniquely identify the data to be processed in the definition field table; Step 3, perform normalized design on the obtained definition field table, and the normalized design includes eliminating redundancy and update anomalies of the data to be processed; Step 4, design a query index in the definition field table to obtain the database table structure, and the query index is used to search, sort and connect the data to be processed in the definition field table.

[0034] In this embodiment, the definition data to be processed is obtained by defining the attributes of the data to be processed; the field types include integers, texts and dates; the field constraints include non-null constraints, unique constraints and default values; the normalization design includes eliminating redundancy and update anomalies of the data to be processed, and the performance and reliability of the database are improved by eliminating data redundancy and update anomalies. Commonly used normalization methods include the first normal form, the second normal form and the third normal form, and the normalization method can be selected according to actual conditions; the design accuracy of the database table structure is improved, and the data processing response time in high concurrency scenarios of the data center is further shortened.

[0035] Furthermore, the specific process of data partitioning is as follows: obtain data volume analysis indicators from the database table structure, the data volume analysis indicators are used to evaluate the data volume of the data to be processed in the database table structure; select a data partitioning method based on the obtained data volume analysis indicators; modify the structure of the database table and add partition definitions based on the selected data partitioning method, and at the same time transfer the data to be processed in the database table structure to the new partition structure.

[0036] In this embodiment, the data partitioning method includes RANGE-based partitioning, LIST-based partitioning and KEY-based partitioning, which needs to be selected according to the data volume analysis index of the actual data to be processed; if there are timestamp or date fields in the database table structure, and the query is based on the range of these fields, the RANGE-based partitioning is generally selected; if the values ​​of the defined fields in the database table structure are discrete and the sets of these values ​​are known, the LIST-based partitioning is generally selected; if the distribution of the data to be processed is relatively uniform and there is no obvious range or list characteristics, the KEY-based partitioning is generally selected; the efficiency of data partitioning of the database table structure is improved, and the data processing response time in the high concurrency scenario of the data center is further shortened.

[0037] Furthermore, the specific method for obtaining the data volume analysis index is as follows: extract the field data differential of the data to be processed from the new partition structure of the database table structure and normalize it. The field data differential includes the total amount of data, the data growth rate and the data query frequency; calculate the data volume analysis index of the data to be processed according to the obtained field data differential. In addition to determining the specific indicators used for data volume analysis and giving them appropriate weights, the data volume analysis index can also be obtained by establishing an evaluation model or scoring system based on the selected indicators and weights. The model can be a mathematical model, a multi-indicator decision model or an expert system. The data volume indicators are comprehensively analyzed through the model to produce the final quantitative evaluation results. More accurate calculations can also be performed through formulas. The data volume analysis index is calculated using the following formula:

[0038] ;

[0039] In the formula, e is a natural constant, f represents the number of the data to be processed, , F is the total amount of data to be processed, is the data volume analysis indicator for the fth data to be processed, is the total amount of data to be processed for the fth data, is the data growth rate of the fth data to be processed, is the data query frequency of the fth data to be processed.

[0040] In this embodiment, if Figure 3 As shown, it is a statistical chart of the changes in the data volume analysis indicators provided in the embodiment of the present application. It can be seen from the figure that the data growth rate and the data query frequency are positively correlated with the data volume analysis indicators; when the data growth rate increases, the larger the data volume analysis indicator is, the larger the data volume of the data to be processed in the database table structure is; when the data query frequency increases, the larger the data volume analysis indicator is, the larger the data volume of the data to be processed in the database table structure is; the digitization of the data volume of the data to be processed in the database table structure is realized, and the data processing response time in the high concurrency scenario of the data center is further shortened.

[0041] Furthermore, the identification process of the bottleneck partition is as follows: write an SQL query script in the new partition structure, the SQL query script is used to simulate the actual data processing scenario; monitor the new partition structure of the database table structure in real time through the database performance monitoring tool, collect the monitoring operation data of the database table structure in combination with the written SQL query script and generate a corresponding monitoring report, the monitoring operation data represents the operation data of the database table structure in the process of processing the data to be processed, and the monitoring report is used to record the historical monitoring operation data; extract the timestamps before and after the execution of the database table structure operation from the monitoring report, and obtain the execution time of the data to be processed through the timestamps before and after the operation execution; compare the execution time of the acquired data to be processed with the preset execution time to obtain the new partition structure with a performance bottleneck.

[0042] In this embodiment, a performance monitoring tool is used to monitor the processing process of the data to be processed in real time and a corresponding monitoring report is generated based on the monitored monitoring operation data. The monitoring operation data represents the operation data of the database table structure in the process of processing the data to be processed, and the monitoring report is used to record the historical monitoring operation data. The time function in the programming language or the time function provided by the database management system is used to extract the timestamps before and after the execution of the database table structure operation, and the execution time of the operation is obtained by comparing the timestamps before and after the operation is executed, and the execution time is recorded in milliseconds for subsequent analysis. Finally, a new partition structure with a performance bottleneck is obtained based on the execution time of the acquired data to be processed. The bottleneck partition is accurately identified, and the data processing response time in the high concurrency scenario of the data center is further shortened.

[0043] Furthermore, after identifying the bottleneck partition in the database table structure, it is necessary to further analyze the cause of the performance bottleneck; the specific analysis process of the performance bottleneck cause is: obtaining data performance indicators of the database table structure within a preset time interval from the monitoring operation data, the data performance indicators include data response time, data throughput and data concurrency, and the data concurrency is the concurrent number of data to be processed within the same preset time interval; obtaining a data performance score based on the obtained data performance indicators, and the data performance score includes a data response time score, a data throughput score and a data concurrency score; the data response time score is the absolute value of the ratio of the difference between the data response time and the corresponding reference value to the corresponding reference deviation, the data throughput score is the absolute value of the ratio of the difference between the data throughput and the corresponding reference value to the corresponding reference deviation, and the data concurrency score is the absolute value of the ratio of the difference between the data concurrency and the corresponding reference value to the corresponding reference deviation; obtaining a data performance bottleneck index of the database table structure based on the obtained data performance score, and identifying the performance bottleneck cause of the database table structure based on the data performance bottleneck index, and the data performance bottleneck index is used to describe the degree of performance bottleneck of the database table structure.

[0044] In this embodiment, the data performance bottleneck index can be obtained by establishing a performance model of the data processing system and performing simulation analysis, and evaluating the performance and bottleneck position of the system under different conditions in combination with the simulation analysis results. Alternatively, a more accurate calculation can be performed using a formula. The data performance bottleneck index is calculated using the following formula:

[0045] ;

[0046] In the formula, e is a natural constant, n represents the number of the new partition structure, , N is the total number of new partition structures, is the data performance bottleneck index of the nth new partition structure, is the response time score of the nth new partition structure, is the data throughput fraction of the nth new partition structure, The data concurrency fraction of the nth new partition structure.

[0047] Specifically, the expression of data response time score is: ,in, is the data response time of the nth new partition structure, is the reference data response time of the nth new partition structure, is the data response time reference deviation; the expression of data throughput fraction is ,in, is the data throughput of the nth new partition structure, is the reference data throughput of the nth new partition structure, is the data throughput reference deviation; the expression of data concurrency fraction is ,in, is the data concurrency of the nth new partition structure, is the reference data concurrency of the nth new partition structure, It provides a reference deviation for data concurrency. It realizes the digitization of the performance bottleneck degree of the database table structure, and further shortens the data processing response time in high-concurrency scenarios in data centers.

[0048] It should be understood that the reference data response time is obtained by analyzing and counting the data response time of the new partition structure and taking the average value, the reference data throughput is obtained by analyzing and counting the data throughput of the new partition structure and taking the average value, and the reference data concurrency is obtained by analyzing and counting the data concurrency of the new partition structure and taking the average value; the data response time reference deviation is obtained by analyzing and counting the data response time of the new partition structure through the reference deviation calculation formula, the data throughput reference deviation is obtained by analyzing and counting the data throughput of the new partition structure through the reference deviation calculation formula, and the data concurrency reference deviation is obtained by analyzing and counting the data concurrency of the new partition structure through the reference deviation calculation formula.

[0049] Furthermore, the specific process of separation processing is as follows: based on the user's behavior pattern, the user's activity level is classified to obtain a classification result, the classification result includes active users and inactive users, active users are users whose interaction frequency is not lower than the average interaction frequency within a preset time interval, and inactive users are users whose interaction frequency is lower than the average interaction frequency within the preset time interval; through the user's activity level classification, the active users are separated into a new partition structure without performance bottlenecks for high-concurrency data processing, and the inactive users are separated into a new partition structure with performance bottlenecks for data storage.

[0050] In this embodiment, active users are users whose interaction frequency is not lower than the average interaction frequency within a preset time interval, and inactive users are users whose interaction frequency is lower than the average interaction frequency within a preset time interval; the accuracy of separating users with different behavior patterns is improved, and the data processing response time in high concurrency scenarios in data centers is further shortened.

[0051] Furthermore, a specific method for obtaining a high-concurrency data evaluation index is as follows: monitoring points are set at key nodes of a transmission path of the data to be processed and real-time monitoring is performed, the key nodes including application servers, database servers and cache servers; a static delay threshold is set by preset professionals, and when it is monitored that the transmission delay exceeds the set static delay threshold, the transmission delay data of the data to be processed during the transmission process is obtained, the transmission delay data including data sending delay, data propagation delay, data queuing delay and data processing delay; the transmission delay coefficient of the data to be processed is obtained based on the obtained transmission delay data, the transmission delay coefficient including data sending delay coefficient, data propagation delay coefficient, data queuing delay coefficient and data processing delay coefficient; the high-concurrency data evaluation index of the data to be processed is obtained based on the obtained transmission delay coefficient.

[0052] In this embodiment, the data sending delay coefficient is the absolute value of the ratio of the difference between the data sending delay and the corresponding reference value to the corresponding reference deviation, the data propagation delay coefficient is the absolute value of the ratio of the difference between the data propagation delay and the corresponding reference value to the corresponding reference deviation, the data queuing delay coefficient is the absolute value of the ratio of the difference between the data queuing delay and the corresponding reference value to the corresponding reference deviation, and the data processing delay coefficient is the absolute value of the ratio of the difference between the data processing delay and the corresponding reference value to the corresponding reference deviation; the data processing efficiency of the database table structure in a high concurrency scenario is quantified, and the data processing response time in a data center high concurrency scenario is further shortened.

[0053] Furthermore, in addition to simulating the system to simulate the performance under different load conditions to evaluate the high concurrency data evaluation index, the high concurrency data evaluation index can also be calculated more accurately through a formula. The high concurrency data evaluation index is calculated using the following formula:

[0054] ;

[0055] In the formula, d represents the number of the data to be processed , D is the total amount of data to be processed, is the high concurrent data evaluation index of the dth data to be processed, is the data transmission delay coefficient of the dth data to be processed, is the data propagation delay coefficient of the dth data to be processed, is the data queuing delay coefficient of the dth data to be processed, is the data processing delay coefficient of the dth data to be processed.

[0056] In this embodiment, the statistical table of changes in the high concurrency data evaluation index is shown in Table 1:

[0057] Table 1 Statistics of changes in high-concurrency data evaluation index

[0058]

[0059] It can be seen from the above table that the high-concurrency data evaluation index cannot be directly obtained through a single data, but needs to be obtained through a comprehensive analysis of the data transmission delay coefficient, data propagation delay coefficient, data queuing delay coefficient and data processing delay coefficient. For example, the data propagation delay coefficient of the second group of data is smaller than the data propagation delay coefficient of the third group of data, but the final high-concurrency data evaluation index is greater than that of the third group, because the data transmission delay coefficient, data queuing delay coefficient and data processing delay coefficient also have an impact, so a comprehensive analysis is required.

[0060] Specifically, the expression of the data transmission delay coefficient is: ,in, is the data sending delay for the dth data to be processed, The reference data sending delay for the dth data to be processed, is the data transmission delay reference deviation; the expression of data propagation delay coefficient is ,in, is the data propagation delay of the dth data to be processed, is the reference data propagation delay of the dth data to be processed, is the data propagation delay reference deviation; the expression of data queuing delay coefficient is ,in, is the data queuing delay of the dth data to be processed, is the reference data queuing delay for the dth data to be processed, is the data queuing delay reference deviation; the expression of data processing delay coefficient is ,in, is the data processing delay of the dth data to be processed, is the reference data processing delay of the dth data to be processed, It provides a reference deviation for data processing delay. It realizes the digitization of the performance bottleneck degree of the database table structure, and further shortens the data processing response time in high-concurrency scenarios in data centers.

[0061] It should be understood that the reference data sending delay is obtained by analyzing and counting the data sending delays of the new partition structure and taking the average value, the reference data propagation delay is obtained by analyzing and counting the data propagation delays of the new partition structure and taking the average value, the reference data queuing delay is obtained by analyzing and counting the data queuing delays of the new partition structure and taking the average value, and the reference data processing delay is obtained by analyzing and counting the data processing delays of the new partition structure and taking the average value; the data sending delay reference deviation is obtained by analyzing and counting the data sending delay of the new partition structure through the reference deviation calculation formula, the data propagation delay reference deviation is obtained by analyzing and counting the data propagation delay of the new partition structure through the reference deviation calculation formula, the data queuing delay reference deviation is obtained by analyzing and counting the data queuing delay of the new partition structure through the reference deviation calculation formula, and the data processing delay reference deviation is obtained by analyzing and counting the data processing delay of the new partition structure through the reference deviation calculation formula.

[0062] like Figure 4 As shown, it is a structural schematic diagram of a data processing system in a high-concurrency scenario of a data center provided by an embodiment of the present application. The data processing system in a high-concurrency scenario of a data center provided by an embodiment of the present application includes: a database table structure design module, a performance bottleneck identification module, a high-concurrency data processing module and a structure optimization module; wherein the database table structure design module is used to collect the data to be processed within a preset time period of the data center and design the database table structure according to business requirements, and at the same time create a cache invalidation processing condition in the database table structure; the performance bottleneck identification module is used to partition the database table structure in combination with the data to be processed to obtain a new partition structure, and record the execution time of the data to be processed to identify the bottleneck partition in the database table structure; the high-concurrency data processing module is used to separate and process the data to be processed in combination with the identified bottleneck partition, and monitor the transmission delay data of the data to be processed in real time during the transmission process, and at the same time obtain a high-concurrency data evaluation index, which is used to describe the data processing efficiency of the database table structure in a high-concurrency scenario; the structure optimization module is used to optimize the database table structure according to the obtained high-concurrency data evaluation index and using a load balancing method until a single server corresponds to a high-concurrency user within a single preset time period.

[0063] In this embodiment, systematization of data processing in high-concurrency scenarios in data centers is achieved, further shortening the data processing response time in high-concurrency scenarios in data centers.

[0064] To summarize, the embodiment of the present application collects the data to be processed within a preset time period of the data center and designs a database table structure according to business needs, then partitions the database table structure based on the data to be processed to obtain a new partition structure and identifies the bottleneck partitions in the database table structure, then separates the data to be processed according to the bottleneck partitions and obtains a high-concurrency data evaluation index, and finally optimizes the database table structure according to the high-concurrency data evaluation index, thereby achieving rapid response processing of data in high-concurrency scenarios, and further shortening the data processing response time in high-concurrency scenarios of the data center, effectively solving the problem of long data processing response time in high-concurrency scenarios of the data center in the prior art.

[0065] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0066] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0067] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0068] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0069] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0070] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A data processing method in a high-concurrency scenario in a data center, characterized in that: The following steps are involved: S1, collect the data to be processed within the preset time period of the data center and design the database table structure according to business needs, and create cache invalidation processing conditions in the database table structure; S2, partitioning the database table structure in combination with the data to be processed to obtain a new partition structure, and recording the execution time of the data to be processed to identify the bottleneck partition in the database table structure; S3, separate and process the data to be processed in combination with the identified bottleneck partition, monitor the transmission delay data of the data to be processed in real time during the transmission process, and obtain a high-concurrency data evaluation index, which is used to describe the data processing efficiency of the database table structure in a high-concurrency scenario; S4, evaluating the index of the acquired high-concurrency data and optimizing the database table structure using a load balancing method until a single server corresponds to a high-concurrency user within a single preset time period; The specific method for obtaining the high concurrent data evaluation index is as follows: Set up monitoring points at key nodes in the transmission path of the data to be processed and conduct real-time monitoring; The static delay threshold is set by a preset professional, and when the transmission delay is detected to exceed the set static delay threshold, the transmission delay data of the data to be processed during the transmission process is obtained; Acquire a transmission delay coefficient of the data to be processed according to the acquired transmission delay data, wherein the transmission delay coefficient includes a data transmission delay coefficient, a data propagation delay coefficient, a data queuing delay coefficient and a data processing delay coefficient; Obtain a high concurrent data evaluation index of the data to be processed according to the obtained transmission delay coefficient; The specific process of data partitioning is as follows: Obtaining a data volume analysis indicator from a database table structure, wherein the data volume analysis indicator is used to evaluate the data volume of the data to be processed in the database table structure; Select the data partitioning method based on the acquired data volume analysis indicators; Modify the structure of the database table and add partition definitions according to the selected data partitioning method, and transfer the pending data in the database table structure to the new partition structure; The specific method for obtaining the data volume analysis indicator is as follows: Extract the field data differentials of the data to be processed from the new partition structure of the database table structure and normalize them; The data volume analysis index of the data to be processed is calculated according to the obtained field data differential, and the data volume analysis index is calculated using the following formula: Where, e is a natural constant, f represents the number of the data to be processed, f = 1, 2, ..., F, F is the total number of data to be processed, δ f is the data volume analysis indicator for the fth data to be processed, Z' f is the total amount of data to be processed for the fth data, V′ f is the data growth rate of the fth data to be processed, μ′ f is the data query frequency of the fth data to be processed.

2. A data processing method in a high-concurrency scenario in a data center as claimed in claim 1, characterized in that: The specific steps of designing the database table structure according to business requirements are: Step 1: Identify the data to be stored according to business requirements and define the attributes to obtain the defined data to be processed; Step 2: Obtain a definition field table and set field constraints, and define a primary key for the definition field table. The definition field table is created by selecting a field type for the definition data to be processed according to the data properties. Step 3: Standardize the obtained definition field table; Step 4: Design a query index in the definition field table to obtain the database table structure.

3. A data processing method in a high-concurrency scenario in a data center as claimed in claim 1, characterized in that: The identification process of the bottleneck partition is as follows: Write SQL query scripts in the new partition structure; Use database performance monitoring tools to monitor the new partition structure of the database table structure in real time, and use the written SQL query scripts to collect the monitoring operation data of the database table structure and generate corresponding monitoring reports; Extract the timestamps before and after the database table structure operation is executed from the monitoring report, and obtain the execution time of the data to be processed through the timestamps before and after the operation is executed; The execution time of the acquired data to be processed is compared with the preset execution time to obtain a new partition structure with a performance bottleneck.

4. A data processing method in a high-concurrency scenario in a data center as claimed in claim 3, characterized in that: After identifying the bottleneck partition in the database table structure, it is necessary to further analyze the performance bottleneck cause; The specific analysis process of the performance bottleneck cause is as follows: Obtain data performance indicators of a database table structure within a preset time interval from the monitoring operation data; obtain a data performance score based on the obtained data performance indicators, wherein the data performance score includes a data response time score, a data throughput score, and a data concurrency score; A data performance bottleneck index of the database table structure is obtained according to the obtained data performance score, and the performance bottleneck cause of the database table structure is identified according to the data performance bottleneck index. The data performance bottleneck index is used to describe the performance bottleneck degree of the database table structure.

5. A data processing method in a high-concurrency scenario in a data center as claimed in claim 1, characterized in that: The specific process of the separation process is as follows: Classifying the activity levels of users based on their behavior patterns to obtain classification results, wherein the classification results include active users and inactive users; By classifying users' activity levels, active users are separated into a new partition structure without performance bottlenecks for high-concurrency data processing, and inactive users are separated into a new partition structure with performance bottlenecks for data storage.

6. A data processing method in a high-concurrency scenario in a data center as claimed in claim 1, characterized in that: The high concurrent data evaluation index is calculated using the following formula: In the formula, d represents the number of the data to be processed, d = 1, 2, ..., D, D is the total number of data to be processed, σ d is the high concurrency data evaluation index of the dth data to be processed, F' d is the data transmission delay coefficient of the dth data to be processed, C' d is the data propagation delay coefficient of the dth data to be processed, D' d is the data queue delay coefficient of the dth data to be processed, L′ d is the data processing delay coefficient of the dth data to be processed.

7. A system using the data processing method in a high-concurrency scenario in a data center as claimed in any one of claims 1 to 6, comprising a database table structure design module, a performance bottleneck identification module, a high-concurrency data processing module and a structure optimization module; in, The database table structure design module is used to collect the data to be processed within a preset time period of the data center and design the database table structure according to business requirements, and at the same time create cache invalidation processing conditions in the database table structure; The performance bottleneck identification module is used to partition the database table structure in combination with the data to be processed to obtain a new partition structure, and record the execution time of the data to be processed to identify the bottleneck partition in the database table structure; The high-concurrency data processing module is used to separate and process the data to be processed in combination with the identified bottleneck partitions, and monitor the transmission delay data of the data to be processed in real time during the transmission process, and obtain a high-concurrency data evaluation index at the same time, which is used to describe the data processing efficiency of the database table structure in a high-concurrency scenario; The structure optimization module is used to optimize the database table structure according to the acquired high-concurrency data evaluation index and using a load balancing method until a single server corresponds to a high-concurrency user within a single preset time period.