Data processing method, device, server and storage medium

Abstract

The embodiment of the present application discloses a data processing method, device, server and storage medium. The embodiment of the present application obtains data to be processed; determines the level of the data to be processed; allocates cache space and cache time for the data to be processed; stores the data to be processed in the cache space according to the level; and transfers the data to be processed from the cache space to a preset storage space in order of level from high to low according to the cache time. This scheme can transfer the data to be processed from the cache space to the preset storage space based on the level and cache time of the data to be processed, avoiding data loss caused by too much data to be processed being obtained at the same time and not being processed in time, thereby improving the reliability and flexibility of data processing.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a data processing method, device, server and storage medium. Background Art

[0002] At present, the server can process and analyze the real-time reported data to determine the current system operation status. Since the reported data may be reported at a relatively uniform time, a large amount of data will be reported at the same time at a certain moment, resulting in a high data peak at a certain moment. During the data peak period, the load is too high, and the time to write to the memory will become very slow, resulting in the inability to write data in time. For example, due to memory or network reasons, the data cannot be processed in the first moment, resulting in memory overflow, which will cause data loss. Therefore, data is often lost in the process of processing data, making the reliability of data transmission low. Summary of the invention

[0003] The embodiments of the present application provide a data processing method, device, server, and storage medium, aiming to improve the reliability and flexibility of data processing.

[0004] To solve the above technical problems, the present application provides the following technical solutions:

[0005] In a first aspect, an embodiment of the present application provides a data processing method, including:

[0006] Get the data to be processed;

[0007] Determining the level of the data to be processed;

[0008] Allocating cache space and cache time for the data to be processed;

[0009] storing the data to be processed into the cache space according to the level;

[0010] According to the cache time, the data to be processed are transferred from the cache space to the preset storage space in order from high to low levels.

[0011] In some implementations, determining the level of the data to be processed includes:

[0012] Obtaining an identifier carried in the data to be processed;

[0013] Determining the type of the data to be processed according to the identifier;

[0014] The level of the data to be processed is determined according to the type.

[0015] In some implementations, allocating cache space and cache time for the data to be processed includes:

[0016] Acquire the time interval of the cache space and the delay time of the data to be processed;

[0017] A cache space and a cache time are allocated for the data to be processed according to the time interval and the delay time.

[0018] In some implementations, allocating cache space and cache time for the to-be-processed data according to the time interval and the delay time includes:

[0019] Construct a circular cache queue formed by multiple cache spaces;

[0020] Setting a time interval for each cache space in the circular cache queue;

[0021] Obtaining the delay time of the data to be processed, and obtaining the sequence number of the cache space in the ring cache queue that is currently transferring data;

[0022] Cache space and cache time are allocated to the data to be processed according to the time interval, delay time and sequence number.

[0023] In some implementations, transferring the to-be-processed data from the cache space to a preset storage space in descending order of levels according to the cache time includes:

[0024] When the cache time is reached, extracting the data to be processed from the cache space in descending order of levels to obtain target data;

[0025] Parsing the target data to obtain parsed target data;

[0026] The parsed target data is sent to a preset storage space at a preset rate.

[0027] In some implementations, after sending the parsed target data to a preset storage space at a preset rate, the method further includes:

[0028] When the cache time ends, determining whether there is any unextracted data to be processed in the cache space;

[0029] If so, storing the unextracted data to be processed to the disk;

[0030] The data to be processed in the disk is parsed in order of high to low levels, and the parsed data is sent to a preset storage space at a preset rate.

[0031] In some implementations, obtaining the data to be processed includes:

[0032] Receive sample data, log data, or performance data reported by clients or other servers.

[0033] In a second aspect, an embodiment of the present application further provides a data processing device, including:

[0034] An acquisition unit, used for acquiring data to be processed;

[0035] A determination unit, used to determine the level of the data to be processed;

[0036] An allocation unit, configured to allocate cache space and cache time for the data to be processed;

[0037] A storage unit, used for storing the to-be-processed data into the cache space according to the level;

[0038] The transfer unit is used to transfer the to-be-processed data from the cache space to a preset storage space in descending order of levels according to the cache time.

[0039] In some implementations, the determining unit is specifically configured to:

[0040] Obtaining an identifier carried in the data to be processed;

[0041] Determining the type of the data to be processed according to the identifier;

[0042] The level of the data to be processed is determined according to the type.

[0043] In some embodiments, the dispensing unit comprises:

[0044] An acquisition module, used to acquire the time interval of the cache space and the delay time of the data to be processed;

[0045] An allocation module is used to allocate cache space and cache time for the data to be processed according to the time interval and delay time.

[0046] In some embodiments, the allocation module is specifically used to:

[0047] Construct a circular cache queue formed by multiple cache spaces;

[0048] Setting a time interval for each cache space in the circular cache queue;

[0049] Obtaining the delay time of the data to be processed, and obtaining the sequence number of the cache space in the ring cache queue that is currently transferring data;

[0050] Cache space and cache time are allocated to the data to be processed according to the time interval, delay time and sequence number.

[0051] In some implementations, the transfer unit is specifically used to:

[0052] When the cache time is reached, extracting the data to be processed from the cache space in descending order of levels to obtain target data;

[0053] Parsing the target data to obtain parsed target data;

[0054] The parsed target data is sent to a preset storage space at a preset rate.

[0055] In some implementations, the transfer unit is further specifically configured to:

[0056] When the cache time ends, determining whether there is any unextracted data to be processed in the cache space;

[0057] If so, storing the unextracted data to be processed to the disk;

[0058] The data to be processed in the disk is parsed in order of high to low levels, and the parsed data is sent to a preset storage space at a preset rate.

[0059] In some implementations, the acquisition unit is specifically used to receive sample data, log data, or performance data reported by a client or other server.

[0060] In a third aspect, an embodiment of the present application further provides a server, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, it executes any data processing method provided in the embodiment of the present application.

[0061] In a fourth aspect, an embodiment of the present application further provides a storage medium, wherein the storage medium is used to store a computer program, and the computer program is loaded by a processor to execute any data processing method provided in the embodiment of the present application.

[0062] The embodiment of the present application can obtain the data to be processed, determine the level of the data to be processed, and allocate cache space and cache time for the data to be processed; then the data to be processed can be stored in the cache space according to the level, and the data to be processed can be transferred from the cache space to the preset storage space in order of level from high to low according to the cache time. This solution can transfer the data to be processed from the cache space to the preset storage space based on the level of the data to be processed and the cache time, avoiding data loss caused by too much data to be processed being obtained at the same time and not being processed in time, thereby improving the reliability and flexibility of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 It is a schematic diagram of a data processing scenario provided by an embodiment of the present application;

[0065] Figure 2 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0066] Figure 3 is another flow chart of the data processing method provided in the embodiment of the present application;

[0067] Figure 4 is another flow chart of the data processing method provided in the embodiment of the present application;

[0068] Figure 5 is a schematic diagram of the structure of the timing wheel provided in an embodiment of the present application;

[0069] Figure 6 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0070] Figure 7 It is a schematic diagram of the structure of the server provided in the embodiment of the present application. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0072] The embodiments of the present application provide a data processing method, device, server, and storage medium, which are described in detail below.

[0073] See also Figure 1 , Figure 1A schematic diagram of a scenario of data processing provided by an embodiment of the present application, wherein the data processing system may include a data processing device, which may be specifically integrated in a server, wherein the server may obtain data to be processed, such as data reported by a receiving terminal or other server, and determine the level of the data to be processed, for example, the data may be graded according to the data type or data purpose. Then, a cache space and a cache time may be allocated for the data to be processed, for example, a cache space of a time wheel may be allocated for the data to be processed and the cache time corresponding to the cache space may be determined. At this time, the data to be processed may be stored in the cache space according to the level, and the data to be processed may be transferred from the cache space to the preset storage space in order from high to low according to the cache time, for example, when the cache time is reached, the data to be processed in the cache space may be parsed in order from high to low in order from the level, and the parsed data may be sent to the preset storage space (for example, Kafka) at a preset rate; and, when the cache time ends, the remaining data to be processed in the cache space may be stored to a disk, and the data to be processed in the disk may be parsed in order from high to low in order from the level, and the parsed data may be sent to the preset storage space at a preset rate. This solution can transfer the data to be processed from the cache space to the preset storage space based on the level of the data to be processed and the cache time, reducing the data loss rate and improving the reliability and flexibility of data processing.

[0074] It should be noted that Figure 1 The data processing scenario diagram shown is only an example. The data processing system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of data processing systems and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0075] See also Figure 2 , Figure 2 : is a flowchart of a data processing method provided by an embodiment of the present application. The execution subject of the data processing method may be a data processing device provided by an embodiment of the present application, or a server integrated with the data processing device, wherein the data processing device may be implemented in hardware or software. The data processing method may include:

[0076] S101. Obtain data to be processed.

[0077] Among them, the data to be processed can be in the form of data packets or files, and may include sampled data, log data or performance data, for example, the sampled data may include error information that occurs during the audio and video playback process of the client on the terminal, and extracting some data related to the error information from the audio and video, etc. The log data may include log-related data such as video transcoding, purchase of video membership, video frequency band, video evaluation, and gift giving, and the performance data may include platform performance-related data such as video playback traffic, video playback speed, and clarity. The performance data has relatively high requirements for real-time and reliability.

[0078] In some implementations, obtaining the data to be processed may include: receiving sample data, log data, or performance data reported by a client or other server.

[0079] For example, the server can receive the data to be processed reported by the client or other servers through the honeycomb system BOSS, which is a real-time data middleware service system that supports reporting data in different ways or languages, can save the reported data to memory or disk, and provide other services for subscription. Alternatively, the server can receive session management Session and other related data reported by the client. Alternatively, the server can receive the data to be processed reported by the client or other servers through the HyperText Transfer Protocol (HTTP). Alternatively, the server can receive the data to be processed reported by the client or other servers through the Transmission Control Protocol (TCP).

[0080] S102: Determine the level of the data to be processed.

[0081] Since different data (i.e., data to be processed) have different importance levels, in order to ensure that more important data can be processed first, the data to be processed can be graded. The higher the level of the data, the more important the data is. Conversely, the lower the level of the data, the less important the data is.

[0082] Among them, the level division method can be flexibly set according to actual needs. For example, the data to be processed can be divided into a first level and a second level according to the type of the data to be processed. The first level is higher than the second level. The data of the first level is important data, and the data of the second level is unimportant data.

[0083] For another example, the data to be processed may be divided into multiple levels (e.g., first level, second level, and third level, etc.) according to the purpose, size, or source of the data to be processed. For another example, the data to be processed may be divided into first level, second level, third level, fourth level, fifth level, and sixth level, etc. according to the characteristic information of the data to be processed. The levels are from the first level to the sixth level, etc., from high to low.

[0084] In some implementations, determining the level of the data to be processed may include: acquiring an identifier carried in the data to be processed; determining the type of the data to be processed according to the identifier; and determining the level of the data to be processed according to the type.

[0085] Among them, the identifier can be an identity identifier (ID, Identification), name or number of the data to be processed, the identifier can be used to uniquely identify the data, and the identifier can be composed of numbers, letters or text. In order to improve the convenience of grading the data to be processed, it can be graded based on the type of the data to be processed. For example, a mapping relationship between the type of data and the level of the data can be established, that is, different data types correspond to different data levels. For example, for three types of data such as sample data, log data and performance data, the level of the sample data can be set to the lowest (the third level), the level of the log data is the second level, and the level of the performance data is the highest (the first level). After obtaining the data to be processed, the identifier carried can be extracted from the data to be processed, and then the type of the data to be processed can be determined according to the identifier of the data to be processed. At this time, the level of the data to be processed can be determined according to the type of the data to be processed. For example, if the data to be processed is data related to the quality of video playback, the type of the data to be processed can be determined as performance data according to the identifier extracted from the data to be processed, and the level of the data to be processed can be determined to be the first level.

[0086] S103: Allocate cache space and cache time for the data to be processed.

[0087] Among them, the cache space can be a cache space in a circular cache queue preset on the server, such as a cache space of a time wheel (also called a grid of a time wheel); or, the cache space can also be a cache space in a strip cache queue preset on the server, for example, each cache space in the strip cache queue can be linked in sequence through a pointer in a linked list; or, the cache space can also be a cache space in a discrete cache queue preset on the server, for example, each cache space in the discrete cache queue can be linked in sequence through a pointer in a linked list.

[0088] The cache time can be a time period or a time point. For example, the cache time corresponding to cache space A can be the time period from a to b, the cache time corresponding to cache space B can be the time period from b to c, the cache time corresponding to cache space C can be the time period from c to d, the cache time corresponding to cache space D can be the time period from d to e, and so on.

[0089] It should be noted that in addition to being ring-shaped, strip-shaped or discrete, the cache queue can also be flexibly set according to actual needs. The size and position of the cache space can be flexibly set according to actual needs, and the size of the cache time can be flexibly set according to actual needs. The specific content is not limited here.

[0090] In some implementations, allocating cache space and cache time to the data to be processed may include: obtaining a time interval of the cache space and a delay time of the data to be processed; and allocating cache space and cache time to the data to be processed according to the time interval and the delay time.

[0091] Specifically, a time interval can be set for each cache space in the cache queue, and the time intervals of all cache spaces in the cache queue can form a continuous time period or a discontinuous time period. Optionally, in order to ensure the continuity of the time for processing the data to be processed, the time intervals of all cache spaces in the cache queue can be set to form a continuous time period. For example, taking the cache queue as a time wheel as an example, the time wheel can be divided into 100 cache spaces, and the total time of the time wheel is set to 5 minutes. At this time, the total time can be evenly distributed to each cache space, and the cache time corresponding to each cache space is 3 seconds, that is, the time interval corresponding to the first cache space of the time wheel is from the current moment to the third second, the time interval corresponding to the second cache space is from the third second to the sixth second, and the time interval corresponding to the third cache space is from the sixth second to the ninth second. By analogy, the time interval corresponding to the 100th cache space is the last three seconds.

[0092] For another example, taking the cache queue as a discrete cache queue, multiple discrete cache spaces in the discrete cache queue can be linked in sequence according to the pointers in the linked list to obtain the first cache space to the 100th cache space, and the total time of 5 minutes can be evenly distributed to each cache space (of course, it can also be unevenly distributed), so that the cache time corresponding to each cache space is 3 seconds, that is, the time interval corresponding to the first cache space of the time wheel is from the current moment to the third second, the time interval corresponding to the second cache space is from the third second to the sixth second, and so on. The time interval corresponding to the 100th cache space is the last three seconds.

[0093] Also, a delay time of the data to be processed can be set, and the delay time can be flexibly set according to actual needs. For example, the delay time can be set to 6 seconds or 10 seconds. At this time, a cache space can be allocated for the data to be processed according to the time interval and the delay time, and the cache time corresponding to the cache of the data to be processed in the cache space can be determined. For example, the processing time of the data to be processed can be calculated according to the delay time, and the processing time can be matched with the time interval of each cache space to determine the time interval in which the processing time is located, so that the cache space in which the data to be processed can be cached can be determined, and the cache time can be determined according to the time interval of the cache space.

[0094] In some embodiments, allocating cache space and cache time for data to be processed based on time intervals and delay times may include: constructing a circular cache queue consisting of multiple cache spaces; setting a time interval for each cache space in the circular cache queue; obtaining the delay time of the data to be processed, and obtaining the sequence number of the cache space in the circular cache queue that is currently transferring data; and allocating cache space and cache time for the data to be processed based on the time interval, delay time, and sequence number.

[0095] Specifically, in order to improve the convenience and flexibility of data caching, a circular cache queue formed by multiple cache spaces can be constructed, such as a time wheel, and a time interval can be set for each cache space in the circular cache queue. For example, when the time wheel includes 100 cache spaces, the total time of the time wheel, 5 minutes, can be evenly distributed to each cache space, and the cache time corresponding to each cache space is 3 seconds, that is, the time interval corresponding to the first cache space of the time wheel is from the current moment to the third second, the time interval corresponding to the second cache space is from the third second to the sixth second, the time interval corresponding to the third cache space is from the sixth second to the ninth second, and so on. The time interval corresponding to the 100th cache space is the last three seconds.

[0096] And, the delay time of the data to be processed can be obtained, and the delay time can be flexibly set according to actual needs. For example, the delay time can be set to 6 seconds or 10 seconds. Since the data to be processed in the cache space in the circular cache queue will be transferred to the preset storage space when the cache time corresponding to the cache space is reached, the sequence number of the cache space currently transferring data in the circular cache queue can be obtained. For example, when the circular cache queue includes 100 cache spaces, these 100 cache spaces can be numbered from 1 to 100. If the cache space currently transferring data is the 9th, the sequence number of the cache space currently transferring data in the circular cache queue can be obtained as 9. At this time, the cache space can be allocated to the data to be processed according to the time interval, the delay time and the sequence number, and the cache time corresponding to the cache space can be determined. For example, according to the time interval of each cache space, it can be determined that the cache time of each cache space is 3 seconds. For example, the time interval of the cache space is from a to b, and the time span from a to b is 3 seconds, then the cache time of the cache space is 3 seconds; and, the serial number of the cache space allocated to the data to be processed = (the serial number of the cache space currently transferring data * the cache time of the cache space + the delay time) / cache time % the number of cache spaces included in the circular cache queue, that is, the data to be processed can be cached in the cache space corresponding to the serial number.

[0097] S104: Storing the data to be processed into a cache space according to the level.

[0098] After the cache space is allocated, the data to be processed can be stored in the cache space according to the level. For example, multiple queues can be set in the same cache space, and each queue stores data to be processed of different levels. For example, high-level data to be processed (i.e., important data) can be stored in an important queue, and low-level data to be processed (i.e., unimportant data) can be stored in an unimportant queue, so that the data to be processed can be extracted from the corresponding queue of the cache space according to the level of the data to be processed and transferred. Taking the time wheel as an example, two queues can be set in each grid of the time wheel. Data with high priority can be stored in the first queue, and data with low priority can be stored in the second queue. Subsequently, data to be processed with higher levels can be extracted first for transfer.

[0099] For another example, the data to be processed can be divided into data groups corresponding to the level according to the level, and the data to be processed can be classified and stored in the cache space according to the data groups to which they belong. For example, data group 1 corresponding to the first level, data group 2 corresponding to the second level, data group 3 corresponding to the third level, etc. can be stored in the cache space in sequence. Subsequently, the data to be processed in data group 1 with a higher level can be preferentially extracted for transfer, and then the data to be processed in data groups 2 and 3 can be extracted in sequence for transfer.

[0100] S105 , transferring the data to be processed from the cache space to the preset storage space in descending order of levels according to the cache time.

[0101] After the data to be processed is stored in the cache space, the server can monitor whether the cache time has arrived. For example, when the cache time is 3 seconds, it can determine whether the start time of these 3 seconds has arrived. If it has arrived, the data to be processed can be extracted from the cache space in order from high to low levels and transferred to the preset storage space; if it has not arrived, it can wait until the start time of the cache time arrives before transferring the data to be processed.

[0102] The preset storage space can be flexibly set according to actual needs. For example, the preset storage space can be a message system Kafka, which is a high-throughput distributed publish-subscribe message system that can process all action flow data in the website. The preset storage space can be a single Kafka or a Kafka cluster composed of multiple Kafkas.

[0103] In some embodiments, transferring the data to be processed from the cache space to the preset storage space in order from high to low levels according to the cache time may include: when the cache time is reached, extracting the data to be processed from the cache space in order from high to low levels to obtain target data; parsing the target data to obtain parsed target data; and sending the parsed target data to the preset storage space at a preset rate.

[0104] In order to reduce the data loss rate, the data in the cache space can be parsed and transferred to the preset storage space. For example, when the cache time is reached, the data to be processed can be extracted from the cache space in order from high to low levels to obtain the target data. For example, the highest-level data to be processed can be extracted from the highest-level queue first, and then the lower-level data to be processed can be extracted from the lower-level queues in turn, and so on.

[0105] Then, the extracted target data is parsed, for example, the protocol (IP, Internet Protoco) for interconnection between networks is parsed to obtain the parsed target data, which may include information such as the country, province, city or operator corresponding to the data. At this time, the parsed target data can be sent to a preset storage space (such as Kafka) at a preset rate. The preset rate can be flexibly set according to actual needs.

[0106] In some embodiments, after sending the parsed target data to the preset storage space at a preset rate, the data processing method may further include: when the cache time ends, determining whether there is any unextracted data to be processed in the cache space; if so, storing the unextracted data to be processed to the disk; parsing the data to be processed in the disk in order from high to low levels, and sending the parsed data to the preset storage space at a preset rate.

[0107] In order to reduce the processing pressure of the server memory on the data and the data loss rate, and improve the stability of the server's data processing, the data that has not been processed in the cache space can be temporarily stored in the disk. Specifically, in the process of extracting the data to be processed from the cache space, it can be determined whether the cache time is over. For example, when the cache time is 3 seconds, it can be determined whether the end time of these 3 seconds has arrived. If it has arrived, it can be further determined whether there is unextracted data to be processed in the cache space. If so, the unextracted data to be processed can be stored in the order from high to low to the disk. For example, the cache space caches the data to be processed of the first level, the second level and the third level. When the cache time ends, if there is unextracted data to be processed in the cache space of the third level, the data to be processed of the third level can be stored in the disk; or, when the cache time ends, if there is unextracted data to be processed in the cache space of the second level and the third level, the data to be processed can be stored in the disk in the order from the second level to the third level.

[0108] Then, the data to be processed in the disk can be parsed in order from high to low levels to obtain the parsed data. For example, the highest level of data to be processed can be extracted from the disk and parsed first, and then the lower level data to be processed can be extracted and parsed in turn, and so on. The parsing can be IP parsing, so that the parsed data containing information such as provinces, cities or operators can be obtained. At this time, the parsed data can be sent to the preset storage space (such as Kafka) at a preset rate, and the preset rate can be flexibly set according to actual needs. The reliability of the data is ensured by storing the data on the disk, and distributed storage, such as distributed file systems (HDFS, Hadoop Distributed File System) or distributed open source databases HBase, can also be used for unified data access. The rate control part can be uniformly limited through memory databases, etc., or dynamic capacity control can be performed according to the server.

[0109] It should be noted that in order to ensure the reliability of data transmission, if the data transmission fails during the process of sending data to the preset storage space, the data transmission will be retried. After multiple failed retries, the data can be included in the gray list, and the data volume sent in the gray list will be set to be smaller, and the data transmission will be downgraded until the subsequent data can be sent normally, and then the data transmission volume will be increased. By adding mechanisms such as data transmission retries and server monitoring, the reliability of the server is increased, the security of the data is improved, and the server is guaranteed to be within the tolerance range.

[0110] By utilizing cache space and disk to cache data instead of the existing method of directly reading and sending data, data peaks can be smoothed through cache space and disk, greatly improving the efficiency of data reading and writing. Data grading and caching methods are used to ensure that data is sent at a certain rate, avoiding data loss during data peaks, improving server stability, and solving the impact of data peaks on server crashes.

[0111] The embodiment of the present application can obtain the data to be processed, determine the level of the data to be processed, and allocate cache space and cache time for the data to be processed; then the data to be processed can be stored in the cache space according to the level, and the data to be processed can be transferred from the cache space to the preset storage space in order of level from high to low according to the cache time. This solution can transfer the data to be processed from the cache space to the preset storage space based on the level of the data to be processed and the cache time, avoiding data loss caused by too much data to be processed being obtained at the same time and not being processed in time, thereby improving the reliability and flexibility of data processing.

[0112] The method described in the above embodiment is further described in detail below with examples.

[0113] See also Figure 3 , Figure 3 The data processing method provided in the embodiment of the present application is a flow chart. The data processing method can be applied to a server, taking the data processing of the server as an example, such as Figure 4 As shown, the server can receive the data to be processed through the honeycomb system BOSS, read the data in the BOSS, classify the data, and then store the classified data in the time wheel, and perform IP resolution on the data in the time wheel according to the preset strategy, and send the resolved data to the message system Kafka at a preset rate, or store the data in the time wheel in a disk Disk, and then perform IP resolution on the data in the Disk, and send the resolved data to the message system Kafka at a preset rate. The following will explain in detail. Figure 3 The method flow may include:

[0114] S201. The server receives data reported by the client through the honeycomb system BOSS.

[0115] Among them, the data may include sampling data, log data or performance data, etc. For example, the sampling data may include error information that occurs during the client's audio and video playback process, the log data may include log-related data such as video transcoding, video frequency band, or video evaluation, and the performance data may include performance-related data such as video playback traffic, video playback speed, clarity, etc.

[0116] S202: The server obtains the type of data, and classifies the data according to the type of data to obtain the level of the data.

[0117] Since different data have different importance, in order to ensure that more important data can be processed first, the server can classify the data into levels. The higher the level of the data, the more important the data is. Conversely, the lower the level of the data, the less important the data is.

[0118] In order to improve the convenience of data classification, the server can classify data based on the type of data. For example, a mapping relationship between the type of data and the level of data can be established, that is, different data types correspond to different data levels. After receiving the data, the server can determine the type of data based on the identifier of the data, and determine the level of the data based on the type of data. For example, for three types of data such as sample data, log data and performance data, the level of the sample data can be set to the third level, the level of the log data can be set to the second level, and the level of the performance data can be set to the first level. If the type of the received data is log data, the level of the data can be determined to be the second level. For another example, if the received data is data related to video playback quality, the type of the data can be determined to be performance data based on the identifier extracted from the data, and the level of the data can be determined to be the first level.

[0119] S203: The server allocates cache space for the data from a preset time wheel, and determines a cache time corresponding to the cache space.

[0120] In order to improve the convenience and flexibility of data caching, the server can construct a time wheel formed by multiple cache spaces, for example Figure 5As shown, a time interval is set for each cache space in the time wheel. For example, when the time wheel includes 100 cache spaces (also called grids), the total time of the time wheel of 5 minutes can be evenly distributed to each cache space, and the cache time corresponding to each cache space is 3 seconds, that is, the time interval corresponding to the first cache space of the time wheel is from the current moment to the third second, the time interval corresponding to the second cache space is from the third second to the sixth second, the time interval corresponding to the third cache space is from the sixth second to the ninth second, and so on. The time interval corresponding to the 100th cache space is the last three seconds.

[0121] And, the server can obtain the delay time of the data, which can be flexibly set according to actual needs. For example, the delay time can be set to 10 seconds. Since the data in the cache space will be transferred to Kafka when the cache time corresponding to the cache space in the time wheel arrives, the sequence number of the cache space currently transferring data in the time wheel (i.e., the current grid) can be obtained. For example, when the time wheel includes 100 cache spaces, these 100 cache spaces can be numbered from 1 to 100. If the cache space currently transferring data is the 9th, the sequence number of the cache space currently transferring data in the time wheel can be obtained as 9. At this time, the cache space can be allocated to the data according to the time interval, the delay time and the sequence number, and the cache time corresponding to the cache space can be determined. Thus, the peak data can be evenly distributed to different time periods (i.e., the cache time is allocated), so that it can be evenly sent to Kafka in the subsequent memory cache and disk cache, etc., wherein the ability of Kafka to receive data with average data at a minute granularity is greater than the total amount of data that needs to be written per minute at this time.

[0122] For example, taking the time wheel as an example, the time wheel is a time wheel that stores data, and the bottom layer is composed of a linked list. The time wheel is composed of multiple grids (i.e., cache space) with corresponding time settings, and each grid represents the basic time span of the current time (i.e., cache time). The total number of grids on the time wheel can be represented by wheelSize, and the overall time span (interval) of the entire time wheel can be calculated by the formula of the basic time span tickMs corresponding to each grid × the total number of grids wheelSize. The time wheel can also be set with a dial pointer (currentTime) to indicate the current time of the time wheel. The dial pointer currentTime is an integer multiple of the basic time span tickMs. The dial pointer currentTime can divide the entire time wheel into an expired part and an unexpired part. The grid currently pointed to by the dial pointer currentTime belongs to the expired part, indicating that the grid has just expired and all tasks TimerTaskList corresponding to this grid need to be processed.

[0123] Take the total time of the time wheel as 5 minutes, every 3 seconds as the basic time span of a grid, and the time wheel has a total of 100 grids as an example, the grid where data is inserted into the time wheel is: (delayMs+startMs) / tickMs%wheelSize, where delayMs represents the delay time, startMs represents the serial number of the grid currently pointed to by the dial pointer, and wheelSize represents the total number of grids of the time wheel. When data is inserted, the delay time allowed by the data is modulo the current dial pointer. For example, if the delay time is 10 seconds, the serial number of the current grid is 5, the time wheel includes 100 grids, and the basic time span of each grid is 3 seconds, then the grid where data can be inserted is: (5*3+10) / 3%100=8, that is, the data is stored in the 8th grid.

[0124] S204: The server stores the data into the allocated cache space in the time wheel according to the level of the data.

[0125] After allocating the cache space, the server can store the data into the allocated cache space in the time wheel according to the level of the data, for example, Figure 5 As shown, high-level data (i.e., very important data) can be stored in the important queue of the cache space, and low-level data (i.e., important data) can be stored in the unimportant queue of the cache space, so that the data can be extracted from the corresponding queue of the cache space and transferred according to the data level. That is, by adopting a data classification method, two queues can be stored in each grid of the time wheel, and high-priority data is placed in the higher-level queue, and low-priority data is placed in the lower-level queue.

[0126] In this embodiment, data can be classified into important and unimportant data according to data classification, and important data can be put into an important queue, and unimportant data can be put into an unimportant queue. Data can be sent according to the delay time of the data, and data that cannot be sent in the future can be saved to the disk. The data in the disk can be sent through another process, and high-priority data is sent first. After IP parsing, the sending speed of the data is controlled by a preset rate controller.

[0127] Using the time wheel to save data replaces the existing method of directly reading and sending data. Taking advantage of the reading, writing and expiration of the time wheel, the efficiency of data reading and writing is greatly improved. Data classification is achieved by saving multiple queues on a single grid of the time wheel. Data classification and caching methods are used to ensure that data is sent at a certain rate, which improves the stability of the server and solves the impact of data peaks on the server.

[0128] S205. The server determines whether the cache time has been reached; if reached, step S06 is executed; if not reached, waiting is performed.

[0129] S206. The server extracts data from the cache space in descending order of levels, and performs IP resolution on the data to obtain resolved data.

[0130] After the data is stored in the cache space, the server can monitor whether the cache time has arrived. For example, when the cache time is 3 seconds, it can determine whether the start time of these 3 seconds has arrived. If it has arrived, the data can be extracted from the cache space of the time wheel in order from high to low levels (also known as data removal). After extracting the data from the cache space, the data can be IP resolved to obtain the resolved data; if it has not arrived, you can wait until the start time of the cache time arrives before transferring the data.

[0131] For data removal, the data can be indexed in the form of a doubly linked list. When removing data, you only need to point the previous pointer in the current doubly linked list corresponding to the current data to the next data, and the pointer of the next data to the previous one. Of course, you can also remove a batch of data at one time.

[0132] Regarding data priority, in order to ensure the reliability of all data, all data that needs to be sent will be sent to the backend Kafka cluster. However, some data have high real-time requirements, while some are not so high and a certain delay is allowed. Therefore, the data can be extracted from the cache space in order from high to low levels and IP resolution can be performed so that the resolved data can be sent to Kafka.

[0133] S207: The server sends the parsed data to the message system Kafka at a preset rate.

[0134] In order to improve the stability of the server, reduce the data loss rate, and solve the scenario where the server crashes due to data peaks at the second level, the data is saved on the time wheel. The advantages of the time wheel such as reading, writing and expiration can be used to make the request response of a large amount of data able to be responded and processed in time, thereby improving the efficiency of data processing. In addition, data is sent at a stable speed (i.e. the preset rate) to ensure that the server will not crash due to data peaks at the second level.

[0135] Since the reported data is valuable, the current system operation status can be determined through real-time data processing and analysis. Since the data middleware BOSS has a limited data storage time, if the data processing is delayed, data loss will occur. Therefore, the BOSS data needs to be connected to the Kafka message middleware according to needs for subsequent analysis and processing.

[0136] To connect data from BOSS to Kafka, you need to subscribe to BOSS data, read messages in real time, and write them to Kafka through the Kafka interface. The data subscription method is a push method. BOSS pushes data to the BOSSAgent of the subscribed host. After receiving the data, the BOSS Agent can write the data to the shared memory. Other services read the data according to the application programming interface (API). For example, you can read data through the BOSS API, then process the data (such as IP resolution, filtering or sampling, etc.), and then write the processed data to the Kafka cluster.

[0137] First, read the data in the shared memory, then perform IP resolution on the read data, and write the resolved data to Kafka. Each piece of data will have a certain processing time (such as reading, parsing and sending, etc.), which belongs to a data processing joint commissioning, so each piece of data must be processed for a certain period of time before it can reach Kafka. During low data peaks, data processing is relatively good. During data peaks, the time to write to Kafka will slow down a lot (when the Kafka load is high, writing will slow down), resulting in data that cannot be written in time. If the data of the BOSS system is not read in time, it will be lost. Moreover, during data peaks, all data will be written to the Kafka cluster. Kafka will not be able to complete the processing at the first moment due to disk or network reasons, which will cause memory overflow, process hang, background cluster size decreases, overall load increases, and cluster avalanche is prone to occur. In order to avoid data loss in this application, data can be cached to the time wheel and transferred to Kafka through the time wheel. In addition, the use of the time wheel can not only make data reading and writing very fast and have low time complexity, but also have a stable time count so that the server can know how long each batch of data has expired.

[0138] S208. The server determines whether the cache time has ended. If so, execute step S209; if not, execute step 206.

[0139] S209. The server determines whether there is any unextracted data in the cache space; if so, execute step S210; if not, execute step S212. When the cache time of the next cache space of the time wheel arrives, perform IP resolution on the data in the next cache space, and send the resolved data to the message system Kafka at a preset rate.

[0140] S210: The server stores the unextracted data to the disk.

[0141] The server can temporarily store the data in the cache space that has not been processed in time to the disk Disk, that is, serialize the expired data to the disk. Specifically, in the process of extracting data from the cache space, it can be determined whether the cache time has ended. For example, when the cache time is 3 seconds, it can be determined whether the end time of these 3 seconds has arrived. If it has arrived, it can be further determined whether there is any unextracted data in the cache space. If so, the unextracted data can be stored in order from high to low to the disk. For example, the cache space caches data of the first level, the second level, and the third level. When the cache time ends, if there is unextracted data of the second level and the third level in the cache space, the data can be stored in the disk in order from the second level to the third level.

[0142] For data processing performance, the disk write speed is relatively high, for example, 200MB / s, which is higher than the peak data read volume. Therefore, even if the data cannot be written to the Kafka cluster, the data can be saved and written to the disk sequentially in a short time.

[0143] After the data is stored in the disk, a separate process will read the data from the disk in descending order, perform IP resolution and other operations on the read data, and measure it according to the real-time data sent, and send the parsed data to the Kafka cluster at a uniform speed. The average and stable way of sending data to Kafka avoids the situation of Kafka downtime when the periodic data peak with a granularity of seconds comes, and the data that cannot be sent in time is cached twice in the disk, etc., which greatly increases the stability of the background service.

[0144] It should be noted that for data expiration, the expiration time of all data in the same cache space in the time wheel is consistent. As long as the cache time corresponding to the cache space ends, all the remaining data in the cache space will be marked as expired and expired data processing will be performed, that is, the expired data will be stored on the disk, so that the reliability of the data can be guaranteed by saving the data on the disk.

[0145] S211. The server performs IP resolution on the data in the disk in descending order of levels, and sends the resolved data to the message system Kafka at a preset rate.

[0146] The server can parse the data in the disk in descending order to obtain the parsed data. For example, the highest-level data can be extracted from the disk first, and then the lower-level data can be extracted in sequence, and so on. The extracted data can be parsed by IP, and the parsed data can be sent to Kafka at a preset rate, which can be flexibly set according to actual needs.

[0147] This embodiment constructs an intermediate cache with a time wheel as data transfer, caches the data read from the honeycomb system BOSS into the time wheel, and then caches the data to disk or sends it to Kafka according to the cache time of the data, ensuring that the data can be evenly written to Kafka to avoid the occurrence of data transmission peaks in a short period of time. In addition, the preservation of tasks in the time wheel is changed to the preservation of data, and according to the characteristics of the time wheel itself, the data is cached in memory, cached on disk, prioritized, operated, and monitored by background services, etc., to achieve the purpose of priority data arrival, reduce the possibility of background service downtime, send data at a uniform speed, ensure that the peak data at the second level is smoothly sent to the background service (such as Kafka), reduce the probability of data loss, and increase the stability of server operation.

[0148] It is understandable that in addition to applying the time wheel method to the data transfer process, it can also be used in the recommendation system. The data to be recommended by the recommendation system is placed in the time wheel, and then the data priority and data elimination (if the real-time requirements of the recommendation system are very high, the data elimination will directly return to the default recommendation) are used to clean up the response regularly to ensure the data response speed.

[0149] It should be noted that for the monitoring and recovery mechanism of the server, when a node in the Kafka cluster hangs up, the current node will be unable to write data, which will trigger the re-election of the master, that is, choose to send data to another Kafka node. In addition, the server can regularly monitor whether the Kafka service exists. If it does not exist, remove the Kafka service sending list, and then check it regularly, and then send it again after it can be sent again. Among them, after a Kafka in the Kafka cluster crashes, it will restart. After waiting for a long enough time to restart, the Kafka service can be restored. In addition, in order to ensure operation and maintainability, it is necessary to ensure that the total number of Kafka in the server is multiple. Even if one Kafka fails, the remaining Kafka can still receive data.

[0150] The server of the embodiment of the present application classifies the received data according to the type, obtains the level of the data, allocates cache space for the data based on the time wheel, and determines the cache time corresponding to the cache space, and then, according to the level of the data, the data can be stored in the cache space allocated in the time wheel. When the cache time is reached, the data in the cache space are IP parsed in order from high to low in order, and the parsed data are sent to Kafka at a preset rate, and the data that has not been processed in time during the cache time is stored in the disk, the data in the disk is IP parsed, and the parsed data is sent to Kafka at a preset rate. The scheme stores the data in the time wheel, applies the writing, execution and expiration of the time wheel to the data, improves the writing and reading speed of the cache space, rotates the peak period of the data in the time dimension, and periodically stores the data that cannot be sent in time in the cache space of the time wheel of the periodic time dimension, so as to ensure the sending speed and quality of the data. In addition, data levels and caches are used to ensure that high-level data is sent first, avoiding data loss due to data peak periods. At the same time, the introduction of disk cache greatly increases data security, and data is saved on disk, reducing the data loss rate.

[0151] In order to better implement the data processing method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above data processing method. The meanings of the terms are the same as those in the above data processing method, and the specific implementation details can refer to the description in the method embodiment.

[0152] See also Figure 6 , Figure 6 A structural diagram of a data processing device provided in an embodiment of the present application, wherein the data processing device 300 may include an acquisition unit 301, a determination unit 302, an allocation unit 303, a storage unit 304, and a transfer unit 305, etc.

[0153] The acquisition unit 301 is used to acquire data to be processed.

[0154] The determination unit 302 is configured to determine the level of the data to be processed.

[0155] The allocation unit 303 is used to allocate cache space and cache time for the data to be processed.

[0156] The storage unit 304 is used to store the to-be-processed data into the cache space according to the level.

[0157] The transfer unit 305 is used to transfer the to-be-processed data from the cache space to a preset storage space in descending order of levels according to the cache time.

[0158] In some implementations, the determination unit 302 is specifically configured to: obtain an identifier carried in the data to be processed; determine a type of the data to be processed according to the identifier; and determine a level of the data to be processed according to the type.

[0159] In some implementations, the allocation unit 303 may include an acquisition module and an allocation module, etc., which may be specifically as follows:

[0160] An acquisition module, used to acquire the time interval of the cache space and the delay time of the data to be processed;

[0161] An allocation module is used to allocate cache space and cache time for the data to be processed according to the time interval and delay time.

[0162] In some embodiments, the allocation module is specifically used to: construct a circular cache queue formed by multiple cache spaces; set a time interval for each cache space in the circular cache queue; obtain the delay time of the data to be processed, and obtain the serial number of the cache space in the circular cache queue that is currently transferring data; and allocate cache space and cache time to the data to be processed according to the time interval, delay time and serial number.

[0163] In some embodiments, the transfer unit 305 is specifically used to: when the cache time is reached, extract the data to be processed from the cache space in descending order to obtain target data; parse the target data to obtain parsed target data; and send the parsed target data to a preset storage space at a preset rate.

[0164] In some embodiments, the transfer unit 305 is also specifically used to: when the cache time ends, determine whether there is any unextracted data to be processed in the cache space; if so, store the unextracted data to be processed to the disk; parse the data to be processed in the disk in order from high to low levels, and send the parsed data to the preset storage space at a preset rate.

[0165] In some implementations, the acquisition unit 301 is specifically configured to receive sample data, log data, or performance data reported by a client or other server.

[0166] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0167] In the embodiment of the present application, the acquisition unit 301 can acquire the data to be processed, the determination unit 302 can determine the level of the data to be processed, and the allocation unit 303 can allocate cache space and cache time for the data to be processed; then the storage unit 304 can store the data to be processed in the cache space according to the level, and the transfer unit 305 can transfer the data to be processed from the cache space to the preset storage space in order from high to low levels according to the cache time. This solution can transfer the data to be processed from the cache space to the preset storage space based on the level of the data to be processed and the cache time, thereby improving the reliability and flexibility of data processing.

[0168] The present application also provides a server, such as Figure 7 As shown, it shows a schematic diagram of the structure of the server involved in the embodiment of the present application, specifically:

[0169] The server may include one or more processing core processors 401, one or more computer-readable storage media memories 402, a power supply 403, an input unit 404 and other components. Those skilled in the art will appreciate that Figure 7 The server structure shown in the figure does not constitute a limitation on the server, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0170] in:

[0171] The processor 401 is the control center of the server, and uses various interfaces and lines to connect various parts of the entire server. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the server and processes data, thereby performing overall detection of the server. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0172] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0173] The server also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions. The power supply 403 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0174] The server may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0175] Although not shown, the server may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the server will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby performing the following operations:

[0176] Acquire the data to be processed; determine the level of the data to be processed; allocate cache space and cache time for the data to be processed; store the data to be processed in the cache space according to the level; transfer the data to be processed from the cache space to the preset storage space in order of level from high to low according to the cache time.

[0177] In some implementations, when determining the level of the data to be processed, the processor 401 further executes: obtaining an identifier carried in the data to be processed; determining the type of the data to be processed according to the identifier; and determining the level of the data to be processed according to the type.

[0178] In some embodiments, when allocating cache space and cache time for data to be processed, the processor 401 further executes: obtaining a time interval of the cache space and a delay time of the data to be processed; and allocating cache space and cache time for the data to be processed according to the time interval and the delay time.

[0179] In some embodiments, when allocating cache space and cache time for data to be processed based on time intervals and delay times, the processor 401 also executes: constructing a circular cache queue formed by multiple cache spaces; setting a time interval for each cache space in the circular cache queue; obtaining the delay time of the data to be processed, and obtaining the sequence number of the cache space in the circular cache queue that is currently transferring data; allocating cache space and cache time for the data to be processed based on the time interval, delay time and sequence number.

[0180] In some embodiments, when transferring the data to be processed from the cache space to the preset storage space in order from high to low levels according to the cache time, the processor 401 also executes: when the cache time is reached, extracting the data to be processed from the cache space in order from high to low levels to obtain target data; parsing the target data to obtain parsed target data; and sending the parsed target data to the preset storage space at a preset rate.

[0181] In some embodiments, after sending the parsed target data to the preset storage space at a preset rate, the processor 401 further executes: when the cache time ends, determining whether there is any unextracted data to be processed in the cache space; if so, storing the unextracted data to be processed to the disk; parsing the data to be processed in the disk in order from high to low levels, and sending the parsed data to the preset storage space at a preset rate.

[0182] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the data processing method above, and will not be repeated here.

[0183] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0184] To this end, an embodiment of the present application provides a storage medium in which a computer program is stored, and the computer program can be loaded by a processor to execute any data processing method provided in the embodiment of the present application. For example, the computer program is loaded by the processor and can execute the following steps:

[0185] Acquire the data to be processed; determine the level of the data to be processed; allocate cache space and cache time for the data to be processed; store the data to be processed in the cache space according to the level; transfer the data to be processed from the cache space to the preset storage space in order of level from high to low according to the cache time.

[0186] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0187] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0188] Since the instructions stored in the storage medium can execute the steps in any data processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any data processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0189] The above is a detailed introduction to a data processing method, device, server and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method, characterized in that: include: Get the data to be processed; Determining the level of the data to be processed; Allocating cache space and cache time for the data to be processed; storing the data to be processed into the cache space according to the level; According to the cache time, the data to be processed are transferred from the cache space to a preset storage space in descending order of levels; The allocating cache space and cache time for the data to be processed includes: Acquire the time interval of the cache space and the delay time of the data to be processed; Allocate cache space and cache time for the data to be processed according to the time interval and delay time; The allocating cache space and cache time for the to-be-processed data according to the time interval and the delay time comprises: Construct a circular cache queue formed by multiple cache spaces; Setting a time interval for each cache space in the circular cache queue; Obtaining the delay time of the data to be processed, and obtaining the sequence number of the cache space in the ring cache queue that is currently transferring data; Cache space and cache time are allocated to the data to be processed according to the time interval, delay time and sequence number.

2. The data processing method according to claim 1, characterized in that: Determining the level of the data to be processed includes: Obtaining an identifier carried in the data to be processed; Determining the type of the data to be processed according to the identifier; The level of the data to be processed is determined according to the type.

3. The data processing method according to claim 1, characterized in that: The transferring the to-be-processed data from the cache space to the preset storage space in descending order of levels according to the cache time includes: When the cache time is reached, extracting the data to be processed from the cache space in descending order of levels to obtain target data; Parsing the target data to obtain parsed target data; The parsed target data is sent to a preset storage space at a preset rate.

4. The data processing method according to claim 3, characterized in that: After sending the parsed target data to a preset storage space at a preset rate, the method further includes: When the cache time ends, determining whether there is any unextracted data to be processed in the cache space; If so, storing the unextracted data to be processed to the disk; The data to be processed in the disk is parsed in order of high to low levels, and the parsed data is sent to a preset storage space at a preset rate.

5. The data processing method according to any one of claims 1 to 4, characterized in that: The obtaining of data to be processed comprises: Receive sample data, log data, or performance data reported by clients or other servers.

6. A data processing device, characterized in that: include: An acquisition unit, used for acquiring data to be processed; A determination unit, used to determine the level of the data to be processed; An allocation unit, configured to allocate cache space and cache time for the data to be processed; A storage unit, used for storing the to-be-processed data into the cache space according to the level; A transfer unit, used for transferring the to-be-processed data from the cache space to a preset storage space in descending order of levels according to the cache time; The distribution unit comprises: An acquisition module, used to acquire the time interval of the cache space and the delay time of the data to be processed; An allocation module, configured to allocate a cache space and a cache time for the data to be processed according to the time interval and the delay time; The allocation module is used to: Construct a circular cache queue formed by multiple cache spaces; Setting a time interval for each cache space in the circular cache queue; Obtaining the delay time of the data to be processed, and obtaining the sequence number of the cache space in the ring cache queue that is currently transferring data; Cache space and cache time are allocated to the data to be processed according to the time interval, delay time and sequence number.

7. A server, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and the processor executes the data processing method according to any one of claims 1 to 5 when calling the computer program in the memory.

8. A storage medium, characterized in that: The storage medium is used to store a computer program, and the computer program is loaded by a processor to execute the data processing method according to any one of claims 1 to 5.

Images