High-concurrency data cache sorting method and system

Abstract

The invention discloses a high-concurrency data cache sorting method and system, and the method comprises the steps: carrying out the time synchronization of a system server and a subsystem server, building a first-level cache and a second-level cache, and setting a double-buffer queue for the second-level cache; a plurality of threads receive high-concurrency data and distribute different thread groups for processing, a local timestamp and a serial number are set to be combined into a primary key, the primary key and a data value v are written into a first-level cache, and an index is established. A thread pool 1 is started to read data from a first-level cache and write the data into a second-level cache, and corresponding processing including discarding repeated data, writing the data into different buffer queues and the like is carried out by judging conditions such as a sequence number and a data value v. Meanwhile, a thread pool 2 is started to read the data from the second-level cache double-buffering queue in sequence and delete the cache, and if a specific situation is encountered, a threshold value is waited for or next data processing is continued. And the index and the cache queue content can be expanded to realize processing in a plurality of protocol modes, so that the data access efficiency is improved, the data completeness can be realized, and the conditions of data disorder and dirty data cannot occur.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data cache sorting method and system in a high-concurrency environment. Background Art

[0002] In the production environment of large-scale projects, complex multi-party system construction is faced, and corresponding coordination processing is required for protocols, interface design, etc., in order to achieve the joint operation of various subsystems and complete the corresponding production operation objectives. However, in some scenarios, because the early construction of the system did not consider more scenario expansion, the data protocol was incomplete and difficult to expand, and it was impossible to transform and rebuild it in a short time by consuming a lot of manpower, material and financial resources.

[0003] In scenarios where new technologies need to be connected to enable them, such as audio, video streams, or other high-concurrency, continuous data transmission that needs to be connected to new scenarios, defects in previous product designs, such as insufficient identification numbers, easy duplication of numbers, lack of reliable identifiers such as timestamps, data disorder, data packet loss, etc., will cause great resistance to connection and subsequent design work.

[0004] Under the existing network conditions and general technical scenarios, the use of caching, distributed caching, database writing, multi-threaded processing, etc. will have many defects due to the repeated sending of high-concurrency data, insufficient sequence numbers and easy duplication, disorder, packet loss and no reliable identification, making it impossible to achieve reliable combination and sorting of data while taking into account performance and stability.

[0005] Therefore, in order to take into account the common access and use of new and old service systems, targeted transformation and innovation are needed in architecture and design to lay a solid foundation for better updating and iterative design in the future. Summary of the invention

[0006] The present invention aims to provide a data caching and sorting method and system in a high-concurrency environment to effectively solve the above-mentioned problems.

[0007] A high-concurrency data cache sorting method comprises the following steps: 1. A high-concurrency data cache sorting method, comprising the following steps: S1. Multithreaded data reception, the received data includes: data value v and data sequence number s, setting segmentation time threshold and interval time threshold, generating local network timestamp t; S2. Combine the local network timestamp t and the data sequence number s into a primary key p(t, s), and then write the combined primary key p(t, s) and the data value v into the first-level cache to form a combined data (p(t, s), v); S3. When receiving data, start thread pool 1 at the same time, read the combined data (p(t, s), v) from the first-level cache, and write it into the current check buffer queue of the second-level cache through the interval time judgment path related to the data sequence number s, the local network timestamp t, and the interval time threshold, or determine whether to write it into another check buffer queue of the second-level cache through the comprehensive judgment path related to the data sequence number s, the local network timestamp t, the data value v, the segment time threshold, and the interval time threshold; S4. When receiving data, thread pool 2 is started at the same time, and the corresponding data sequence number s and the corresponding data value v are read from the secondary cache according to the data sorting order in the two check buffer queues; If all the data sequence numbers s of a segment are complete, the data is read according to the order in which the data in the check buffer queue is sorted and the corresponding data in the buffer queue is removed, and the data is subjected to the next step of analysis and processing; If the data sequence number s is incomplete, it is read in the order in which the data in the check buffer queue is sorted. When the data sequence number s is missing, thread pool 1 is waiting to read and analyze the corresponding data sequence number s+1 and the data value v corresponding to the data sequence number s+1. If the waiting time exceeds the interval time threshold, the data sequence number s+1 and the corresponding data value v are ignored, and the next data is taken for analysis and removal.

[0008] Furthermore, the interval time determination path is specifically as follows: S311. Start thread pool 1 and read combined data (p(t, s), v) from the first-level cache; S312. Obtaining the data sequence number s in the combined data (p(t, s), v), and the data sequence number s does not exist in any check buffer queue; S313. The current check queue is not an empty queue, otherwise go to S315; S314. Subtract the local network timestamp t of the last data in the current inspection queue from the local network timestamp t of the combined data (p(t, s), v) to obtain a difference value, and the difference value is less than or equal to the interval time threshold; S315. Write the combined data (p(t, s), v) into the current check queue and delete the combined data (p(t, s), v) in the first-level cache and go to S1.

[0009] Furthermore, the comprehensive judgment path is as follows: S321. Start thread pool 1 and read combined data (p(t, s), v) from the first-level cache. S322. Obtain the data sequence number s in the combined data (p(t, s), v), and the data sequence number s exists in any check buffer queue; S323. Obtain a data value v in the combined data (p(t, s), v), and the data value v is the same as the existing data value corresponding to the existing data sequence number; S324. Subtract the local network timestamp t of the combined data from the local network timestamp corresponding to the existing data sequence number with the same data sequence number s to obtain a time difference; S325. The time difference is greater than the segment time threshold; otherwise, go to S330; S326. The combined data (p(t, s), v) does not exist in another check buffer queue; otherwise, go to S330; S327. The other buffer queue is not an empty queue, otherwise go to S329; S328. Subtract the local network timestamp of the last data in another buffer queue from the local network timestamp of the combined data. If the difference is less than or equal to the interval time threshold, go to S329; otherwise go to S330; S329. Write the combined data (p(t, s), v) into another check buffer queue; S330. Delete the corresponding combined data (p(t, s), v) in the first-level cache and discard it, and go to S1.

[0010] Furthermore, the comprehensive judgment path is as follows: S321. Start thread pool 1 and read combined data (p(t, s), v) from the first-level cache. S322. Obtain the data sequence number s in the combined data (p(t, s), v), and the data sequence number s exists in any check buffer queue; S323. Obtain a data value v in the combined data (p(t, s), v), and the data value v is different from the existing data value corresponding to the existing data sequence number; S324. The combined data (p(t, s), v) does not exist in another check buffer queue; otherwise, go to S328; S325. The other buffer queue is not an empty queue, otherwise go to S327; S326. Subtract the local network timestamp of the last data in another buffer queue from the local network timestamp of the combined data. If the difference is less than or equal to the interval time threshold, go to S327; otherwise, go to S328. S327. Write the combined data (p(t, s), v) into another check buffer queue; S328. Delete the corresponding combined data (p(t, s), v) in the first-level cache and discard it, and go to S1.

[0011] Furthermore, the comprehensive judgment path is as follows: S321. Start thread pool 1 and read combined data (p(t, s), v) from the first-level cache. S322. Obtain the data sequence number s in the combined data (p(t, s), v), and the data sequence number s does not exist in any check buffer queue; S323. The current check queue is not an empty queue; S324. Subtract the local network timestamp of the last data in the current inspection queue from the local network timestamp of the combined data (p(t, s), v) to obtain a difference, and the difference is greater than the interval time threshold; S325. The combined data (p(t, s), v) does not exist in another check buffer queue; otherwise, go to S329; S326. The other buffer queue is not an empty queue, otherwise go to S328; S327. Subtract the local network timestamp of the last data in another buffer queue from the local network timestamp of the combined data, and when the difference is less than or equal to the interval time threshold, go to S328; S328. Write the combined data (p(t, s), v) into another check buffer queue; S329. Delete the corresponding combined data (p(t, s), v) in the first-level cache and discard it and go to S1.

[0012] Furthermore, If the data value v is large in size, compressed storage is adopted.

[0013] Furthermore, The first-level cache and the second-level cache can use local storage, virtual machine storage, or distributed cache based on the actual data volume and data complexity.

[0014] Furthermore, The primary key p and the data sequence number s are indexed by hashing or built-in indexing.

[0015] Furthermore, The segment time threshold is the time threshold between the local network timestamps of two data when the data sequence numbers s of two data segments are the same; the interval time threshold is the interval time threshold calculated between the data sequence numbers s and s+constant N in the same segment of data; the segment time threshold and the interval time threshold are constants set based on expert experience of the quality of network transmission data.

[0016] Furthermore, A high-concurrency data caching and sorting system, including subsystems, The integrated sub-server sends data to the system server, and the sub-server and system server are synchronized in time, using the network time protocol or high-precision time synchronization; The system servers include: A memory for storing instructions; The processor is used to execute the instructions, so as to implement a high-concurrency data cache sorting method.

[0017] The present invention has the beneficial effects: 1. The present invention forms a primary key p (t, s) by combining a local network timestamp t with a data sequence number s, and both the primary key p and the data sequence number s are indexed. The data value v is combined with the primary key p (t, s) to form combined data (p (t, s), v), so that a data sequence number s corresponds to a local network timestamp t and a data value v, thereby realizing rapid data positioning and further improving data retrieval efficiency.

[0018] 2. Based on multiple judgments, the present invention makes a difference in the local network timestamps t corresponding to the two identical s when the data sequence number s exists and the data value v is the same, and makes a judgment based on the obtained difference and the segmentation time threshold, thereby achieving the purpose of removing duplicate data.

[0019] 3. Determine whether the acquired data is in the same segment of the two inspection buffer queues based on the interval time threshold. If it is not in the same segment of the two inspection buffer queues, discard the acquired data to solve the problem of data disorder. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the flow chart of the method of the present invention; Figure 2 This is a schematic diagram of the first-level cache process of the method of the present invention; Figure 3 It is a schematic diagram of the thread pool 1 and the secondary cache flow of the method of the present invention; Figure 4 This is a schematic diagram of the data processing flow of the method of the present invention and thread pool 2; DETAILED DESCRIPTION

[0022] The present invention will be further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0023] Example 1 See also Figures 1 to 4 , A high-concurrency data cache sorting method comprises the following steps: S1. Multithreaded data reception, the received data includes: data value v and data sequence number s, setting segmentation time threshold and interval time threshold, generating local network timestamp t; S2. Combine the local network timestamp t and the data sequence number s into a primary key p(t, s), and then write the combined primary key p(t, s) and the data value v into the first-level cache to form a combined data (p(t, s), v); S3. When receiving data, start thread pool 1 at the same time, read the combined data (p(t, s), v) from the first-level cache, and write it into the current check buffer queue of the second-level cache through the interval time judgment path related to the data sequence number s, the local network timestamp t, and the interval time threshold, or determine whether to write it into another check buffer queue of the second-level cache through the comprehensive judgment path related to the data sequence number s, the local network timestamp t, the data value v, the segment time threshold, and the interval time threshold; S4. When receiving data, thread pool 2 is started at the same time, and the corresponding data sequence number s and the corresponding data value v are read from the secondary cache according to the data sorting order in the two check buffer queues; If all the data sequence numbers s of a segment are complete, the data is read according to the order in which the data in the check buffer queue is sorted and the corresponding data in the buffer queue is removed, and the data is subjected to the next step of analysis and processing; If the data sequence number s is incomplete, it is read in the order in which the data in the check buffer queue is sorted. When the data sequence number s is missing, thread pool 1 is waiting to read and analyze the corresponding data sequence number s+1 and the data value v corresponding to the data sequence number s+1. If the waiting time exceeds the interval time threshold, the data sequence number s+1 and the corresponding data value v are ignored, and the next data is taken for analysis and removal.

[0024] Furthermore, the interval time determination path is specifically as follows: S311. Start thread pool 1 and read combined data (p(t, s), v) from the first-level cache; S312. Obtaining the data sequence number s in the combined data (p(t, s), v), and the data sequence number s does not exist in any check buffer queue; S313. The current check queue is not an empty queue, otherwise go to S315; S314. Subtract the local network timestamp t of the last data in the current inspection queue from the local network timestamp t of the combined data (p(t, s), v) to obtain a difference value, and the difference value is less than or equal to the interval time threshold; S315. Write the combined data (p(t, s), v) into the current check queue and delete the combined data (p(t, s), v) in the first-level cache and go to S1.

[0025] Furthermore, the comprehensive judgment path is as follows: S321. Start thread pool 1 and read combined data (p(t, s), v) from the first-level cache. S322. Obtain the data sequence number s in the combined data (p(t, s), v), and the data sequence number s exists in any check buffer queue; S323. Obtain a data value v in the combined data (p(t, s), v), and the data value v is the same as the existing data value corresponding to the existing data sequence number; S324. Subtract the local network timestamp t of the combined data from the local network timestamp corresponding to the existing data sequence number with the same data sequence number s to obtain a time difference; S325. The time difference is greater than the segment time threshold; otherwise, go to S330; S326. The combined data (p(t, s), v) does not exist in another check buffer queue; otherwise, go to S330; S327. The other buffer queue is not an empty queue, otherwise go to S329; S328. Subtract the local network timestamp of the last data in another buffer queue from the local network timestamp of the combined data. If the difference is less than or equal to the interval time threshold, go to S329; otherwise go to S330; S329. Write the combined data (p(t, s), v) into another check buffer queue; S330. Delete the corresponding combined data (p(t, s), v) in the first-level cache and discard it, and go to S1.

[0026] Furthermore, the comprehensive judgment path is as follows: S321. Start thread pool 1 and read combined data (p(t, s), v) from the first-level cache. S322. Obtain the data sequence number s in the combined data (p(t, s), v), and the data sequence number s exists in any check buffer queue; S323. Obtain a data value v in the combined data (p(t, s), v), and the data value v is different from the existing data value corresponding to the existing data sequence number; S324. The combined data (p(t, s), v) does not exist in another check buffer queue; otherwise, go to S328; S325. The other buffer queue is not an empty queue, otherwise go to S327; S326. Subtract the local network timestamp of the last data in another buffer queue from the local network timestamp of the combined data. If the difference is less than or equal to the interval time threshold, go to S327; otherwise, go to S328. S327. Write the combined data (p(t, s), v) into another check buffer queue; S328. Delete the corresponding combined data (p(t, s), v) in the first-level cache and discard it, and go to S1.

[0027] Furthermore, the comprehensive judgment path is as follows: S321. Start thread pool 1 and read combined data (p(t, s), v) from the first-level cache. S322. Obtain the data sequence number s in the combined data (p(t, s), v), and the data sequence number s does not exist in any check buffer queue; S323. The current check queue is not an empty queue; S324. Subtract the local network timestamp of the last data in the current inspection queue from the local network timestamp of the combined data (p(t, s), v) to obtain a difference, and the difference is greater than the interval time threshold; S325. The combined data (p(t, s), v) does not exist in another check buffer queue; otherwise, go to S329; S326. The other buffer queue is not an empty queue, otherwise go to S328; S327. Subtract the local network timestamp of the last data in another buffer queue from the local network timestamp of the combined data, and when the difference is less than or equal to the interval time threshold, go to S328; S328. Write the combined data (p(t, s), v) into another check buffer queue; S329. Delete the corresponding combined data (p(t, s), v) in the first-level cache and discard it and go to S1.

[0028] The present invention generates a local network timestamp t while receiving data, which represents the time when the data is received, rather than the data itself. The received data includes a data value v and a data sequence number s. The combined data (p(t, s), v) is obtained in a combined form and written into a first-level cache. The first-level cache is used as a storage place for all combined data (p(t, s), v). The primary key p and the data sequence number s in the combined data (p(t, s), v) are indexed. The data value v, the local network timestamp t, and the data sequence number s correspond to each other one by one. While receiving the data, thread pool 1 is started to obtain the combined data (p(t, s), v) in the first-level cache, and the local network timestamp t, data sequence number s, and data value v are obtained. Then, the data sequence number s is queried in the two check buffer queues. If the data sequence number s does not exist in either of the two check buffer queues, it means that the data is not duplicate data. Then, the difference between the obtained local network timestamp t and the timestamp of the previous data of the current inspection queue is calculated. If the difference is less than or equal to the set interval time threshold, it means that the combined data (p(t, s), v) belongs to the same segment of data in the current inspection buffer queue, and it is written into the current inspection buffer queue; if the difference is greater than the set interval time threshold, it means that the combined data (p(t, s), v) is not the same segment of data in the current inspection queue, and it is necessary to operate on another inspection buffer queue: query the data sequence number s in another inspection buffer queue. If the data sequence number s cannot be found, the obtained local network timestamp t is used to make a difference with the timestamp of the previous data of another inspection buffer queue. If the obtained difference is less than or equal to the interval time threshold, it means that the combined data (p(t, s), v) is the same segment of data in another inspection buffer queue, and it is written into another inspection buffer queue; As long as one of the two inspection buffer queues has it, it means that the combined data (p(t, s), v) may be duplicate data. Next, it is necessary to judge the data value v and the data value corresponding to the data sequence number that is the same as the data sequence number s of the combined data (p(t, s), v) in the inspection buffer queue. If they are not the same, it is necessary to perform the above operation on the other inspection buffer queue; if the data values ​​v are the same, the local network timestamp t obtained is subtracted from the timestamp corresponding to the existing data sequence number to obtain the time difference. If the time difference is greater than the segmentation time threshold, it means that the combined data (p(t, s), v) is not the same segment data in the inspection buffer queue, and it is necessary to perform the above operation on the other inspection buffer queue; In other cases, the data will not be stored in the check buffer queue in the L2 cache, and the combined data (p(t, s), v) will be deleted from the L1 cache; While receiving data, thread pool 2 is started to read the corresponding data sequence number s and the corresponding data value v from the secondary cache according to the data sorting order in the two check buffer queues. When the data sequence s is missing, thread pool 1 is waiting to read and analyze the corresponding data sequence number s+1 and the data value v corresponding to the data sequence number s+1. If the waiting time exceeds the interval time threshold, the data sequence number s and the corresponding data value v are ignored, and the next data is taken for analysis and processing and removed from the corresponding check buffer queue.

[0029] Therefore, the present invention determines whether the combined data (p(t, s), v) is duplicate data in the inspection buffer queue with the same data sequence number through the segmentation time threshold, so as to achieve the purpose of removing duplicate data, and determines whether the combined data (p(t, s), v) is the same segment data in the inspection buffer queue through the interval time threshold, so as to avoid that data that is not the same segment is also written into the inspection buffer queue, resulting in disordered data sorting, and indexes are established for the primary key p and the data sequence number s in the combined data (p(t, s), v), so as to speed up data retrieval, and the data value v, the local network timestamp t, and the data sequence number s have a one-to-one correspondence, so that the data is more stable, and when reading data from the secondary cache, corresponding processing is made for the situation where the data sequence number is missing, so that the system is more stable.

[0030] Furthermore, If the data value v is large in size, compressed storage is adopted.

[0031] Furthermore, The first-level cache and the second-level cache can use local storage, virtual machine storage, or distributed cache based on the actual data volume and data complexity.

[0032] Furthermore, The primary key p and the data sequence number s are indexed by hashing or built-in indexing.

[0033] Furthermore, The segment time threshold is the time threshold between the local network timestamps of two data when the data sequence numbers s of two data segments are the same; the interval time threshold is the interval time threshold calculated between the data sequence numbers s and s+constant N in the same segment of data; the segment time threshold and the interval time threshold are constants set based on expert experience of the quality of network transmission data.

[0034] Furthermore, A high-concurrency data caching and sorting system, including subsystems, The integrated sub-server sends data to the system server, and the sub-server and system server are synchronized in time, using the network time protocol or high-precision time synchronization; The system servers include: A memory for storing instructions; The processor is used to execute the instructions, so as to implement a high-concurrency data cache sorting method.

[0035] Example 2 For subsystem single-channel audio data without timestamps, with data serial numbers but with a small data serial number range, and data serial numbers that are easily repeated in a short period of time, first, the system server and the subsystem server use network time protocol or high-precision time synchronization. Then, establish the first and second level caches and set up double buffer queues for the second level caches. Multithreading receives high-concurrency continuous audio streams and assigns different multithreading groups to process according to multi-channel audio. Set the segmentation time threshold and interval time threshold, and generate a local network timestamp t when receiving data.

[0036] The local network timestamp t is combined with the data sequence number s to form the primary key p (t, s). The primary key p and the data value v are written into the first-level cache to obtain the combined data (p (t, s), v). The first-level cache can use local storage, virtual machine storage or distributed cache according to the actual data volume and complexity. At the same time, p and s are indexed by hashing or built-in indexing.

[0037] At the same time, thread pool 1 is started to read data from the first-level cache in sequence and write it into the second-level cache, and determine whether the read data sequence number s exists in the second-level cache buffer queue 1 and the cache buffer queue 2. If not, write it into the buffer queue 1. If it exists, process it according to different situations. If the data value v is the same and the segment time threshold is within the segment time threshold, it is determined to be duplicate data and the data in the first-level cache is deleted and discarded. If the data value v is the same and exceeds the segment time threshold or the data value v is different, write it into the buffer queue 2 and delete the corresponding data of the first-level cache; When data sequence number s does not exist in buffer queue 1 or buffer queue 2, it is determined whether the time difference between the local network timestamp of the combined data (p(t, s), v) and the local network timestamp of the previous data exceeds the interval time threshold to decide whether to write or discard, etc., and the data is received and processed in a loop.

[0038] At the same time, thread pool 2 is started to read the corresponding data in sequence from the secondary cache double buffer queues 1 and 2 and delete the cache. If the data with sequence number s is read but the data with sequence number s+1 that does not exceed the range does not exist, wait. If the time exceeds the interval time threshold and the data is still not obtained, continue to process the next data. Otherwise, clear the data in sequence and proceed to the next round of processing.

[0039] For multi-channel audio or other data, the p index can be expanded accordingly as needed, and different cache queue contents can be expanded in the first and second level caches as needed to achieve multi-channel or multi-protocol processing.

[0040] It is understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A high-concurrency data cache sorting method, characterized in that: The following steps are involved: S1. Multithreaded data reception, the received data includes: data value v and data sequence number s, setting segmentation time threshold and interval time threshold, generating local network timestamp t; S2. Combine the local network timestamp t and the data sequence number s into a primary key p(t, s), and then write the combined primary key p(t, s) and the data value v into the first-level cache to form a combined data (p(t, s), v); S3. When receiving data, start thread pool 1 at the same time, read the combined data (p(t, s), v) from the first-level cache, and write it into the current check buffer queue of the second-level cache through the interval time judgment path related to the data sequence number s, the local network timestamp t, and the interval time threshold, or determine whether to write it into another check buffer queue of the second-level cache through the comprehensive judgment path related to the data sequence number s, the local network timestamp t, the data value v, the segment time threshold, and the interval time threshold; S4. When receiving data, thread pool 2 is started at the same time, and the corresponding data sequence number s and the corresponding data value v are read from the secondary cache according to the data sorting order in the two check buffer queues; If all the data sequence numbers s of a segment are complete, the data is read according to the order in which the data in the check buffer queue is sorted and the corresponding data in the buffer queue is removed, and the data is subjected to the next step of analysis and processing; If the data sequence number s is incomplete, it is read in the order in which the data in the check buffer queue is sorted. When the data sequence number s is missing, thread pool 1 is waiting to read and analyze the corresponding data sequence number s+1 and the data value v corresponding to the data sequence number s+1. If the waiting time exceeds the interval time threshold, the data sequence number s+1 and the corresponding data value v are ignored, and the next data is taken for analysis and removal.

2. A high-concurrency data cache sorting method according to claim 1, characterized in that: The interval time determination path is specifically as follows: S311. Start thread pool 1 and read combined data (p(t, s), v) from the first-level cache; S312. Obtaining the data sequence number s in the combined data (p(t, s), v), and the data sequence number s does not exist in any check buffer queue; S313. The current check queue is not an empty queue, otherwise go to S315; S314. Subtract the local network timestamp t of the last data in the current inspection queue from the local network timestamp t of the combined data (p(t, s), v) to obtain a difference value, and the difference value is less than or equal to the interval time threshold; S315. Write the combined data (p(t, s), v) into the current check queue and delete the combined data (p(t, s), v) in the first-level cache and go to S1.

3. A high-concurrency data cache sorting method according to claim 1, characterized in that: The comprehensive judgment path is specifically as follows: S321. Start thread pool 1 and read combined data (p(t, s), v) from the first-level cache. S322. Obtain the data sequence number s in the combined data (p(t, s), v), and the data sequence number s exists in any check buffer queue; S323. Obtain a data value v in the combined data (p(t, s), v), and the data value v is the same as the existing data value corresponding to the existing data sequence number; S324. Subtract the local network timestamp t of the combined data from the local network timestamp corresponding to the existing data sequence number with the same data sequence number s to obtain a time difference; S325. The time difference is greater than the segment time threshold; otherwise, go to S330; S326. The combined data (p(t, s), v) does not exist in another check buffer queue; otherwise, go to S330; S327. The other buffer queue is not an empty queue, otherwise go to S329; S328. Subtract the local network timestamp of the last data in another buffer queue from the local network timestamp of the combined data. If the difference is less than or equal to the interval time threshold, go to S329; otherwise go to S330; S329. Write the combined data (p(t, s), v) into another check buffer queue; S330. Delete the corresponding combined data (p(t, s), v) in the first-level cache and discard it, and go to S1.

4. A high-concurrency data cache sorting method according to claim 1, characterized in that: The comprehensive judgment path is specifically as follows: S321. Start thread pool 1 and read combined data (p(t, s), v) from the first-level cache. S322. Obtain the data sequence number s in the combined data (p(t, s), v), and the data sequence number s exists in any check buffer queue; S323. Obtain a data value v in the combined data (p(t, s), v), and the data value v is different from the existing data value corresponding to the existing data sequence number; S324. The combined data (p(t, s), v) does not exist in another check buffer queue; otherwise, go to S328; S325. The other buffer queue is not an empty queue, otherwise go to S327; S326. Subtract the local network timestamp of the last data in another buffer queue from the local network timestamp of the combined data. If the difference is less than or equal to the interval time threshold, go to S327; otherwise, go to S328. S327. Write the combined data (p(t, s), v) into another check buffer queue; S328. Delete the corresponding combined data (p(t, s), v) in the first-level cache and discard it, and go to S1.

5. A high-concurrency data cache sorting method according to claim 1, characterized in that: The comprehensive judgment path is specifically as follows: S321. Start thread pool 1 and read combined data (p(t, s), v) from the first-level cache. S322. Obtain the data sequence number s in the combined data (p(t, s), v), and the data sequence number s does not exist in any check buffer queue; S323. The current check queue is not an empty queue; S324. Subtract the local network timestamp of the last data in the current inspection queue from the local network timestamp of the combined data (p(t, s), v) to obtain a difference, and the difference is greater than the interval time threshold; S325. The combined data (p(t, s), v) does not exist in another check buffer queue; otherwise, go to S329; S326. The other buffer queue is not an empty queue, otherwise go to S328; S327. Subtract the local network timestamp of the last data in another buffer queue from the local network timestamp of the combined data, and when the difference is less than or equal to the interval time threshold, go to S328; S328. Write the combined data (p(t, s), v) into another check buffer queue; S329. Delete the corresponding combined data (p(t, s), v) in the first-level cache and discard it and go to S1.

6. A high-concurrency data cache sorting method according to claim 1, characterized in that: If the data value v is large in size, compressed storage is adopted.

7. A high-concurrency data cache sorting method according to claim 1, characterized in that: The first-level cache and the second-level cache can use local storage, virtual machine storage, or distributed cache based on the actual data volume and data complexity.

8. A high-concurrency data cache sorting method according to claim 1, characterized in that: The primary key p and the data sequence number s are indexed by hashing or built-in indexing.

9. A high-concurrency data cache sorting method according to claim 1, characterized in that: The segment time threshold is the time threshold between the local network timestamps of two data when the data sequence numbers s of two data segments are the same; the interval time threshold is the interval time threshold calculated between the data sequence numbers s and s+constant N in the same segment of data; the segment time threshold and the interval time threshold are constants set based on expert experience of the quality of network transmission data.

10. A high-concurrency data cache sorting system, including a subsystem characterized in that: The integrated sub-server sends data to the system server, and the sub-server and system server are synchronized in time, using the network time protocol or high-precision time synchronization; The system servers include: A memory for storing instructions; A processor is used to execute the instructions so as to implement any one of the high-concurrency data cache sorting methods of claims 1 to 9.

Images