Stream data transfer processing method, device and server

Abstract

The embodiment of the present invention provides a method, device and server for transferring and processing stream data, the method comprising: receiving stream data sent in real time by a distributed message system; writing the stream data into a group set corresponding to the subject to which the stream data belongs, the group set being deployed in a memory database; the group set comprising a plurality of groups, each group having a group number, and the maximum storage capacity of each group being the same; when the stream data is stored in each group, after a group with a group number is full, a group with the next group number is stored; scanning the groups to be transferred in the group set corresponding to the subject to which the stream data belongs according to a preset time interval; if the groups to be transferred have reached the maximum storage capacity, reading the stream data in the groups to be transferred from the memory database and transferring them to a relational database. This can reduce the access to the relational database, and prevent the failure of the stream data transfer due to temporary failures, thereby ensuring the stable operation of the system.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of big data technology, and in particular to a method, device and server for streaming data transfer processing. Background Art

[0002] With the rise of big data, stream computing has become a common computing model. Stream computing usually includes real-time data computing. Compared with batch processing, it has advantages such as low latency and good window computing. However, it also has disadvantages such as high requirements for computer program performance, difficulty in realizing the association of multiple stream data, and high cost of repeated execution. Therefore, in order to take advantage of the advantages of batch processing and stream computing at the same time, it is necessary to process large-capacity offline data and quickly process real-time data. Stream-batch integration frameworks such as Lambda and Kappa have been proposed. In the stream-batch integration framework, the results of stream computing (i.e., stream data) need to be stored in the database.

[0003] Currently, Storm Trident and Flink are two existing stream computing engines that can provide micro-batch computing models. However, both methods divide the stream data into small batches, which are then processed by the stream computing engine. After micro-batch processing, the stream data is transferred to the relational database in micro-batches. However, this micro-batch model transfers the data stream in micro-batches, which reduces the access to the target relational database to a certain extent, but the stream computing engine still needs to establish a direct connection with the relational data. When the network fails or the stream data reaches a peak, the relational data transfer and write failure will cause the stream data transfer to fail. Summary of the invention

[0004] The embodiments of the present invention provide a method, device and server for processing streaming data transfer, so as to overcome the problem in the prior art that when a network failure or a streaming data peak occurs, a relational data transfer write failure may occur, resulting in streaming data transfer failure.

[0005] In a first aspect, an embodiment of the present invention provides a method for processing stream data transfer, including:

[0006] Receive stream data sent in real time by a distributed messaging system through a stream computing engine; write the stream data into a group set corresponding to a topic to which the stream data belongs, wherein the group set is deployed in a memory database; the group set includes a plurality of groups, each group has a group number, and the maximum storage capacity of each group is the same; when the stream data is stored in each group, a group with a group number is stored until it is full, and then a group with the next group number is stored;

[0007] The transfer program scans the groups to be transferred in the group set corresponding to the topic to which the stream data belongs at a preset time interval; if the groups to be transferred have reached the maximum storage capacity, the stream data in the groups to be transferred are read from the memory database and transferred to the relational database.

[0008] In a possible design, storing the stream data in a grouping set corresponding to a topic to which the stream data belongs includes:

[0009] Obtain a sequence number from the memory database, wherein the sequence number includes subject information to which the stream data belongs;

[0010] Determine the group number according to the sequence number and the maximum storage capacity of the subject group corresponding to the subject information to which the stream data belongs;

[0011] According to the group number, the stream data is stored in the group corresponding to the group number until the group is full, and the step of obtaining a new serial number from the memory database is re-executed, wherein the new serial number is a serial number that is automatically incremented and updated based on the serial number.

[0012] In a possible design, the transfer program obtains the groups to be transferred in the group set corresponding to the subject to which the stream data belongs at a preset time interval; if the groups to be transferred have reached the maximum storable amount, the stream data in the groups to be transferred are read from the memory database and transferred to the relational database, including:

[0013] At a preset time interval, access the group numbers in the group set corresponding to the topic to which the stream data belongs and are marked as groups to be transferred;

[0014] If the number of storage records of the group to be transferred reaches the maximum storage capacity, the flow data in the group to be transferred is read and transferred to the relational database;

[0015] The group number of the group to be transferred is incremented to the group number of the next group, and the group number of the next group is marked as the group to be transferred, and the step of determining that the group number in the group set corresponding to the topic to which the stream data belongs is marked as the group to be transferred is repeated.

[0016] In a possible design, the dump program is deployed in a cluster, and the method further includes:

[0017] Obtaining the distributed lock of the topic to which the group to be transferred belongs from the memory database;

[0018] If the distributed lock is successfully acquired, the group to be transferred on the current date is acquired from the memory database;

[0019] Determine whether the group number of the group to be transferred is empty;

[0020] If it is empty, get the number of records in the first group of the current date;

[0021] If the number of records in the group is greater than 0, then obtain the group to be transferred on the date before the current date;

[0022] If the number of records of the group to be transferred on the date before the current date is not 0, transferring the stream data of the group to be transferred on the date before the current date to the relational database;

[0023] Delete all the flow data of the groups to be transferred on the date before the current date in the memory database;

[0024] Set the group number of the group to be transferred for the current date obtained from the memory database to 0;

[0025] Delete the obtained distributed lock of the topic to which the group to be transferred belongs.

[0026] In a possible design, after determining whether the group number of the group to be transferred is empty, the method further includes:

[0027] If it is not empty, the group number of the group to be transferred is recorded as the initial group number;

[0028] Acquire the number of records of the group corresponding to the initial group number from the memory database;

[0029] Determine whether the number of records reaches the maximum storable amount;

[0030] If reached, the corresponding group to be transferred is transferred to the relational database; and the flow data of the group to be transferred is deleted from the memory database, and the initial group number is incremented by 1, and the step of obtaining the number of records of the group corresponding to the initial group number from the memory database is repeated until it is determined that the number of records of the group to be transferred is less than the maximum storable amount.

[0031] In a possible design, after determining that the number of records of the group to be transferred is less than the maximum storable amount, the method further includes:

[0032] Determine whether the group number of the group to be transferred is greater than the initial group number;

[0033] If not, the sequence number of the current stream data is obtained from the memory database, and the current real group number is determined according to the sequence number and the maximum storage capacity of the topic group corresponding to the topic information to which the stream data belongs;

[0034] If the group number of the group to be transferred is smaller than the current real group number and the number of records of the group to be transferred is 0, the group number of the group to be transferred is cumulatively increased by 1 until the group number of the group to be transferred is larger than the real group number;

[0035] Determine whether the group number of the group to be transferred is greater than the initial group number;

[0036] If so, the group number of the group to be transferred is updated in the memory database, so as to transfer the updated flow data of the group number of the group to be transferred to the relational database.

[0037] In a second aspect, an embodiment of the present invention provides a stream data transfer processing device, including:

[0038] A packet cache module is used to receive stream data sent in real time by a distributed messaging system through a stream computing engine; write the stream data into a packet set corresponding to the topic to which the stream data belongs, wherein the packet set is deployed in a memory database; the packet set includes a plurality of packets, each packet has a packet number, and the maximum storage capacity of each packet is the same; when the stream data is stored in each packet, a packet with a packet number is stored after it is full, and then a packet with the next packet number is stored;

[0039] The group transfer module is used to scan the groups to be transferred in the group set corresponding to the topic to which the stream data belongs according to a preset time interval through a transfer program; if the groups to be transferred have reached the maximum storage capacity, the stream data in the groups to be transferred are read from the memory database and transferred to the relational database.

[0040] In a third aspect, an embodiment of the present invention provides a server, including: at least one processor and a memory;

[0041] The memory stores computer-executable instructions;

[0042] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the stream data transfer processing method described in the first aspect and various possible designs of the first aspect.

[0043] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the streaming data transfer processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0044] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the streaming data transfer processing method described in the first aspect and various possible designs of the first aspect.

[0045] The embodiment of the present invention provides a method, device and server for processing stream data transfer. The method caches the stream data to be transferred into a memory database according to groups, wherein each group has a group number and the maximum storage capacity of each group is the same. When the stream data is stored in each group, the group with the next group number is stored after the group with a group number is full; then the stream data of the group cached in the memory database is scanned according to a preset time interval, and when each group is full, the stream data of the group is transferred to a relational database. The embodiment of the present invention can cache the stream data in micro-batches, and then transfer the micro-batch cached stream data to a relational database in micro-batches, which can reduce the access to the relational database and prevent the failure of stream data transfer due to temporary failures. As long as the transfer program is restarted after the failure, the micro-batch transfer can be continued, thereby ensuring the stable operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 A schematic diagram of the system architecture for stream data transfer processing provided by an embodiment of the present invention;

[0048] Figure 2 Schematic diagram of the process of the stream data transfer processing method provided by the embodiment of the present invention Figure 1 ;

[0049] Figure 3 Schematic diagram of the process of the stream data transfer processing method provided by the embodiment of the present invention Figure 2 ;

[0050] Figure 4 Schematic diagram of the process of the stream data transfer processing method provided by the embodiment of the present invention Figure 3 ;

[0051] Figure 5 Schematic diagram of the process of the stream data transfer processing method provided by the embodiment of the present invention Figure 4 ;

[0052] Figure 6 A schematic diagram of the structure of a stream data transfer processing device provided by an embodiment of the present invention;

[0053] Figure 7 A schematic diagram of the hardware structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] At present, if the stream data is only the summary data of certain dimensions, the data volume is not particularly large, and the business needs can be met by querying with specific keywords or keyword combinations, we can write the stream data into the in-memory database. If the stream data is detailed data, and the application system needs to perform a combination query of multiple conditions on these stream data in real time, these stream data need to be transferred and stored in a relational database in real time. At present, for the stream data in the financial system (such as transaction detail data), a distributed messaging system such as Kafka is usually used to build a real-time data bus. The front-end business system that generates real-time stream data sends messages to Kafka, and the back-end analysis system that performs real-time analysis and monitoring of the data obtains stream data messages from Kafka. However, when the amount of stream data is relatively large, if the real-time stream data with relatively large data is obtained from Kafka and directly saved in a relational database, it will cause great pressure on the access to the relational database, thereby affecting the performance and stability of the system. In addition, if the acquisition of stream data includes a series of stream computing processes, when the stream computing engine is usually used to integrate the real-time computing data sent by Kafka, it is also necessary to consider that the processing speed of the stream computing engine cannot keep up with the real-time computing data sent by Kafka, which may easily lead to stream computing timeouts, or even retransmission of stream data or business logic errors. For example, the stream data processed by the stream computing engine usually includes the calculation of multi-dimensional summary data, and data retransmission may easily cause errors in the summary calculation results. In summary, when the real-time data volume of stream data is large, it is not suitable to save the stream data in real time to a relational database; and when the stream computing engine is used, it is considered that the processing speed of the stream computing engine cannot keep up with the real-time computing data sent by Kafka, which may easily lead to stream computing timeouts. In order to solve the above problems, a more reasonable solution is to transfer the stream data to a relational database in a micro-batch (micro-batch) manner. However, Storm Trident and Flink, which can provide micro-batch computing models, are available in the existing stream computing engines. However, both methods divide the stream data into small batches, which are then processed by the stream computing engine, and the stream data is transferred to the relational database after micro-batch processing. However, this micro-batch model transfers data streams in micro-batches, which reduces the access to the target relational database to a certain extent, but still requires the stream computing engine to establish a direct connection with the relational database. When network failure or peak flow data causes relational data transfer and write failure, the flow data transfer will fail.

[0056] In order to solve the above technical problems, the present invention provides the following technical ideas: the stream data is cached in micro-batches according to the groups, wherein the stream data groups are cached in the memory database; then the groups of the stream data cached in the micro-batches in the memory database are transferred to the relational database in micro-batches. This can reduce the access to the relational database and prevent the failure of the stream data transfer due to temporary failures. As long as the cached stream data groups are continuously transferred in micro-batches after the failure, the stable operation of the system is guaranteed.

[0057] Figure 1 Schematic diagram of the system architecture for stream data transfer processing provided by an embodiment of the present invention. Figure 1 As shown, the system provided in this embodiment includes a terminal 101 and a server 102. The terminal 101 may be a mobile phone, a tablet, a personal computer, etc. This embodiment does not impose any particular restrictions on the implementation of the terminal 101, as long as the terminal 101 can interact with the user for input and output. The server 102 may be a single server or a cluster of several servers.

[0058] Figure 2 Schematic diagram of the process of the stream data transfer processing method provided by the embodiment of the present invention Figure 1 , the execution subject of this embodiment can be Figure 1 The server of the embodiment shown in the figure is not particularly limited in this embodiment. Figure 2 As shown, the method includes:

[0059] S201: Receive stream data sent in real time by a distributed messaging system through a stream computing engine; write the stream data into a group set corresponding to the topic to which the stream data belongs, wherein the group set is deployed in a memory database; the group set includes a plurality of groups, each group has a group number, and the maximum storage capacity of each group is the same; wherein when the stream data is stored in each group, after a group with a group number is full, a group with the next group number is stored.

[0060] In this embodiment, the stream data received by the stream computing engine from the distributed messaging system in real time may be stream data sent in real time by a distributed messaging system such as Kafka.

[0061] The in-memory database may be a Redis in-memory database.

[0062] In this embodiment, the stream data sent by different distributed messaging systems may correspond to one topic, and the stream data sent by each distributed messaging system is stored in a grouping set of the topic corresponding to the distributed messaging system.

[0063] The group set includes multiple groups, each group has a group number, and the maximum storage capacity of each group is the same. Figure 2The group set includes multiple groups whose group numbers are “group 1, group 2, group 3…”.

[0064] When the stream data is cached in the group set according to the group, it is stored in the order of group number from small to large. After the group of a group number is full (that is, the maximum storage capacity is reached), the group of the next group number is stored.

[0065] refer to Figure 2 , group 1 caches the flow data; when group 1 is full, group 2 caches the flow data; when group 2 is full, group 3 caches the flow data, and so on.

[0066] S202: Scan the groups to be transferred in the group set corresponding to the topic to which the stream data belongs according to a preset time interval through the transfer program; if the groups to be transferred have reached the maximum storage capacity, read the stream data in the groups to be transferred from the memory database and transfer them to the relational database.

[0067] In this embodiment, the preset time interval can be set in the transfer program, for example, 10 seconds. The maximum storage capacity of each group can be set, for example, the maximum storage capacity of each group is 100 records.

[0068] From the above description, it can be seen that the stream data to be transferred is cached in the memory database according to the group, wherein each group has a group number and the maximum storage capacity of each group is the same, wherein when the stream data is stored in each group, the group with a group number is stored after it is full, and then the stream data of the group cached in the memory database is scanned according to the preset time interval, and when each group is full, the stream data of the group is transferred to the relational database. The embodiment of the present invention can cache the stream data in micro batches, and then transfer the micro batch cached stream data to the relational database in micro batches, which can reduce the access to the relational database and will not cause the transfer of the stream data to fail due to temporary failures, as long as the transfer program is restarted after the failure to continue the micro batch transfer. In addition, when the throughput of the stream data changes significantly, it will only affect the amount of data cached in the memory database, and will not affect the stable operation of the transfer program. Due to the existence of the memory database storage space, the stability of the system is guaranteed.

[0069] Figure 3 Schematic diagram of the process of the stream data transfer processing method provided by the embodiment of the present invention Figure 2 In this embodiment, Figure 2 Based on the embodiment, the specific implementation process of storing the stream data in the group set corresponding to the subject to which the stream data belongs in the above embodiment is described in detail. Figure 3As shown, the method includes:

[0070] S301: Obtain a sequence number from the memory database, where the sequence number includes subject information to which the stream data belongs.

[0071] In this embodiment, the sequence number may be a self-incrementing sequence number obtained by the stream computing engine from the Redis memory database through the incr method, and the key parameter of the self-incrementing sequence number includes the subject information.

[0072] S302: Determine the group number according to the sequence number and the maximum storage capacity of the topic group corresponding to the topic information to which the stream data belongs.

[0073] In this embodiment, based on the sequence number, the sequence number is reduced by 1 and then divided by the maximum storable capacity (number of records) of the group to obtain the group number.

[0074] S303: According to the group number, the stream data is stored in the group corresponding to the group number until the group is full, and the step of obtaining a new serial number from the memory database is re-executed, wherein the new serial number is a serial number that is automatically incremented and updated based on the serial number.

[0075] In this embodiment, the stream data is added to the list object of the corresponding group through the rpush method of the Redis memory database.

[0076] It can be seen from the above description that by updating the self-incrementing sequence number, the group numbers of the groups are increased in an orderly manner, thereby achieving orderly storage of the stream data in groups of the same size.

[0077] In one embodiment of the present invention, since the continuous stream data is infinite, in order to prevent the sequence number from being insufficient, the daily switching of the sequence number is considered. That is, the key parameter of the self-incrementing sequence number includes not only the subject information but also the date information. When the date information of the sequence number changes, the sequence number starts counting again (for example, starts counting again from 1).

[0078] Figure 4 Schematic diagram of the process of the stream data transfer processing method provided by the embodiment of the present invention Figure 3 In this embodiment, Figure 2 Based on the embodiment, the specific implementation process of step S202 in the above embodiment is described in detail. Figure 4 As shown, the method includes:

[0079] S401: At a preset time interval, access the group numbers in the group set corresponding to the topic to which the stream data belongs, and mark them as groups to be transferred.

[0080] S402: If the number of stored records of the group to be transferred reaches the maximum storable amount, the flow data in the group to be transferred is read and transferred to the relational database.

[0081] S403: Increment the group number of the group to be transferred to the group number of the next group, and mark the group number of the next group as the group to be transferred, and repeat the step of determining that the group number in the group set corresponding to the topic to which the stream data belongs is marked as the group to be transferred.

[0082] In this embodiment, the transfer program obtains the group to be transferred from the Redis memory database, specifically, identifies the group whose group number is marked as the group to be transferred. When the number of storage records of the group to be transferred reaches the maximum storage capacity (i.e., the maximum number of records), the flow data of the group is read, and the flow data in each full group is saved in batches to the relational database, and then the flow data in the transferred group in the Redis memory database is deleted.

[0083] Figure 5 Schematic diagram of the process of the stream data transfer processing method provided by the embodiment of the present invention Figure 4 In one embodiment of the present invention, the transfer program is deployed in a cluster, and the method further includes:

[0084] Obtaining the distributed lock of the topic to which the group to be transferred belongs from the memory database;

[0085] If the distributed lock is successfully acquired, the group to be transferred on the current date is acquired from the memory database;

[0086] Determine whether the group number of the group to be transferred is empty;

[0087] If it is empty, get the number of records in the first group of the current date;

[0088] If the number of records in the group is greater than 0, then obtain the group to be transferred on the date before the current date;

[0089] If the number of records of the groups to be transferred on the date before the current date is not 0, then transferring the flow data of all the groups to be transferred on the date before the current date to the relational database;

[0090] Delete all the flow data of the groups to be transferred on the date before the current date in the memory database;

[0091] Set the group number of the group to be transferred for the current date obtained from the memory database to 0;

[0092] Delete the obtained distributed lock of the topic to which the group to be transferred belongs.

[0093] In this embodiment, if the distributed lock acquisition fails, the process ends.

[0094] If the number of records of the group is equal to 0, the obtained distributed lock of the topic to which the group to be transferred belongs is deleted, and the process ends.

[0095] In this embodiment, the in-memory database is a Redis in-memory database.

[0096] Specifically, if the number of records of the to-be-transferred groups on the date before the current date is not 0, then all the flow data of the to-be-transferred groups on the date before the current date are transferred to the relational database;

[0097] Deleting all the flow data of the groups to be transferred on the date before the current date in the memory database, specifically comprising: if the number of records of the groups to be transferred on the date before the current date is not 0, transferring the flow data of the group to the relational database, deleting the flow data of the group in the memory data, and then increasing the group number of the group to be transferred by 1, and repeatedly executing the step of transferring all the flow data of the groups to be transferred on the date before the current date to the relational database if the number of records of the groups to be transferred on the date before the current date is not 0, until the number of records of the groups to be transferred on the date before the current date is equal to 0.

[0098] From the above description, it can be seen that when the statistical date of the flow data changes, the flow data of the previous date is transferred once, so that the flow data can be segmented according to the date and the flow data can be segmented by day.

[0099] Continue to refer Figure 5 In another example of the present invention, after determining whether the group number of the group to be transferred is empty, the method further includes:

[0100] If it is not empty, the group number of the group to be transferred is recorded as the initial group number;

[0101] Acquire the number of records of the group corresponding to the initial group number from the memory database;

[0102] Determine whether the number of records reaches the maximum storable amount;

[0103] If reached, the corresponding group to be transferred is transferred to the relational database; and the flow data of the group to be transferred is deleted from the memory database, and the initial group number is incremented by 1, and the step of obtaining the number of records of the group corresponding to the initial group number from the memory database is repeated until it is determined that the number of records of the group to be transferred is less than the maximum storable amount.

[0104] In this embodiment, the in-memory database is a Redis in-memory database.

[0105] It can be seen from the above description that when the group number of the group to be transferred is not empty, batch transfer of the flow data of each current group to be transferred is performed.

[0106] Continue to refer Figure 5 In another example of the present invention, after determining that the number of records of the group to be transferred is less than the maximum storable amount, the method further includes:

[0107] Determine whether the group number of the group to be transferred is greater than the initial group number;

[0108] If not, the sequence number of the current stream data is obtained from the memory database, and the current real group number is determined according to the sequence number and the maximum storage capacity of the topic group corresponding to the topic information to which the stream data belongs;

[0109] If the group number of the group to be transferred is smaller than the current real group number and the number of records of the group to be transferred is 0, the group number of the group to be transferred is cumulatively increased by 1 until the group number of the group to be transferred is larger than the real group number;

[0110] Determine whether the group number of the group to be transferred is greater than the initial group number;

[0111] If so, the group number of the group to be transferred is updated in the memory database, so as to transfer the updated flow data of the group number of the group to be transferred to the relational database.

[0112] In this embodiment, the in-memory database is a Redis in-memory database.

[0113] In this embodiment, after determining whether the group number of the group to be transferred is greater than the initial group number, if the group number of the group to be transferred is greater than the initial group number, the group number of the group to be transferred is updated in the memory database to transfer the stream data of the updated group number of the group to be transferred to the relational database.

[0114] In this embodiment, after the group number of the group to be transferred is updated in the memory database to transfer the updated stream data of the group number of the group to be transferred to the relational database, it also includes: deleting the obtained distributed lock of the topic to which the group to be transferred belongs, and ending the process.

[0115] From the above description, we can see that by updating the group numbers of the groups that have completed the transfer of streaming data but have not yet been transferred, the problem of the group numbers to be transferred not being set successfully after the data transfer is completed is solved; at the same time, it also ensures that in some extreme cases (such as when there is a lot of data in the Redis cache after the application system is shut down), within the invalidation time of the Redis distributed lock, a transfer program has not completed the data transfer, and another transfer program starts to execute, and the streaming data will not be read and transferred repeatedly.

[0116] Figure 6 The structure diagram of the stream data transfer processing device provided by the embodiment of the present invention is as follows. Figure 6 As shown, the stream data transfer processing device 60 includes: a packet buffer module 601 and a packet transfer module 602 .

[0117] The group cache module 601 is used to receive the stream data sent in real time by the distributed message system through the stream computing engine; write the stream data into the group set corresponding to the topic to which the stream data belongs, wherein the group set is deployed in the memory database; the group set includes multiple groups, each group has a group number, and the maximum storage capacity of each group is the same; when the stream data is stored in each group, when the group with a group number is full, the group with the next group number is stored;

[0118] The group transfer module 602 is used to scan the groups to be transferred in the group set corresponding to the topic to which the stream data belongs at a preset time interval through a transfer program; if the groups to be transferred have reached the maximum storage capacity, the stream data in the groups to be transferred are read from the memory database and transferred to the relational database.

[0119] In one embodiment of the present invention, the group cache module 601 is specifically used for the process of storing the stream data in a group set corresponding to the topic to which the stream data belongs, including: obtaining a serial number from the memory database, the serial number containing the topic information to which the stream data belongs; determining the group number based on the serial number and the maximum storage capacity of the topic group corresponding to the topic information to which the stream data belongs; storing the stream data in the group corresponding to the group number based on the group number until the group is full, and then re-executing the step of obtaining a new serial number from the memory database, wherein the new serial number is a serial number that is automatically incremented and updated based on the serial number.

[0120] In one embodiment of the present invention, the group transfer module 602 is specifically used to access the group numbers in the group set corresponding to the subject to which the stream data belongs, which are marked as groups to be transferred according to preset time intervals; if the number of storage records of the group to be transferred reaches the maximum storable amount, the stream data in the group to be transferred is read and transferred to a relational database; the group number of the group to be transferred is incremented to the group number of the next group, and the group number of the next group is marked as the group to be transferred, and the step of determining that the group number in the group set corresponding to the subject to which the stream data belongs, which is marked as the group to be transferred, is the group to be transferred is repeated.

[0121] In one embodiment of the present invention, the transfer program is deployed in a cluster, and the device further includes:

[0122] The first distributed lock processing module 603 is used to obtain the distributed lock of the topic to which the group to be transferred belongs from the memory database; if the distributed lock is successfully obtained, the group to be transferred of the current date is obtained from the memory database; it is determined whether the group number of the group to be transferred is empty; if it is empty, the number of records of the first group of the current date is obtained; if the number of records of the group is greater than 0, the group to be transferred of the date before the current date is obtained; if the number of records of the group to be transferred of the date before the current date is not 0, the stream data of the group to be transferred of the date before the current date is transferred to the relational database; delete all the stream data of the group to be transferred of the date before the current date in the memory database; set the group number of the group to be transferred of the current date obtained from the memory database to 0; delete the distributed lock of the topic to which the obtained group to be transferred belongs.

[0123] The second processing module 604 of the distributed lock is used to determine whether the group number of the group to be transferred is empty, and also includes: if it is not empty, recording the group number of the group to be transferred as the initial group number; obtaining the number of records of the group corresponding to the initial group number from the memory database; determining whether the number of records reaches the maximum storable amount; if reached, transferring the corresponding group to be transferred to the relational database; and deleting the flow data of the group to be transferred from the memory database, and increasing the initial group number by 1, and repeating the step of obtaining the number of records of the group corresponding to the initial group number from the memory database until it is determined that the number of records of the group to be transferred is less than the maximum storable amount.

[0124] The third processing module 605 of the distributed lock is used for determining that the number of records of the group to be transferred is less than the maximum storable amount, and also includes: judging whether the group number of the group to be transferred is greater than the initial group number; if not, obtaining the serial number of the current stream data from the memory database, and determining the current real group number according to the serial number and the maximum storable amount of the subject group corresponding to the subject information to which the stream data belongs; if the group number of the group to be transferred is less than the current real group number and the number of records of the group to be transferred is 0, the group number of the group to be transferred is cumulatively increased by 1 until the group number of the group to be transferred is greater than the real group number; judging whether the group number of the group to be transferred is greater than the initial group number; if so, updating the group number of the group to be transferred in the memory database to transfer the stream data of the updated group number of the group to be transferred to the relational database.

[0125] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0126] Figure 7 The hardware structure diagram of the server provided in the embodiment of the present invention is shown in FIG. Figure 7 As shown, the server 70 of this embodiment includes: a processor 701 and a memory 702;

[0127] Memory 702, used to store computer-executable instructions;

[0128] The processor 701 is used to execute the computer-executable instructions stored in the memory to implement the various steps executed by the server in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0129] Optionally, the memory 702 may be independent or integrated with the processor 701 .

[0130] When the memory 702 is independently configured, the server further includes a bus 703 for connecting the memory 702 and the processor 701 .

[0131] An embodiment of the present invention further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the streaming data transfer processing method as described above is implemented.

[0132] An embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by a processor, the streaming data transfer processing method as described above is implemented.

[0133] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0134] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0135] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit. The above-mentioned module-composed unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0136] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application.

[0137] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0138] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0139] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0140] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0141] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0142] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for transferring and processing stream data, characterized in that: include: Receive streaming data sent in real time by the distributed messaging system through the stream computing engine; The stream data is written into a group set corresponding to the subject to which the stream data belongs, wherein the group set is deployed in a memory database; the group set includes a plurality of groups, each group has a group number, and the maximum storage capacity of each group is the same; when the stream data is stored in each group, after a group with a group number is full, a group with the next group number is stored; Scanning the groups to be transferred in the group set corresponding to the subject to which the stream data belongs according to a preset time interval by a transfer program; If the number of packets to be transferred has reached the maximum storable amount, the flow data in the packets to be transferred is read from the memory database and transferred to the relational database; The transfer program is deployed in a cluster, and the method further includes: Obtaining the distributed lock of the topic to which the group to be transferred belongs from the memory database; If the distributed lock is successfully acquired, the group to be transferred on the current date is acquired from the memory database; Determine whether the group number of the group to be transferred is empty; If it is empty, get the number of records in the first group of the current date; If the number of records in the group is greater than 0, obtain the group to be transferred on the date before the current date; If the number of records of the group to be transferred on the date before the current date is not 0, transferring the stream data of the group to be transferred on the date before the current date to the relational database; Delete all the flow data of the groups to be transferred on the date before the current date in the memory database; Set the group number of the group to be transferred for the current date obtained from the memory database to 0; Delete the obtained distributed lock of the topic to which the group to be transferred belongs.

2. The method according to claim 1, characterized in that: The storing the stream data in a grouping set corresponding to the subject to which the stream data belongs includes: Obtain a sequence number from the memory database, wherein the sequence number includes subject information to which the stream data belongs; Determine the group number according to the sequence number and the maximum storage capacity of the subject group corresponding to the subject information to which the stream data belongs; According to the group number, the stream data is stored in the group corresponding to the group number until the group is full, and the step of obtaining a new serial number from the memory database is re-executed, wherein the new serial number is a serial number that is automatically incremented and updated based on the serial number.

3. The method according to claim 1, characterized in that: The method comprises: obtaining, by a transfer program, a group to be transferred in a group set corresponding to a subject to which the stream data belongs at a preset time interval; if the group to be transferred has reached the maximum storage amount, reading the stream data in the group to be transferred from the memory database and transferring it to a relational database, including: At a preset time interval, access the group numbers in the group set corresponding to the topic to which the stream data belongs and are marked as groups to be transferred; If the number of storage records of the group to be transferred has reached the maximum storage capacity, the flow data in the group to be transferred is read and transferred to the relational database; The group number of the group to be transferred is incremented to the group number of the next group, and the group number of the next group is marked as the group to be transferred, and the step of accessing the group number marked as the group to be transferred in the group set corresponding to the topic to which the stream data belongs is repeated.

4. The method according to claim 1, characterized in that After determining whether the group number of the group to be transferred is empty, the method further includes: If it is not empty, the group number of the group to be transferred is recorded as the initial group number; Acquire the number of records of the group corresponding to the initial group number from the memory database; Determine whether the number of records reaches the maximum storable amount; If reached, the corresponding group to be transferred is transferred to the relational database; and the flow data of the group to be transferred is deleted from the memory database, and the initial group number is incremented by 1, and the step of obtaining the number of records of the group corresponding to the initial group number from the memory database is repeated until it is determined that the number of records of the group to be transferred is less than the maximum storable amount.

5. The method according to claim 4, characterized in that After determining that the number of records of the group to be transferred is less than the maximum storable amount, the method further includes: Determine whether the group number of the group to be transferred is greater than the initial group number; If not, the sequence number of the current stream data is obtained from the memory database, and the current real group number is determined according to the sequence number and the maximum storage capacity of the topic group corresponding to the topic information to which the stream data belongs; If the group number of the group to be transferred is smaller than the current real group number and the number of records of the group to be transferred is 0, the group number of the group to be transferred is cumulatively increased by 1 until the group number of the group to be transferred is larger than the real group number; Determine whether the group number of the group to be transferred is greater than the initial group number; If so, the group number of the group to be transferred is updated in the memory database, so as to transfer the updated flow data of the group number of the group to be transferred to the relational database.

6. A stream data transfer processing device, characterized in that: include: A grouping cache module is used to receive stream data sent in real time by a distributed messaging system through a stream computing engine; The stream data is written into a group set corresponding to the subject to which the stream data belongs, wherein the group set is deployed in a memory database; the group set includes a plurality of groups, each group has a group number, and the maximum storage capacity of each group is the same; when the stream data is stored in each group, after a group with a group number is full, a group with the next group number is stored; A packet transfer module, used to scan the packets to be transferred in the packet set corresponding to the subject to which the stream data belongs according to a preset time interval through a transfer program; If the group to be transferred has reached the maximum storable amount, the flow data in the group to be transferred is read from the memory database and transferred to the relational database; the transfer program is deployed in the cluster; A first distributed lock processing module, used for obtaining the distributed lock of the subject to which the group to be transferred belongs from the memory database; If the distributed lock is successfully acquired, the group to be transferred on the current date is acquired from the memory database; it is determined whether the group number of the group to be transferred is empty; if it is empty, the number of records of the first group of the current date is acquired; If the number of records of the group is greater than 0, the group to be transferred on the date before the current date is obtained; if the number of records of the group to be transferred on the date before the current date is not 0, the flow data of the group to be transferred on the date before the current date is transferred to the relational database; Delete all the flow data of the groups to be transferred on the date before the current date in the memory database; The group number of the group to be transferred on the current date obtained from the memory database is set to 0; and the distributed lock of the topic to which the obtained group to be transferred belongs is deleted.

7. A server, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the stream data transfer processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the stream data transfer processing method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the stream data transfer processing method according to any one of claims 1 to 5 is implemented.

Images