Data stream processing method, device, computer readable storage medium and equipment

Abstract

The present application provides a data stream processing method, device, computer-readable storage medium and equipment. Among them, the data stream processing method is applied to process multiple data streams. Taking the multiple data streams including a first data stream and a second data stream as an example, the method includes: writing the data transmitted by the first data stream and the second data stream into a cache respectively; in response to the data from the first data stream and the second data stream being written into the cache, all the data in the cache are written into a memory. Correspondingly, the data stream processing device includes: a cache that can be shared by the at least two data streams to cache data; a memory connected to the cache and capable of receiving and storing the data in the cache; and a processor connected to the cache and the memory respectively, and in response to the data of the at least two data streams being written into the cache, the data in the cache is written into the memory.

Description

Technical Field

[0001] The present invention relates to the field of network technology, and in particular to a data stream processing method, device, computer-readable storage medium and equipment. Background Art

[0002] In modern networks, data is ever-changing, and there are situations where multiple data streams need to be merged. When multiple data streams need to be merged, the data needs to be cached and queued for transmission. When data is cached, each data stream needs to have a corresponding memory for caching.

[0003] In the scenario where the total amount of data is fixed, the sum of the data sent by each data stream is the total amount of data. The maximum amount of data sent by each data stream is the total amount of data, and the minimum amount is 0, that is, the amount of data sent by each data stream is not fixed. In order to cache all the data of each data stream, the capacity setting of the memory needs to be based on the maximum amount of data.

[0004] like Figure 1 As shown in the figure, the total number of data is 8192 and there are 4 data streams. When the total number of data is 8192, it is divided into 4 data streams for sending. If the number of data sent by each data stream is 2048, the depth of each memory block only needs 2048. If data stream 1 and data stream 2 do not send data, the total number of data sent by data stream 0 and data stream 3 is 8192. At this time, if only the two data streams are considered to send data equally, both are 4096, the depth of memory 0 and memory 3 needs to be 4096. If only data stream 0 sends data, the depth of memory 0 needs to be 8192. The above only considers the case of equal distribution of data, and the actual situation will be more complicated. This results in that the depth of the memory for caching data cannot be fixed, but must be adjusted according to the number of data sent by each data stream, which is difficult to achieve in actual design. In order to cache data to the maximum extent and prevent data loss, the memory depth of cache data needs to be the maximum number 8192. Such a setting will waste a lot of memory storage space. Summary of the invention

[0005] In order to reduce the waste of memory storage space when processing multiple data streams, the present invention provides a data stream processing method, wherein the data stream includes a first data stream and a second data stream, and the method includes:

[0006] The data transmitted by the first data stream and the second data stream are written into a cache respectively; in response to the data from the first data stream and the second data stream being written into the cache respectively, all the data in the cache are written into a memory.

[0007] Optionally, the method further includes: detecting whether the data in the memory is empty, and if so, directly reading the data from the cache.

[0008] Optionally, the method further includes: in response to the completion of data transmission and only data of the first data stream or the second data stream is stored in the cache, reading out data in the memory until the data in the memory is empty, and then reading out data from the cache.

[0009] The present invention also provides a data stream processing device, which is used to process at least two data streams, including: a cache, which can be shared by the at least two data streams to cache data; a memory, which is connected to the cache and can receive and store the data in the cache; and a processor, which writes the data in the cache into the memory in response to the data of the at least two data streams written into the cache.

[0010] Optionally, the data stream processing device further includes an arbitrator connected to the cache and the memory respectively, and configured to distribute data read from the memory or the cache.

[0011] Optionally, the width of the memory is the sum of the widths of the data of at least two data streams, and the depth of the memory is equal to the number of the larger data in the at least two data streams.

[0012] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium includes a data stream processing method program, and when the data stream processing method program is executed by a processor, the steps of the above-mentioned data stream processing method are implemented.

[0013] The present invention also provides a device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the above-mentioned data flow processing method.

[0014] The present application provides a data stream processing device, which can reduce the storage space of the memory and reduce the waste of storage space. In addition, based on the data stream processing device, the present application provides a data stream processing method, which can improve the efficiency of data transmission and avoid data residue at the end of data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a structural schematic diagram of a data stream processing device in the prior art;

[0016] Figure 2 It is a structural diagram of the data stream processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0018] An embodiment of the present invention provides a data stream processing device, which is used to process at least two data streams. For the convenience of description, in this embodiment, at least two data streams including a first data stream 10 and a second data stream 20 are taken as an example. Figure 2 As shown, the data stream processing device includes: a cache 1, which can be shared by a first data stream 10 and a second data stream 20 to cache data; a memory 2, which is connected to the cache 1 and can receive and store data in the cache 1; a processor 3, which is connected to the cache 1 and the memory 2 respectively, and in response to the data of the first data stream 10 and the second data stream 20 written into the cache 1, writes the data of the first data stream 10 and the second data stream 20 stored in the cache 1 into the memory 2 together; an arbitrator 4, which is connected to the cache 1, the memory 2 and the processor 3 respectively, and is used to distribute the data read out from the memory 2 or the cache 1.

[0019] Specifically, the arbitrator 4 may select storage data from the cache 1 or the memory 2 under the control of the processor 3 based on the storage status of the cache 1 and the memory 2 .

[0020] For example, when the memory 2 is empty, the arbitrator 4 directly reads data from the cache 1 .

[0021] For example, if the total number of data is 8192 and the data width is 200 bits, Figure 1 In the traditional data stream processing method shown in FIG. 1 , the total storage space required by the memory is 2*8192*200=3276800 bits. However, by using the data stream processing device provided in this embodiment, the depth of the memory 2 can be fixed to 2048, and the width of the memory is twice the width of one data, that is, 400 bits. The total storage space is 2048*400=819200 bits. That is, compared with the traditional data stream processing method, the data stream processing device provided in this embodiment can save about 75% of the storage space of the memory 2.

[0022] According to the solution provided in this embodiment, those skilled in the art can easily know that the solution is applied to the processing of N data streams (N is an integer greater than or equal to 2).

[0023] It should be noted that this embodiment provides a special example in which the width and data number of the two data streams are the same, but in actual application, the width and data number of each data stream may be different. Based on this, the data stream still includes the first data stream 10 and the second data stream 20 as an example. Assuming that the data number of the data of the first data stream 10 is greater than the data number of the data of the second data stream 20, the depth of the memory 2 can be set to be equal to the data number of the data of the first data stream 10. In addition, the width of the memory 2 can be set to the sum of the width of the data of the first data stream 10 and the width of the data of the second data stream 20.

[0024] The data scheduling of the prior art usually adopts a polling method to read and send data from each memory, that is, only one data needs to be sent out each time to empty the corresponding memory. However, the data stream processing device provided in this embodiment usually writes two data to memory 2 each time, and only one data needs to be sent out each time. Therefore, it is necessary to send data twice before memory 2 can be read out.

[0025] Based on this, this embodiment provides a data stream processing method, which processes data streams based on the above data stream processing device, and the method includes: writing data transmitted by the first data stream 10 and the second data stream 20 into the cache 1 respectively;

[0026] In response to the data from the first data stream 10 and the second data stream 20 being written into the cache 1 , all the data in the cache 1 are written into the memory 2 .

[0027] The method further comprises: detecting whether the data in the memory 2 is empty, and if so, directly reading the data from the cache 1 .

[0028] Such an arrangement can make data transmission more efficient.

[0029] The method further includes: in response to the end of data transmission and only the data of the first data stream 10 or the second data stream 20 is stored in the cache 1, reading the data in the memory 2 until the data in the memory 2 is empty, and then reading the data from the cache 1.

[0030] This configuration ensures that no data will remain in the cache 1 at the end of the data transmission.

[0031] This embodiment further provides a computer-readable storage medium, which includes a data stream processing method program. When the data stream processing method program is executed by a processor, the steps of the above-mentioned data stream processing method are implemented.

[0032] This embodiment also provides a device, including a processor (not shown) and a memory (not shown), wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the above-mentioned data flow processing method.

[0033] The technical contents and technical features of the present invention have been disclosed as above, and those skilled in the art can still make various substitutions and modifications without departing from the spirit of the present invention based on the teachings and disclosures of the present invention. Therefore, the protection scope of the present invention is not limited to the contents disclosed in the embodiments, but should include various substitutions and modifications without departing from the present invention, and be covered by the claims of this patent application.

Claims

1. A data stream processing method, wherein the data stream includes a first data stream and a second data stream, characterized in that: The method comprises: Writing the data transmitted by the first data stream and the second data stream into the cache respectively; In response to the data from the first data stream and the second data stream being written into the cache, writing all the data in the cache into a memory; The depth of the memory is set to the larger number of data in the first data stream and the second data stream, and the width of the memory is set to the sum of the width of the data in the first data stream and the width of the data in the second data stream.

2. The data stream processing method according to claim 1, characterized in that: The method further includes: detecting whether the data in the memory is empty, and if so, directly reading the data from the cache.

3. The data stream processing method according to claim 1, characterized in that: The method further comprises: In response to the data transmission being completed and only the data of the first data stream or the second data stream being stored in the cache, the data in the memory is read out until the data in the memory is empty, and then the data is read out from the cache.

4. A data stream processing device, used for processing at least two data streams, characterized in that: include: A cache, which can be shared by the at least two data streams to cache data; A memory, connected to the cache, capable of receiving and storing data in the cache; as well as A processor is connected to the cache and the memory respectively, and writes the data in the cache to the memory in response to the data of the at least two data streams being written into the cache; wherein the depth of the memory is set to be the larger number of data in the first data stream and the second data stream, and the width of the memory is set to be the sum of the width of the data of the first data stream and the width of the data of the second data stream.

5. The data stream processing device according to claim 4, characterized in that: Also includes: An arbiter is connected to the cache, the memory and the processor respectively, and is used to distribute data read from the memory or the cache.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a data stream processing method program, and when the data stream processing method program is executed by a processor, the steps of a data stream processing method according to any one of claims 1 to 3 are implemented.

7. A device, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a data stream processing method as described in any one of claims 1-3.

Images