Data stream processing method, device and multi-core processor

Abstract

The present application provides a data stream processing method, device and multi-core processor, the method executed in the multi-core processor includes: receiving a processing request for a target data stream including a unique identifier and an operation type of the target data stream; if the unique identifier belongs to a non-high-frequency identifier, judging whether the unique identifier is stored in a non-high-frequency identifier lookup table based on a non-high-frequency identifier address mapping table, if not, processing the system data object corresponding to the target data stream pre-stored in the DDR according to the operation type or updating the non-high-frequency identifier address mapping table and the non-high-frequency identifier lookup table, the non-high-frequency identifier lookup table is used to store a part of each system data object preset in the DDR. The present application can effectively improve the scalability of data storage of the multi-core processor, and perform data addition, deletion, modification and query processing with the system data object, which can effectively improve the effectiveness and reliability of data stream processing, and can improve the application scope of the multi-core processor and the data stream processing performance.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data stream processing method, device and multi-core processor. Background Art

[0002] In the face of processing massive amounts of data, multi-core computing can already complete many large-scale computing tasks. Data reading in memory and auxiliary memory often becomes a bottleneck for improving the speed of application programs. Multi-processing core systems face limited internal storage resources in terms of memory access and data interaction, and cannot be expanded infinitely.

[0003] Specifically, in multi-core processors such as DPU, different processing cores will store and access the processed data streams according to their different functions. Now, most processors have different functional cores inside and will establish separate mapping tables to record storage information. Such a design has a fixed storage performance for multi-core processors and is not scalable in space. In addition, the core processors are facing more diverse application scenarios, and each processing core on it processes data streams of different lengths for different scenarios, and the work pressure is also different. If there is a lack of reasonable data object management, the data stream of the processing core will be subject to certain restrictions, which will reduce the application scope of the multi-core processor and reduce the processing performance of the multi-core. Summary of the invention

[0004] In view of this, embodiments of the present application provide a data stream processing method, device, and multi-core processor to eliminate or improve one or more defects existing in the prior art.

[0005] One aspect of the present application provides a data stream processing method, which is executed in a multi-core processor, and the data stream processing includes:

[0006] Receiving a processing request for a target data stream, wherein the processing request includes a unique identifier and an operation type of the target data stream;

[0007] If the unique identifier of the target data stream is a non-high-frequency identifier, it is determined based on the locally preset non-high-frequency identifier address mapping table whether the unique identifier of the target data stream is stored in the local non-high-frequency identifier lookup table corresponding to the non-high-frequency identifier address mapping table; if not, the system data object corresponding to the target data stream pre-stored in the DDR is processed accordingly according to the operation type, or the non-high-frequency identifier address mapping table and the non-high-frequency identifier lookup table are updated accordingly, wherein the non-high-frequency identifier lookup table is used to store a part of each of the system data objects preset in the DDR.

[0008] In some embodiments of the present application, it also includes:

[0009] If it is determined based on the locally preset non-high frequency identification address mapping table that the unique identifier of the target data stream is stored in a local non-high frequency identification lookup table corresponding to the non-high frequency identification address mapping table, then the system data object corresponding to the target data stream in the non-high frequency identification lookup table is processed according to the operation type.

[0010] In some embodiments of the present application, it also includes:

[0011] If the unique identifier of the target data stream is a high-frequency identifier, the system data object corresponding to the target data stream in the locally preset high-frequency identifier linked list is processed according to the operation type, wherein the target data stream is used to store the system data objects corresponding to each unique identifier belonging to the high-frequency identifier.

[0012] In some embodiments of the present application, the non-high frequency identification address mapping table is used to store the corresponding relationship between the unique identification, state identification and storage address of each system data object in the non-high frequency identification lookup table;

[0013] Correspondingly, the method of determining whether the unique identifier of the target data stream is stored in a local non-high frequency identifier lookup table corresponding to the non-high frequency identifier address mapping table based on the locally preset non-high frequency identifier address mapping table, and if not, processing the system data object corresponding to the target data stream pre-stored in the DDR according to the operation type, or updating the non-high frequency identifier address mapping table and the non-high frequency identifier lookup table accordingly, including:

[0014] Determine whether each of the unique identifiers in the non-high-frequency identifier address mapping table contains the unique identifier of the target data stream and whether the status identifier corresponding to the unique identifier of the target data stream is currently displayed as a valid state; if not, process the system data object corresponding to the target data stream pre-stored in the DDR according to the operation type, or update the non-high-frequency identifier address mapping table and the non-high-frequency identifier lookup table accordingly.

[0015] In some embodiments of the present application, the processing of the system data object corresponding to the target data stream pre-stored in the DDR according to the operation type, or updating the non-high frequency identification address mapping table and the non-high frequency identification lookup table accordingly, includes:

[0016] If the operation type is object writing, determining whether the non-high frequency identification lookup table contains any remaining storage bits;

[0017] If so, writing the system data object corresponding to the target data stream into the frontmost storage bit of the current remaining storage bits of the non-high frequency identification lookup table, and updating the non-high frequency identification address mapping table accordingly;

[0018] If not, the last storage bit in the non-high frequency identification lookup table is occupied to write the system data object corresponding to the target data stream, and the non-high frequency identification address mapping table is updated accordingly.

[0019] In some embodiments of the present application, the processing of the system data object corresponding to the target data stream pre-stored in the DDR according to the operation type, or updating the non-high frequency identification address mapping table and the non-high frequency identification lookup table accordingly, includes:

[0020] If the operation type is object access or object deletion, the system data object corresponding to the target data stream pre-stored in the DDR is extracted or deleted accordingly.

[0021] In some embodiments of the present application, the processing, according to the operation type, of the system data object corresponding to the target data stream in the non-high frequency identification lookup table includes:

[0022] If the operation type is object writing, then the system data object corresponding to the target data stream in the non-high frequency identification lookup table is updated accordingly, and is replaced to the frontmost storage position in the non-high frequency identification lookup table;

[0023] If the operation type is object access, then correspondingly extracting the system data object corresponding to the target data stream in the non-high frequency identification lookup table;

[0024] If the operation type is object deletion, the system data object corresponding to the target data flow in the non-high frequency identification lookup table is deleted and the address is released, and the non-high frequency identification address mapping table is updated accordingly.

[0025] In some embodiments of the present application, the system data object is used to store the chain head address, chain tail address and ID status information corresponding to the unique identifier of the data stream.

[0026] Another aspect of the present application provides a data stream processing device, which is arranged in a multi-core processor, wherein:

[0027] A request receiving module, used for receiving a processing request for a target data stream, wherein the processing request includes a unique identifier and an operation type of the target data stream;

[0028] A fast and slow table query module is used to determine whether the unique identifier of the target data stream is stored in a local non-high-frequency identifier lookup table corresponding to the non-high-frequency identifier address mapping table based on a locally preset non-high-frequency identifier address mapping table if the unique identifier of the target data stream is a non-high-frequency identifier; if not, the system data object corresponding to the target data stream pre-stored in the DDR is processed accordingly according to the operation type, or the non-high-frequency identifier address mapping table and the non-high-frequency identifier lookup table are updated accordingly, wherein the non-high-frequency identifier lookup table is used to store a part of each of the system data objects preset in the DDR.

[0029] A third aspect of the present application provides a multi-core processor, wherein the type of the multi-core processor includes a DPU;

[0030] The multi-core processor is provided with a data stream processing device, and the data stream processing device is used to execute the data stream processing method.

[0031] The data stream processing method provided by the present application is executed in a multi-core processor by receiving a processing request for a target data stream, wherein the processing request includes a unique identifier and an operation type of the target data stream; if the unique identifier of the target data stream belongs to a non-high-frequency identifier, then based on a locally preset non-high-frequency identifier address mapping table, it is determined whether the unique identifier of the target data stream is stored in a local non-high-frequency identifier lookup table corresponding to the non-high-frequency identifier address mapping table; if not, then the system data object corresponding to the target data stream pre-stored in the DDR is processed accordingly according to the operation type, or the non-high-frequency identifier address mapping table and the non-high-frequency identifier lookup table are updated accordingly, wherein the non-high-frequency identifier lookup table is used to store a part of each of the system data objects preset in the DDR, which can effectively improve the scalability of data storage of the multi-core processor, and perform data addition, deletion, modification, and query processing with the system data object, which can effectively improve the effectiveness and reliability of data stream processing, and can improve the application scope of the multi-core processor and the data stream processing performance, thereby ensuring the high performance and completeness of the multi-core processor such as DPU, and can ensure high performance and increase the storage scalability of the multi-core processor under the condition of limited multi-core processor design resources.

[0032] Additional advantages, purposes, and features of the present application will be partially described in the following description, and will become partially apparent to those skilled in the art after studying the following, or may be learned from the practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the specification and the drawings.

[0033] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0035] Figure 1 This is a schematic diagram of a first flow chart of a data stream processing method in an embodiment of the present application.

[0036] Figure 2 This is a second flow chart of the data stream processing method in one embodiment of the present application.

[0037] Figure 3 The present invention is a schematic diagram of the storage of a non-high frequency identification address mapping table, a non-high frequency identification lookup table and a high frequency identification linked table in one embodiment of the present application.

[0038] Figure 4 This is a data structure diagram of a non-high frequency identification address mapping table in an example of the present application.

[0039] Figure 5 This is a schematic diagram of the data structure of the sdo chain in another example of the present application.

[0040] Figure 6 This is a third flow chart of the data stream processing method in one embodiment of the present application.

[0041] Figure 7 Schematic diagram of the execution process of data writing in one embodiment of the present application.

[0042] Figure 8 Schematic diagram of the execution process of data query in one embodiment of the present application.

[0043] Fig. 9 Schematic diagram of the execution process of data deletion in one embodiment of the present application.

[0044] Fig.10 Schematic diagram of the structure of a data stream processing device in one embodiment of the present application.

[0045] Fig.11 Schematic diagram of the structure of a multi-core processor in one embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the implementation modes and the accompanying drawings. Here, the illustrative implementation modes and descriptions of the present application are used to explain the present application, but are not intended to limit the present application.

[0047] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the scheme according to the present application are shown in the accompanying drawings, while other details that are not very relevant to the present application are omitted.

[0048] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0049] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0050] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0051] In order to improve the scalability of data storage of multi-core processors and ensure the high performance of multi-core processors, the embodiments of the present application respectively provide a data stream processing method, a data stream processing device for executing the data stream processing method, and a multi-core processor such as a DPU provided with the data stream processing device, and adopt a combination of fast and slow tables to ensure the high performance and completeness inside the DPU chip.

[0052] The details are described in detail through the following examples.

[0053] Based on this, the embodiment of the present application provides a data stream processing method that can be implemented by a functional unit or device in a multi-core processor, see Figure 1 , the data stream processing method specifically includes the following contents:

[0054] Step 100: Receive a processing request for a target data flow, wherein the processing request includes a unique identifier and an operation type of the target data flow.

[0055] It can be understood that the operation type may at least include writing (creating or updating), deleting or accessing the local target data stream of the multi-core processor.

[0056] In one or more embodiments of the present application, the unique identifier of the target data stream may be an ID or an identifier generated by other user-defined encoding methods.

[0057] Step 200: If the unique identifier of the target data stream is a non-high frequency identifier, then based on a locally preset non-high frequency identifier address mapping table, determine whether the unique identifier of the target data stream is stored in a local non-high frequency identifier lookup table corresponding to the non-high frequency identifier address mapping table; if not, then process the system data object corresponding to the target data stream pre-stored in the DDR accordingly according to the operation type, or update the non-high frequency identifier address mapping table and the non-high frequency identifier lookup table accordingly, wherein the non-high frequency identifier lookup table is used to store a portion of each of the system data objects preset in the DDR.

[0058] In one or more embodiments of the present application, the non-high frequency identification address mapping table and the non-high frequency identification lookup table are in a one-to-one correspondence, that is, the non-high frequency identification address mapping table will change in real time according to changes in the non-high frequency identification lookup table.

[0059] It can be understood that the non-high frequency identification lookup table is used to store a part of each of the system data objects preset in the DDR, which means that the number of system data objects stored in the non-high frequency identification lookup table is limited and is only used to store a part of each of the system data objects preset in the DDR, but not all. The specific number can be set according to actual needs. For example, the storage bits (which can be referred to as positions) used for one-to-one storage of system data objects in the non-high frequency identification lookup table can be set to 8, and correspondingly, the positions in the non-high frequency identification lookup table are also set to 8.

[0060] That is to say, the non-high frequency identification lookup table stored in the local storage area of ​​the multi-core processor is provided with a plurality of storage bits for one-to-one storage of system data objects corresponding to the non-high frequency identifications, and the non-high frequency identification address mapping table is provided with a plurality of storage bits for one-to-one storage of data such as mapping addresses corresponding to the non-high frequency identifications of the non-high frequency identification lookup table.

[0061] It should be noted that the multi-core processor is also locally provided with a plurality of high-frequency identifier linked lists for storing one-to-one storage bits of system data objects corresponding to the high-frequency identifiers.

[0062] Based on this, the non-high frequency identification lookup table and the high frequency identification linked table constitute a slow table and a fast table respectively, so as to realize a fast and slow table query mechanism applicable to step 200 .

[0063] For the sake of convenience, the non-high frequency identification address mapping table in the embodiment of the present application can be abbreviated as a non-high frequency ID mapping table, the non-high frequency identification lookup table can be abbreviated as a non-high frequency ID LUT table, and the high frequency identification linked table can be abbreviated as a high frequency ID linked table. LUT means lookup table.

[0064] In order to further improve the application scope and data stream processing performance of multi-core processors, in one or more embodiments of the present application, the system data object is used to store the chain head address, chain tail address and ID status information corresponding to the unique identifier of the data stream. The system data object can be abbreviated as Sdo or sdo (System data object), and the system data object stored in the non-high-frequency identifier lookup table and the high-frequency identifier linked table can be written as an sdo chain. Correspondingly, the operation type corresponding to the processing request of the target data stream can be abbreviated as an sdo operation.

[0065] From the above description, it can be seen that the data stream processing method provided in the embodiment of the present application can effectively improve the scalability of data storage of multi-core processors, and perform data addition, deletion, modification and query processing with system data objects, which can effectively improve the effectiveness and reliability of data stream processing, and can improve the application scope of multi-core processors and data stream processing performance, thereby ensuring high performance and completeness within multi-core processors such as DPU, and can ensure high performance and increase the storage scalability of multi-core processors under limited multi-core processor design resources.

[0066] In order to further improve the scalability of data storage and data stream processing efficiency of multi-core processors, in a data stream processing method provided in an embodiment of the present application, see Figure 2 , step 200 in the data stream processing method specifically includes the following contents:

[0067] Step 210: Determine whether the unique identifier of the target data stream is a non-high-frequency identifier. If so, execute step 220; if not, execute step 250.

[0068] In step 210, an implementation method for determining whether the unique identifier of the target data stream is a non-high-frequency identifier may include: querying a pre-stored identifier frequency table, wherein the identifier frequency table may include the frequency or number of occurrences of each preset unique identifier within a preset time period; and then judging whether the frequency or number of occurrences of the unique identifier of the target data stream found in the identifier frequency table within the preset time period exceeds a preset threshold, and if so, determining the unique identifier of the target data stream as a high-frequency identifier, and if not, determining the unique identifier of the target data stream as a non-high-frequency identifier. In addition, during or after each execution of step 200, the frequency or number of occurrences of the unique identifier of the target data stream in the identifier frequency table within the preset time period must be updated accordingly.

[0069] And, another implementation method of determining whether the unique identifier of the target data stream is a non-high-frequency identifier may include: determining whether the unique identifier of the target data stream is a high-frequency identifier based on a numerical value or mark preset in the frequency identifier bit in the unique identifier of the target data stream. For example, the frequency identifier bit may include a high-frequency mark or a non-high-frequency mark. If the frequency identifier bit in the unique identifier of the target data stream is a high-frequency mark, then the unique identifier of the target data stream is determined to be a high-frequency identifier.

[0070] Step 220: Based on the locally preset non-high frequency identification address mapping table, determine whether the unique identifier of the target data stream is stored in the local non-high frequency identification lookup table corresponding to the non-high frequency identification address mapping table. If so, execute step 230; if not, execute step 240.

[0071] Step 230: If it is determined based on the locally preset non-high frequency identification address mapping table that the unique identifier of the target data stream is stored in a local non-high frequency identification lookup table corresponding to the non-high frequency identification address mapping table, then the system data object corresponding to the target data stream in the non-high frequency identification lookup table is processed according to the operation type.

[0072] Step 240: If it is determined based on the locally preset non-high frequency identification address mapping table that the unique identification of the target data stream is not stored in the local non-high frequency identification lookup table corresponding to the non-high frequency identification address mapping table, then the system data object corresponding to the target data stream pre-stored in the DDR is processed accordingly according to the operation type, or the non-high frequency identification address mapping table and the non-high frequency identification lookup table are updated accordingly.

[0073] Step 250: If the unique identifier of the target data stream is a high-frequency identifier, the system data object corresponding to the target data stream in the locally preset high-frequency identifier linked list is processed according to the operation type, wherein the target data stream is used to store the system data objects corresponding to each unique identifier belonging to the high-frequency identifier.

[0074] In one or more embodiments of the present application, in order to further improve the application effectiveness and reliability of the non-high frequency identification address mapping table, the non-high frequency identification address mapping table is provided with a plurality of storage bits, and each storage bit is used to store the corresponding relationship between the unique identification, status identification and storage address of each system data object in the non-high frequency identification lookup table one by one. Figure 3 , the state flag can be abbreviated as mapbit, and the storage address can be abbreviated as lut_addr.

[0075] On this basis, in order to further illustrate the above non-high frequency identification address mapping table, non-high frequency identification lookup table and high frequency identification chain table, in a specific application example of a fast and slow lookup table of the present application, the following contents are specifically included:

[0076] (1) The ID numbers of data streams are divided into high-frequency IDs and non-high-frequency IDs.

[0077] (2) In order to improve efficiency, a high-frequency ID linked list is designed to store the chain head address, chain tail address, and ID status related information of a portion of high-frequency IDs that are frequently used.

[0078] (3) The non-high frequency ID stores the chain head address, chain tail address, length and ID status related information of the ID chain list in the auxiliary information area (SDO area) of DDR. At the same time, a non-high frequency ID LUT table is established for fast query and update of non-high frequency ID. When the 8 IDs in the non-high frequency ID (address) mapping table corresponding to the non-high frequency ID LUT table are full, the highest position will be occupied by the new ID, and the original 8 IDs will move backwards. The last ID will be squeezed out of the LUT table and only exist in DDR. The query system data object Sdo (System data object) command of the non-high frequency ID first queries whether the non-high frequency ID to be queried is in the non-high frequency ID LUT table. If it is not found, it will query DDR.

[0079] (4) The non-high-frequency ID (address) mapping table supports the storage of up to 8 non-high-frequency ID indexes and their corresponding random access memory RAM addresses lut_addr that store the sdo chain. The mapbit is used to indicate which of the data at these 8 locations is valid. When this chain is taken out, all valid information is compared with the non-high-frequency ID with query. If none of them matches, it proves that the information chain of the non-high-frequency ID to be queried does not exist in this non-high-frequency ID LUT table, and it is necessary to obtain the information chain of the query ID in DDR; if it matches (only one can match), go to the corresponding address of the non-high-frequency ID LUT table to read the sdo chain.

[0080] Among them, the data structure of the non-high frequency ID (address) mapping table is as follows Figure 4 As shown, the data structures of the system data objects (sdo chains) in the non-high frequency identification lookup table (non-high frequency ID LUT table) and the high frequency identification chain table (high frequency ID chain table) are as follows Figure 5 shown.

[0081] exist Figure 4 The data structure of the LUT table is shown in Table 1 below:

[0082] Table 1

[0083]

[0084] exist Figure 5 The meaning of the data structure of the linked list is shown in Table 2 below:

[0085] Table 2

[0086]

[0087]

[0088] In order to further improve the reliability and effectiveness of determining whether the unique identifier of the target data stream is stored in the non-high-frequency identifier lookup table, in a data stream processing method provided in an embodiment of the present application, see Figure 6 , step 220 in the data stream processing method specifically includes the following contents:

[0089] Step 221: Determine whether each unique identifier in the non-high-frequency identifier address mapping table contains the unique identifier of the target data stream and whether the status identifier corresponding to the unique identifier of the target data stream is currently displayed as a valid state. If so, execute step 230; if not, execute step 240.

[0090] In order to further improve the writing efficiency and reliability of the system data objects corresponding to the non-high-frequency identifiers, in a data stream processing method provided in an embodiment of the present application, see Figures 6 to 9, step 240 in the data stream processing method specifically includes the following contents:

[0091] Step 241: If the operation type is object writing, determine whether the non-high-frequency identification lookup table contains any remaining positions; if so, execute step 242; if not, execute step 243.

[0092] Step 242: Write the system data object corresponding to the target data stream into the frontmost position of the current remaining positions in the non-high frequency identification lookup table, and update the non-high frequency identification address mapping table accordingly.

[0093] Step 243: Occupy the last position in the non-high frequency identification lookup table to write the system data object corresponding to the target data stream, and update the non-high frequency identification address mapping table accordingly.

[0094] In order to further improve the access or deletion efficiency and reliability of the system data objects corresponding to the non-high-frequency identifiers, in a data stream processing method provided in an embodiment of the present application, see Figures 6 to 9 , step 240 in the data stream processing method further specifically includes the following contents:

[0095] Step 244: If the operation type is object access or object deletion, then the system data object corresponding to the target data stream pre-stored in the DDR is extracted or deleted accordingly.

[0096] In order to further improve the writing, accessing or deleting efficiency and reliability of the system data objects corresponding to the high-frequency identifiers, in a data stream processing method provided in an embodiment of the present application, see Figures 6 to 9 , step 250 in the data stream processing method further specifically includes the following contents:

[0097] Step 251: If the operation type is object writing, then the system data object corresponding to the target data stream in the non-high frequency identification lookup table is updated accordingly, and replaced to the frontmost storage position in the non-high frequency identification lookup table.

[0098] Step 252: If the operation type is object access, then the system data object corresponding to the target data stream in the non-high-frequency identification lookup table is extracted accordingly.

[0099] Step 253: If the operation type is object deletion, the system data object corresponding to the target data stream in the non-high frequency identification lookup table is deleted and the address is released, and the non-high frequency identification address mapping table is updated accordingly.

[0100] Specifically, in Figure 7In the process, the multi-core processor receives a write sdo operation. If the ID to be queried is a high-frequency ID, the sdo chain is directly written to the corresponding address of the high-frequency identification linked list (i.e., high-frequency ID ram). If the ID to be queried is not a high-frequency ID, the information chain of the mapped address on the chip is read. If the ID to be read is compared, the corresponding sdo chain in the ram is updated and brought to the front. If the ID to be read is not compared, a ram address is requested from the request module. If it is not requested, the position of the first written ID is occupied and the priority is updated. If it is requested, the status bit (i.e., storage bit) is detected. If all 8 are full, the position of the first written ID is occupied and the priority is updated. If there are remaining positions, the frontmost position among the remaining positions is occupied.

[0101] exist Figure 8 In the process, the multi-core processor receives the query sdo operation. If the ID to be queried is a high-frequency ID, the high-frequency identification linked list (i.e., high-frequency ID ram) is queried and the result is returned. If the ID to be queried is not a high-frequency ID, the information chain of the mapped address on the chip is read. If the ID to be queried is matched, the ram is read through the corresponding address and the result is returned. If the ID to be queried is not matched, it returns that the LUT is not found and the DDR is queried.

[0102] exist Fig. 9 In the process, the multi-core processor receives the delete sdo operation. If the ID to be deleted is a high-frequency ID, the status position of the corresponding address in the high-frequency identification list (i.e., high-frequency ID ram) is set to 0. If the ID to be deleted is not a high-frequency ID, the information chain of the mapped address on the chip is read. If the ID to be deleted is compared, the corresponding ID is deleted, the address is released, and the information chain is reconstructed. If the ID to be deleted is not compared, the DDR is queried and the ID is deleted.

[0103] The present application also provides a data stream processing device for executing all or part of the data stream processing method, which is arranged in a multi-core processor, see Fig.10 , the data stream processing device specifically includes the following contents:

[0104] The request receiving module 10 is used to receive a processing request for a target data stream, wherein the processing request includes a unique identifier and an operation type of the target data stream;

[0105] The fast and slow table query module 20 is used to determine whether the unique identifier of the target data stream is stored in a local non-high-frequency identifier lookup table corresponding to the non-high-frequency identifier address mapping table based on a locally preset non-high-frequency identifier address mapping table if the unique identifier of the target data stream is a non-high-frequency identifier; if not, the system data object corresponding to the target data stream pre-stored in the DDR is processed accordingly according to the operation type, or the non-high-frequency identifier address mapping table and the non-high-frequency identifier lookup table are updated accordingly, wherein the non-high-frequency identifier lookup table is used to store a part of each of the system data objects preset in the DDR.

[0106] The embodiments of the data stream processing device provided in the present application can be specifically used to execute the processing flow of the embodiments of the data stream processing method in the above embodiments. Its functions will not be described in detail here, and reference can be made to the detailed description of the embodiments of the above data stream processing method.

[0107] The data stream processing device may perform data stream processing in a multi-core processor. The selection may be made based on the processing capability of the multi-core processor and the restrictions of the user's usage scenario. This application does not limit this.

[0108] The multi-core processor may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0109] The server and the multi-core processor may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocol may include, for example, TCP / IP protocol, UDP / IP protocol, HTTP protocol, HTTPS protocol, etc. Of course, the network protocol may also include, for example, RPC protocol (Remote Procedure Call Protocol) and REST protocol (Representational State Transfer) used on top of the above protocols.

[0110] From the above description, it can be seen that the data stream processing device provided in the embodiment of the present application can effectively improve the scalability of data storage of multi-core processors, and perform data addition, deletion, modification and query processing with system data objects, which can effectively improve the effectiveness and reliability of data stream processing, and can improve the application scope of multi-core processors and data stream processing performance, thereby ensuring high performance and completeness within multi-core processors such as DPU, and can ensure high performance and increase the storage scalability of multi-core processors under limited multi-core processor design resources.

[0111] The present application also provides a multi-core processor. Fig.11 , which can be specifically a DPU; the multi-core processor is provided with a data stream processing device, and the data stream processing device is used to execute all or part of the content of the data stream processing method described in the above embodiment.

[0112] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0113] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0114] In the present application, features described and / or illustrated for one embodiment may be used in the same manner or in a similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0115] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data stream processing method, characterized in that: Executed in a multi-core processor, the data stream processing includes: Receiving a processing request for a target data stream, wherein the processing request includes a unique identifier and an operation type of the target data stream; If the unique identifier of the target data stream belongs to a non-high frequency identifier, then based on a locally preset non-high frequency identifier address mapping table, determine whether the unique identifier of the target data stream is stored in a local non-high frequency identifier lookup table corresponding to the non-high frequency identifier address mapping table; if not, then process the system data object corresponding to the target data stream pre-stored in the DDR according to the operation type, or update the non-high frequency identifier address mapping table and the non-high frequency identifier lookup table accordingly, wherein the non-high frequency identifier lookup table is used to store a part of each of the system data objects preset in the DDR; If the unique identifier of the target data stream belongs to a high-frequency identifier, the system data object corresponding to the target data stream in the locally preset high-frequency identifier chain list is processed according to the operation type, wherein the target data stream is used to store the system data objects corresponding to each unique identifier belonging to the high-frequency identifier; The system data object is used to store the chain head address, chain tail address and ID status information corresponding to the unique identifier of the data stream.

2. The data stream processing method according to claim 1, characterized in that: Also includes: If it is determined based on the locally preset non-high frequency identification address mapping table that the unique identifier of the target data stream is stored in a local non-high frequency identification lookup table corresponding to the non-high frequency identification address mapping table, then the system data object corresponding to the target data stream in the non-high frequency identification lookup table is processed according to the operation type.

3. The data stream processing method according to claim 1, characterized in that: The non-high frequency identification address mapping table is used to store the corresponding relationship between the unique identification, state identification and storage address of each system data object in the non-high frequency identification lookup table; Correspondingly, the method of determining whether the unique identifier of the target data stream is stored in a local non-high frequency identifier lookup table corresponding to the non-high frequency identifier address mapping table based on the locally preset non-high frequency identifier address mapping table, and if not, processing the system data object corresponding to the target data stream pre-stored in the DDR according to the operation type, or updating the non-high frequency identifier address mapping table and the non-high frequency identifier lookup table accordingly, including: Determine whether each of the unique identifiers in the non-high-frequency identifier address mapping table contains the unique identifier of the target data stream and whether the status identifier corresponding to the unique identifier of the target data stream is currently displayed as a valid state; if not, process the system data object corresponding to the target data stream pre-stored in the DDR according to the operation type, or update the non-high-frequency identifier address mapping table and the non-high-frequency identifier lookup table accordingly.

4. The data stream processing method according to claim 1, characterized in that: The corresponding processing of the system data object corresponding to the target data stream pre-stored in the DDR according to the operation type, or corresponding updating of the non-high frequency identification address mapping table and the non-high frequency identification lookup table, includes: If the operation type is object writing, determining whether the non-high frequency identification lookup table contains any remaining storage bits; If so, writing the system data object corresponding to the target data stream into the frontmost storage bit of the current remaining storage bits of the non-high frequency identification lookup table, and updating the non-high frequency identification address mapping table accordingly; If not, the last storage bit in the non-high frequency identification lookup table is occupied to write the system data object corresponding to the target data stream, and the non-high frequency identification address mapping table is updated accordingly.

5. The data stream processing method according to claim 1, characterized in that: The corresponding processing of the system data object corresponding to the target data stream pre-stored in the DDR according to the operation type, or corresponding updating of the non-high frequency identification address mapping table and the non-high frequency identification lookup table, includes: If the operation type is object access or object deletion, the system data object corresponding to the target data stream pre-stored in the DDR is extracted or deleted accordingly.

6. The data stream processing method according to claim 1, characterized in that: The processing of the system data object corresponding to the target data stream in the non-high frequency identification lookup table according to the operation type includes: If the operation type is object writing, then the system data object corresponding to the target data stream in the non-high frequency identification lookup table is updated accordingly, and is replaced to the frontmost storage position in the non-high frequency identification lookup table; If the operation type is object access, then correspondingly extracting the system data object corresponding to the target data stream in the non-high frequency identification lookup table; If the operation type is object deletion, the system data object corresponding to the target data flow in the non-high frequency identification lookup table is deleted and the address is released, and the non-high frequency identification address mapping table is updated accordingly.

7. A data stream processing device, characterized in that: Set in a multi-core processor, the data stream processing device: A request receiving module, used for receiving a processing request for a target data stream, wherein the processing request includes a unique identifier and an operation type of the target data stream; A fast and slow table query module is used for judging whether the unique identifier of the target data stream is stored in a local non-high frequency identifier lookup table corresponding to the non-high frequency identifier address mapping table based on a locally preset non-high frequency identifier address mapping table if the unique identifier of the target data stream belongs to a non-high frequency identifier; if not, processing the system data object corresponding to the target data stream pre-stored in the DDR according to the operation type, or updating the non-high frequency identifier address mapping table and the non-high frequency identifier lookup table accordingly, wherein the non-high frequency identifier lookup table is used to store a part of each of the system data objects preset in the DDR; The data stream processing device is further used to execute: if the unique identifier of the target data stream belongs to a high-frequency identifier, then correspondingly processing the system data object corresponding to the target data stream in the locally preset high-frequency identifier chain list according to the operation type, wherein the target data stream is used to store the system data objects corresponding to each unique identifier belonging to the high-frequency identifier; The system data object is used to store the chain head address, chain tail address and ID status information corresponding to the unique identifier of the data stream.

8. A multi-core processor, characterized in that: The type of the multi-core processor includes DPU; The multi-core processor is provided with a data stream processing device, and the data stream processing device is used to execute the data stream processing method according to any one of claims 1 to 6.

Images