Data consumption method, data processing system, storage medium and program product

Abstract

A data consumption method, a data processing system, a storage medium and a program product, which relate to the technical field of data storage. The data consumption method comprises: a first processor (a producer) writing target data to a plurality of storage areas in a memory, wherein different storage areas among the plurality of storage areas are used for storing the same amount of data, and the storage areas storing the target data are marked as a consumable state, such that during the process of the first processor writing the target data to the plurality of storage areas, a second processor (a consumer) can identify a first storage area, and consume the target data in the first storage area, wherein the first storage area is at least one storage area marked as a consumable state among the plurality of storage areas. In this way, during the process of a first processor writing target data to a memory, a second processor can consume, on the basis of a marker for a consumable state, data in a first storage area having the marker for the consumable state, thereby improving the efficiency of the second processor in data consumption.

Description

Data consumption methods, data processing systems, storage media and application products

[0001] This application claims priority to Chinese patent application filed on July 24, 2024, with application number 202411005121.5 and entitled "Data Consumption Method, Data Processing System, Storage Medium and Program Product", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of data storage technology, and in particular to a data consumption method, a data processing system, a storage medium, and a program product. Background Technology

[0003] The producer-consumer model is a model for handling the production and consumption of data. In this model, producers are responsible for producing data and can write the data into memory; consumers read the data from memory and consume it, such as performing data matching and filtering operations.

[0004] In practical applications, after generating data, the producer typically divides the complete data into multiple data packets and writes each packet into memory. During the data writing process, the producer receives a completion message from the memory (via the memory controller). Once the memory has returned completion messages for all data packets, the producer confirms that all data packets have been successfully written to memory. At this point, the producer notifies the consumer that it can now consume the data in memory, and the producer will then begin consuming the multiple data packets written to memory upon receiving the notification.

[0005] However, when producers provide data to consumers in this way, it can easily lead to significant delays and low efficiency in consumer data consumption. Summary of the Invention

[0006] This application provides a data consumption method to improve the efficiency of consumer data consumption and reduce the latency of data consumption. Furthermore, this application also provides a corresponding data processing system, a computer-readable storage medium, and a computer program product.

[0007] Firstly, this application provides a data consumption method applicable to a data processing system, which includes a first processor, a second processor, and memory. Exemplarily, the first processor may be a CPU, and the second processor may be a GPU (Graphics Processing Unit). During the process of the first processor providing target data to be consumed to the second processor, the first processor writes the target data to multiple storage areas in memory. Different storage areas in the multiple storage areas are used to store the same amount of data; that is, the maximum amount of data that each storage area can store can be the same. The storage area storing the target data is marked as consumable. Thus, during the process of the first processor writing the target data to the multiple storage areas, the second processor can identify a first storage area among the multiple storage areas, which is at least one storage area marked as consumable. Therefore, the second processor can begin consuming the target data in the first storage area during the process of the first processor writing data to memory.

[0008] Thus, during the process of the first processor (producer) writing multiple data packets into memory, the storage area where the data is written will be marked as consumable. This allows the second processor (consumer) to consume the data in the first storage area marked as consumable based on the consumable status, without having to wait for the first processor to write all the data into memory before consuming the data. This can reduce the latency of the second processor consuming data and improve the efficiency of the second processor consuming data.

[0009] In one possible implementation, the first storage area includes a data area and a non-data area. After the first processor stores the target data in the data area, it adds a marker in the non-data area to indicate that the target data in the data area is in a consumable state. Thus, by adding a marker in the non-data area, the first processor enables the second processor to identify storage areas marked as consumable, allowing the second processor to begin consuming data in those storage areas while the first processor is writing data to memory, thereby improving the efficiency of the second processor's data consumption.

[0010] In one possible implementation, when the first processor adds a marker in the non-data area, it can specifically configure the target field in the non-data area to a first value (e.g., 1), which indicates that the target data in the data area is in a consumable state. Optionally, when the target field in the storage area is configured to a second value (e.g., 0), this second value indicates that the target data in the data area is in a non-consumable state. Alternatively, the first processor adds a first error correction code in the non-data area, which is obtained by inverting the second error correction code corresponding to the target data in the data area. This first error correction code indicates that the target data in the data area is in a consumable state. Or, the first processor adds a second error correction code corresponding to the target data in the data area in the non-data area, which indicates that the target data in the data area is in a consumable state. In this way, the first processor adds a marker in the non-data area through the above various methods, so that the second processor can start consuming the data in the storage area marked as consumable during the process of the first processor writing data to memory, thereby improving the efficiency of the second processor in consuming data.

[0011] In one possible implementation, when the second processor identifies a first storage region marked as consumable in multiple storage regions, it may specifically read the value of a target field in a non-data region. If the target field value is a first value, the second processor determines that the first storage region is marked as consumable. Optionally, if the target field value in a storage region is a second value, the second processor determines that the storage region is marked as non-consumable. Alternatively, the second processor calculates a third error correction code based on the target data in the data region. If the second error correction code in the non-data region is the same as the inverted third error correction code, the second processor determines that the first storage region is marked as consumable. Or, the second processor calculates a third error correction code based on the target data in the data region. If the first error correction code in the non-data region is the same as the third error correction code, the second processor determines that the first storage region is marked as consumable. Thus, during the process of the first processor writing data to memory, the second processor can identify the data in the storage region marked as consumable and begin consuming the data, thereby improving the efficiency of the second processor's data consumption.

[0012] In one possible implementation, before the first processor writes target data to multiple storage areas in memory, the first processor may mark the multiple storage areas as unconsumable. After determining that the multiple storage areas are marked as unconsumable, the first processor sends a first notification message to the second processor. This first notification message notifies the second processor to consume data from the storage areas marked as consumable. Thus, the second processor can not only determine the storage area containing the data to be consumed based on the notification message, but the first processor, by marking the multiple storage areas as unconsumable before writing the target data, ensures the correctness of the data consumed by the second processor, preventing the second processor from consuming data from other storage areas marked as consumable but not necessarily data that the second processor should consume.

[0013] In one possible implementation, the data processing system further includes a third processor. Then, after the second processor has consumed all the target data in the multiple storage areas, the first processor can re-mark the multiple storage areas as unconsumable. Furthermore, after determining that the multiple storage areas have been re-marked as unconsumable, the first processor sends a second notification message to the third processor. This second notification message notifies the third processor to consume the data in the storage areas marked as consumable. Thus, when the first processor provides data to be consumed to different processors based on the same multiple storage areas, the correctness of data consumption by different processors can be characterized by individually marking the multiple storage areas as unconsumable and sending notification messages.

[0014] In one possible implementation, when the first processor writes target data to multiple storage areas in memory, it may specifically write multiple data packets corresponding to the target data to the multiple storage areas in parallel to improve the efficiency of the first processor writing data to memory; or, the first processor may write multiple data packets corresponding to the target data to the multiple storage areas one by one to ensure the order of the data written to memory by the first processor.

[0015] Secondly, this application provides a data processing system, which includes a first processor, a second processor, and memory; the first processor is used to write target data to multiple storage areas in memory, each of the multiple storage areas is used to store the same amount of data, wherein the storage area storing the target data is marked as consumable; the second processor is used to identify the first storage area during the process of the first processor writing the target data to the multiple storage areas, the first storage area being at least one of the multiple storage areas marked as consumable, and to consume the target data in the first storage area.

[0016] In one possible implementation, the first storage area includes a data area and a non-data area; the first processor is further configured to add a marker in the non-data area after the target data is stored in the data area, the marker being used to indicate that the target data in the data area is in a consumable state.

[0017] In one possible implementation, the first processor is configured to: configure the value of the target field in the non-data area to a first value, the first value being used to indicate that the target data in the data area is in a consumable state; or, add a first error correction code in the non-data area, the first error correction code being obtained by inverting the second error correction code corresponding to the target data in the data area, the first error correction code being used to indicate that the target data in the data area is in a consumable state; or, add a second error correction code corresponding to the target data in the data area in the non-data area, the second error correction code being used to indicate that the target data in the data area is in a consumable state.

[0018] In one possible implementation, the second processor is configured to: read the value of a target field in a non-data region, and determine that the first storage region is marked as consumable when the value of the target field is a first value; or, calculate a third error correction code based on the target data in the data region, and determine that the first storage region is marked as consumable when the second error correction code in the non-data region is the same as the inverted third error correction code; or, calculate a third error correction code based on the target data in the data region, and determine that the first storage region is marked as consumable when the first error correction code in the non-data region is the same as the third error correction code.

[0019] In one possible implementation, the first processor is further configured to: mark multiple storage regions as unconsumable before writing target data to multiple storage regions in memory; and after determining that multiple storage regions are marked as unconsumable, send a first notification message to the second processor, the first notification message being used to notify the second processor to consume the data in the storage regions marked as consumable among the multiple storage regions.

[0020] In one possible implementation, the data processing system further includes a third processor. The first processor is further configured to: after the second processor has consumed all the target data in the multiple storage areas, re-mark the multiple storage areas as unconsumable; and after determining that the multiple storage areas have been re-marked as unconsumable, send a second notification message to the third processor, the second notification message being used to notify the third processor to consume the data in the storage areas marked as consumable in the multiple storage areas.

[0021] In one possible implementation, the first processor is configured to: write multiple data packets corresponding to the target data into multiple storage areas in parallel; or write multiple data packets corresponding to the target data into multiple storage areas one by one.

[0022] The data processing system provided in the second aspect corresponds to the data consumption method provided in the first aspect. Therefore, the technical effects of any implementation method in the data processing system provided in the second aspect can be referred to the technical effects of the corresponding implementation method in the first aspect, and will not be elaborated further.

[0023] Thirdly, this application provides a computer-readable storage medium storing instructions that, when executed on a computing device, cause the computing device to perform the operational steps of the data consumption method described in the first aspect or any implementation thereof.

[0024] Fourthly, this application provides a computer program product containing instructions that, when run on a computing device, causes the computing device to perform the operational steps of the data consumption method described in the first aspect or any implementation thereof.

[0025] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0026] Figure 1 is a schematic diagram of the structure of an exemplary data processing system provided in this application;

[0027] Figure 2 is a schematic diagram of another exemplary data processing system provided in this application;

[0028] Figure 3 is a flowchart illustrating a data consumption method provided in this application;

[0029] Figure 4 is a schematic diagram of the storage area, which includes a data area and a non-data area.

[0030] Figure 5 is a schematic diagram of processor 201 initializing memory 100 and providing data to processor 202 through memory 100. Detailed Implementation

[0031] To reduce the latency and improve the efficiency of data consumption, this application provides a data consumption method. For storage areas that store new data written by producers (such as CPUs), a consumable status mark is added. This allows consumers (such as NPUs) to start consuming data in storage areas marked as consumable without waiting for producers to write all the data to memory. Instead, they can start consuming data in the storage areas marked as consumable while the producers are writing data to multiple storage areas in memory. This improves the efficiency of data consumption and reduces the latency of data consumption.

[0032] To facilitate understanding of the technical solution of this application, the relevant technical terms involved in this application will be explained below.

[0033] A producer is the entity responsible for producing data. For example, a producer can generate business data such as text during operation, and the data it generates will be provided to consumers for consumption.

[0034] Consumers are the entities that consume the data produced by producers, and can perform consumption operations such as matching and filtering the data.

[0035] A cache line is a caching mechanism that operates in memory. Producers or consumers read and write data in memory in fixed-size units, which is the smallest unit for reading and writing data in memory.

[0036] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0037] Referring to Figure 1, a schematic diagram of a data processing system is shown. As shown in Figure 1, the data processing system 10 may include a memory 100 and multiple processors. Figure 1 illustrates this using processors 201, 202, and 203 as an example. In practical applications, the data processing system 10 may also include four or more processors.

[0038] For example, processor 201 (and similar processors 202 and 203) may be any type of processor or any combination thereof, such as application-specific integrated circuit (ASIC), programmable logic device (PLD), complex programmable logical device (CPLD), field-programmable gate array (FPGA), generic array logic (GAL), system on chip (SoC), software-defined infrastructure (SDI) chip, artificial intelligence (AI) chip, data processing unit (DPU).

[0039] Furthermore, processor 201 may be equipped with a memory controller 2011, thereby enabling processor 201 to write data to memory 100 using the built-in memory controller 2011. Processor 202 may be equipped with a memory controller 2021, and processor 202 may read data from memory 100 using the built-in memory controller 2021. Processor 203 may be equipped with a memory controller 2031, and processor 203 may read data from memory 100 using the built-in memory controller 2031.

[0040] Memory 100 can be, for example, random access memory (RAM) or read-only memory (ROM). RAM can be, for example, dynamic random access memory (DRAM), static random access memory (SRAM), or storage class memory (SCM). ROM can be, for example, programmable read-only memory (PROM) or erasable programmable read-only memory (EPROM). Alternatively, memory 100 can also be a dual in-line memory module (DIMM), i.e., a module composed of dynamic random access memory (DRAM); or, memory 100 can be a solid-state disk (SSD). Furthermore, memory 100 can have a power-saving function, meaning that when memory 100 experiences a power outage and is then powered on again, the data stored in it will not be lost due to the power failure.

[0041] The memory 100, processor 201, processor 202 and processor 203 can be connected by an interconnect bus. This interconnect bus can be, for example, a peripheral component interconnect express (PCIe), a system bus (SB), a memory bus (MB), or a hyper transport (HT) bus, or other types of buses. There is no limitation on this.

[0042] In the data processing system 10 shown in Figure 1, processor 201 acts as a producer, writing generated data into memory 100, while processor 202 acts as a consumer, reading data from memory 100 and consuming it. In this process, if processor 201 writes all the generated data into memory 100 before notifying processor 202 to start consuming the data, processor 202 will have to wait a considerable amount of time before consuming the data. Specifically, the larger the amount of data that processor 201 needs to write, the longer it takes to write the data into memory, resulting in a longer waiting time for processor 202 to consume the data and lower data consumption efficiency.

[0043] Based on this, in the data processing system 10 shown in Figure 1, when the processor 201 writes data to the memory 100, it specifically writes the generated target data into multiple storage areas in the memory 100. Each storage area is used to store a portion of the target data, and each storage area can store the same amount of data, such as 64 bytes or 128 bytes. The storage area where the target data is written is marked as consumable, for example, by the processor 201. Thus, during the process of the processor 201 writing target data to the memory 100, the processor 202 can determine the storage areas in the memory 100 marked as consumable and consume the target data in those storage areas.

[0044] Thus, while the processor 201 is writing target data to the memory 100, the processor 202 can begin to consume the target data already written to the memory 100 (i.e., the processor 202 begins to consume a portion of the complete target data, which has already been written to the memory 100), without having to wait for the processor 201 to write all the target data to the memory 100. This can effectively reduce the time spent by the processor 202 waiting to consume data, thereby reducing the latency of the processor 202 consuming data and improving the efficiency of consuming data.

[0045] It is worth noting that the above explanation uses processor 201 as the producer and processor 202 as the consumer as an example. In practical applications, processor 202 can also act as a producer, writing data into memory 100, and processor 201 can read the data from memory 100 and consume that data.

[0046] The data processing system 10 shown in Figure 1 is merely an example. In other possible implementations, as shown in Figure 2, the data processing system 20 may further include a memory controller 300, and processors 201, 202, and 203 may not integrate a memory controller, allowing processors 201, 202, and 203 to write data to and read data from memory 100 via the memory controller 300. Alternatively, in other possible data processing systems, based on the data processing systems shown in Figure 1 or Figure 2, fewer or more processors may be included, and data produced by different processors may be consumed by other processors via memory.

[0047] For ease of understanding, embodiments of the data consumption method provided in this application will be described below with reference to the accompanying drawings.

[0048] Referring to Figure 3, which is a flowchart illustrating a data consumption method according to an embodiment of this application, this method can be applied to the data processing system 10 shown in Figure 1, or to the data processing system 20 shown in Figure 2, or to other applicable data processing systems. For ease of explanation, this embodiment uses the data processing system 10 shown in Figure 1 as an example, with processor 201 as the producer and processor 202 as the consumer.

[0049] The data consumption method shown in Figure 3 may specifically include:

[0050] S301: Processor 201 generates target data.

[0051] In this embodiment, processor 201 acts as a generator, generating data during operation, which is then consumed by processor 202. For example, processor 201 can generate data such as tables or text during operation, and processor 202 can subsequently perform data matching, data filtering, and other consumption operations on this data. For ease of description and distinction, this data will be referred to as target data below.

[0052] After generating the target data, the processor 201 determines multiple storage areas in memory 100 to store the target data. Specifically, it determines the addresses of these storage areas so that the processor 201 can subsequently write the generated target data into these storage areas. Each storage area stores a portion of the target data, such as 64 bytes of data. The process of the processor 201 writing data into memory 100 will be described below.

[0053] S302: Processor 201 splits the target data into multiple data packets.

[0054] S303: The processor 201 writes the multiple data packets to multiple storage areas in the memory 100, each storage area being able to store the same amount of data, and marks the storage area containing the target data in the multiple storage areas as a consumable state.

[0055] Typically, processor 201 can read and write data to memory 100 based on a fixed size. This fixed size is the smallest unit for data interaction between processor 201 and memory 100, specifically the length of data access to / from memory 100 performed by processor 201 within one cycle (the minimum time interval between two consecutive access operations). The smallest unit of data that processor 201 can write at any given time can be stored in a storage area within memory 100.

[0056] As shown in Figure 4, each storage area can include a data region and a non-data region. The data region, also known as a memory cache line, stores data written by the processor 201. The amount of data that this data region can store is the smallest unit of data read and written by the processor 201 from memory 100. For example, the amount of data that a data region can store can be 64 bytes, 128 bytes, etc. While the maximum amount of data that each data region can store can be consistent, in actual application scenarios, the actual amount of data stored in some data regions can differ. For example, data region A stores 64 bytes of data, while data region B stores 56 bytes of data. The non-data region stores information related to the data in the data region. For example, the non-data region can include multiple fields, such as a validity field and an error correction field. The validity field indicates whether the data in the data region is valid; for example, the validity field can include one bit, where a value of 0 indicates that the data in the data region is invalid, and a value of 1 indicates that the data in the data region is valid. Error correction fields are used to correct errors in the data in the data area. For example, an error correction field may include multiple bits, the values ​​of which can form an error correcting code (ECC). This ECC can then be used to correct and rectify erroneous data stored in the data area.

[0057] In practical applications, before writing target data to memory 100, processor 201 can obtain the address information of the multiple storage areas in advance. For example, the operating system in data processing system 10 (not shown in Figure 1) can assign the address of the multiple storage areas to processor 201, so that processor 201 can write the target data to the multiple storage areas indicated by the address information.

[0058] During the process of writing target data to memory 100, processor 201 can first divide the generated target data into multiple data packets according to the specified size. Each data packet includes a portion of the target data, and then write the target data to memory 100 at the data packet granularity. For example, assuming processor 201 generates 768 bytes of target data (including 12 rows of data in the table, with each row containing 64 bytes), processor 201 can split the 768 bytes of target data into 4 data packets, each data packet containing 192 bytes of target data (each data packet includes 3 rows of data in the table).

[0059] For each data packet, processor 201 can write the data packet into one or more data areas of storage regions in memory 100. For example, when the data packet contains 192 bytes, processor 201 can write the target data in the data packet into the data areas of three storage regions in memory 100, with each data area storing 64 bytes of target data (i.e., storing one row of data in a table). As shown in Figure 5, when processor 201 writes data packet 0 into memory 100, it specifically writes the target data in data packet 0 into the data areas of three storage regions. Furthermore, after successfully writing the target data into the data areas, processor 201 can also record relevant information about the target data in the data areas in non-data areas, such as setting the value of valid fields in non-data areas to 1 and using error correction fields to record the error correction code corresponding to the target data in the data areas.

[0060] In this embodiment, the following implementation examples of writing multiple data packets into memory 100 are provided.

[0061] In the first implementation example, the processor 201 can write multiple data packets one by one into multiple storage areas in memory 100 in sequence, such as the packet number.

[0062] In the second implementation example, processor 201 can write multiple data packets in parallel to multiple storage areas in memory 100. For example, processor 201 can utilize multiple paths to write multiple data packets in parallel to memory 100 based on direct memory access (DMA) technology, as shown in Figure 5. The data transfer rate on different paths can be the same. Alternatively, the data transfer rate on different paths can differ, as shown in Figure 5. When all data in data packet 0 is successfully written via a fast write path, the data successfully written via a slow write path is only a portion of the data in data packet 2 (i.e., the data in data area 6 shown in Figure 5). For example, some paths have more I / O stacks (lower data transfer rate), while other paths have fewer I / O stacks (higher data transfer rate). In this case, the order in which the multiple data packets are written to memory 100 is out of order; that is, the order in which the multiple data packets are successfully written to memory 100 is different from the order in which the multiple data packets are numbered. In this way, the processor 201 can achieve a high level of efficiency in writing multiple data packets to the memory 100; and after the data packet is successfully written to the memory 100, the memory 100 does not need to send a response to the processor 201, which can reduce the memory bandwidth occupied by data writing.

[0063] In the third implementation example, processor 201 can write multiple data packets to memory 100 in parallel. During this process, after successfully saving a data packet in its storage area, memory 100 can send a completion message to processor 201 for that data packet. This completion message can carry identification information such as the data packet's encoding, allowing processor 201 to determine that the data packet was successfully written. If processor 201 does not receive a completion message from memory 100 for the currently written data packet within a specified time period, it can rewrite the data packet to memory 100. Thus, during the process of processor 201 writing multiple data packets to memory 100, the received multiple completion messages can be used to determine that each data packet has been successfully written, thereby ensuring the reliability of data writing to memory 100 by processor 201.

[0064] It is understood that the above implementation of the processor 201 writing multiple data packets to the memory 100 is only an example. In actual applications, the processor 201 can also write multiple data packets to the memory 100 in other ways, and there is no limitation on this.

[0065] In this embodiment, for the storage area containing successfully obtained target data, the processor 201 marks the storage area as consumable, so that the subsequent processor 202 (consumer) can consume the target data in the storage area based on the consumable status mark. The storage area's status can include both consumable and non-consumable states. When a storage area is marked as consumable, the data in that storage area can be consumed by the processor 202 (or other processors); when a storage area is marked as non-consumable, the data in that storage area cannot be consumed by the processor 202, or no data is stored in that storage area.

[0066] In this embodiment, the following implementation examples are provided for marking storage areas containing target data as consumable.

[0067] In the first implementation example, the multiple fields in the non-data area may include a target field. This target field can be any field in the non-data area other than the valid field and the error correction field, and it may include one or more bits. The processor 201 can then use at least one bit in the target field to mark whether the storage area is in a consumable state. For example, using one bit in the target field to mark a consumable or non-consumable state: after successfully saving the target data in the data area of ​​the storage area, the processor 201 can set the value of the bit in the non-data area belonging to the target field to "1" to mark the storage area as consumable. Conversely, when the value of the bit in the non-data area belonging to the target field is set to "0", the value "0" is used to mark the storage area as non-consumable.

[0068] In the second implementation example, after the target data is successfully saved in the data area of ​​the storage area, the processor 201 can generate an error correction code based on the target data in the data area and record the error correction code in the non-data area. At this time, the error correction code recorded in the non-data area can be used to indicate that the storage area is in a consumable state, that is, to indicate that the target data stored in the data area is in a consumable state. Conversely, if the non-data area does not contain an error correction code, or if the error correction code in the non-data area is different from the error correction code corresponding to the target data in the data area, then the storage area can be indicated as being in a non-consumable state.

[0069] In the third implementation example, after the target data is successfully saved in the data area of ​​the storage area, the processor 201 can generate an error correction code based on the target data in the data area and then invert the error correction code. For example, assuming the error correction code generated by the processor 201 based on the target data in the data area is "1101011", the processor 201 can invert the error correction code to obtain "0010100". Then, the processor 201 can record the inverted error correction code in the non-data area. At this time, the inverted error correction code recorded in the non-data area can be used to indicate that the storage area is in a consumable state. Conversely, if the non-data area does not contain an error correction code, or if the error correction code in the non-data area is the same as the error correction code corresponding to the data in the data area, then the storage area can be indicated as a non-consumable state.

[0070] It is understood that the above implementation of the processor 201 marking the storage area as consumable is only an example. In actual applications, the processor 201 can also mark the storage area as consumable based on other methods. For example, the processor 201 can redefine some fields (such as valid fields) in the non-data area and use at least one bit in the part of the field to mark whether the storage area is consumable, etc., without limitation.

[0071] In practical applications, before the processor 201 writes the target data to the memory 100, the processor 201 can initialize multiple storage areas in the memory 100 used to store the target data, marking these storage areas as unconsumable. Then, after the processor 201 writes the target data to the storage area, it changes the state of that storage area from unconsumable to consumable.

[0072] In one possible implementation, processor 201 can access multiple storage regions in memory 100 using the built-in memory controller 2011 and mark these multiple storage regions as unconsumable, such as setting an unconsumable mark in the non-data areas of each storage region, as shown in Figure 5. Furthermore, after all multiple storage regions are marked as unconsumable, processor 201 can send notification message 1 to processor 202 to notify processor 202 that it can begin consuming the target data in the storage regions marked as consumable, as shown in Figures 3 and 5.

[0073] The processor 201 may include identifiers of multiple storage regions in the notification message 1 it sends, so that the processor 202 can determine the multiple storage regions where the target data is located based on the identifiers. These identifiers may be, for example, the addresses of the multiple storage regions. Alternatively, the processor 201 may notify the processor 202 of the multiple storage regions before initializing them in memory 100, so that the processor 202 can know the multiple storage regions where the target data to be consumed is located.

[0074] Thus, during the process of processor 201 writing multiple data packets to memory 100, processor 202 can begin consuming the portion of these data packets that are already stored in the multiple storage areas. The following section further describes the implementation process of processor 202 consuming data from the multiple storage areas.

[0075] S304: During the process of processor 201 writing multiple data packets to multiple storage areas, processor 202 identifies the first storage area in the multiple storage areas that is marked as consumable.

[0076] In this embodiment, during the process of the processor 201 writing multiple data packets into the memory 100, the storage area that has been written with the target data will be marked as consumable. Therefore, the processor 202 can identify the storage area marked as consumable from multiple storage areas.

[0077] The following explanation uses one of multiple storage regions as an example. For ease of distinction and description, this storage region will be referred to as the first storage region.

[0078] In the first implementation example, processor 201 uses the value of a target field in the non-data region of the first storage area to mark the state of the first storage area as either consumable or non-consumable. Then, processor 202 can read the value of the target field in the non-data region of the first storage area, and when the target field value is a first value, such as "1", processor 202 can determine that the first storage area is marked as consumable. When the target field value is a second value, such as "0", processor 202 can determine that the first storage area is marked as non-consumable.

[0079] In the second implementation example, processor 201 uses the error correction code of the data to mark the state of the storage area as consumable. Then, processor 202 can calculate error correction code 1 based on the target data stored in the data area of ​​the first storage area, and read error correction code 2 stored in the non-data area of ​​the first storage area. Then, processor 202 can compare error correction code 1 and error correction code 2. When error correction code 1 and error correction code 2 are the same, processor 202 can determine that the first storage area is marked as consumable. When error correction code 1 and error correction code 2 are different, processor 202 can determine that the first storage area is marked as non-consumable.

[0080] In the third implementation example, processor 201 uses the inverted value of the error correction code of the target data to mark the state of the first storage area as consumable. Then, processor 202 can calculate error correction code 1 based on the data stored in the data area of ​​the first storage area, and invert the error correction code to obtain error correction code 2. For example, assuming error correction code 1 is "1101011", then the error correction code 2 obtained after inverting error correction code 1 is "0010100". Simultaneously, processor 202 can also read error correction code 3 stored in the non-data area of ​​the first storage area. Then, processor 202 can compare error correction code 2 and error correction code 3. When error correction code 2 and error correction code 3 are the same, processor 202 can determine that the first storage area is marked as consumable. When error correction code 2 and error correction code 3 are different, processor 202 can determine that the first storage area is marked as non-consumable.

[0081] In practical applications, the processor 202 can also identify storage regions marked as consumable in multiple storage regions based on other methods, without limitation.

[0082] Similarly, for each storage region, the processor 202 can use the above method to determine whether the storage region is marked as consumable, that is, whether the storage region stores the target data newly written by the processor 201. In practical applications, the processor 202 can periodically traverse the storage regions in the memory 100, and in each round of traversal, the processor 202 can identify the storage regions marked as consumable from multiple storage regions based on the above-described similar method.

[0083] In the first round of traversal, processor 202 can traverse all storage areas used to store multiple data packets generated by processor 201 to identify storage areas in a consumable state. In the second round of traversal, processor 202 can traverse at least one of the remaining storage areas marked as non-consumable to identify consumable storage areas. Storage areas already identified as consumable in the previous round do not need to be traversed. This process continues until processor 202 identifies all storage areas as consumable. For example, suppose processor 201 writes target data to 10 storage areas in memory 100, namely storage area 1 to storage area 10, each used to store one-tenth of the complete target data. In the first round of traversal, processor 202 can identify storage area 10 as consumable, meaning that storage area 10 successfully stores part of the target data generated by processor 201, while storage areas 1 to 9 have not successfully written target data. Then, during the second round of traversal, the processor 202 can traverse the remaining storage regions 1 to 9 to identify the storage regions marked as consumable. This process continues until all storage regions 1 to 10 are identified by the processor 202 as being marked as consumable.

[0084] S305: Processor 202 consumes target data in the first memory region identified.

[0085] After identifying a first storage area marked as consumable, the processor 202 can read the target data stored in that first storage area, that is, read the target data stored in the data area of ​​the first storage area, and perform corresponding consumption operations on the read target data. Consuming the target data means that the processor 202 performs preset processing operations on the target data, such as matching, filtering, or summing.

[0086] When processor 202 iterates through the storage areas in memory 100, after each iteration and identifying a storage area marked as consumable, processor 202 can read and consume the target data in that storage area. As shown in Figure 5, after one iteration, processor 202 can read and consume data from data areas 0, 1, 2, and 6. This process continues until processor 202 has identified all storage areas as consumable and read the target data from all of them. At this point, processor 202 can terminate its access to memory 100 and complete the consumption of all target data provided by processor 201. When processor 201 writes multiple data packets corresponding to the target data into memory 100 in parallel, the target data can be stored sequentially in memory 100. Therefore, when processor 202 reads the target data from memory 100, it can determine the data order based on the target data's storage address in memory 100.

[0087] Thus, while processor 201 is writing target data to memory 100, processor 202 can begin consuming the target data. This effectively reduces the latency of processor 202 consuming the target data and improves the efficiency of consuming the target data. Furthermore, the granularity of the target data consumed by processor 202 can reach the granularity of data stored in a single storage area, such as processor 202 consuming 64 bytes of data from the target data at a time.

[0088] In the embodiment shown in Figure 3 above, processor 201 provides target data to processor 202 through memory 100 for processor 202 to consume. In actual applications, processor 201 can also provide data to processor 203 through memory 100 for processor 203 to consume. The implementation method of processor 201 providing data to processor 203 through memory 100 can be found in the description of the relevant aspects of the embodiment shown in Figure 3 above, and will not be repeated here.

[0089] Furthermore, when processor 201 provides different data to processor 202 and processor 203 successively through the same storage area in memory 100, processor 202 and processor 203 can begin the process of consuming data upon individual notification from processor 201.

[0090] In specific implementation, assuming processor 201 first provides data to be consumed to processor 202, processor 201 can provide target data to processor 202 according to the embodiment shown in Figure 3. After processor 202 has consumed all the target data in multiple storage areas, processor 201 completes the task of providing data to be consumed to processor 202. Then, processor 201 can re-mark the multiple storage areas as unconsumable. After determining that the multiple storage areas have been re-marked as unconsumable, processor 201 can send notification message 2 to processor 203 to notify processor 203 that it can start consuming the data in the storage areas marked as consumable. In this way, during the process of processor 201 continuously writing data into the multiple storage areas and marking the storage areas containing data from unconsumable to consumable, processor 203 can read data and start consuming it according to the consumable status of each storage area, thereby improving the efficiency of processor 203 in consuming data.

[0091] It is worth noting that in the embodiment shown in Figure 3 above, the processor 201 integrates a memory controller 2011. Therefore, the operations of writing data to the memory 201 and marking consumable / non-consumable states can be specifically performed by the memory controller 2011 within the processor 201. Similarly, the processor 202 can read target data from the memory 100 through its built-in memory controller 2021. In other embodiments, when the processors 201 and 202 do not integrate a memory controller, they can utilize a separately configured memory controller (such as the memory controller 300 shown in Figure 2) to perform data reading and writing in the memory 100 and to mark consumable / non-consumable states for the storage area; this is not limited.

[0092] Furthermore, in the embodiment shown in Figure 3 above, the example is that the processor 201 notifies the processor 202 to start consuming target data in multiple storage areas in the memory 100. In other embodiments, when the data processing system only includes the processor 201 and the processor 202, the processor 202 can continuously have the permission to consume data in all storage areas of the memory 100. Thus, after the processor 201 completes the initialization of multiple storage areas, it does not need to send a separate notification message to the processor 202 to instruct the processor 202 to consume the target data.

[0093] It is worth noting that other reasonable combinations of steps that can be conceived by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0094] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the above-described data consumption method.

[0095] This application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0096] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0097] The computer program product can be a software installation package. When any of the aforementioned data consumption methods is required, the computer program product can be downloaded and executed on a computing device.

[0098] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0099] The terminology used in the above embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, “one or more” refers to one, two, or more; the character “ / ” generally indicates that the preceding and following objects are in an “or” relationship. In the embodiments of this application, “simultaneously” means within the same time period, including situations where they are at the same moment. The terms “first,” “second,” etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate, and this is merely a way of distinguishing objects with the same attributes in the embodiments of this application.

[0100] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0101] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data consumption method, characterized in that, The method is applied to a data processing system including a first processor, a second processor and a memory, and includes: The first processor writes target data into a plurality of storage areas in the memory, different storage areas in the plurality of storage areas being used to store the same amount of data, wherein a storage area storing the target data is marked as consumable; In a process in which the first processor writes target data into the plurality of storage areas, the second processor identifies a first storage area, the first storage area being at least one storage area in the plurality of storage areas that is marked as consumable, and consumes target data in the first storage area.

2. The method of claim 1, wherein, The first storage area includes a data area and a non-data area, and the method further includes: After the data area stores the target data, the first processor adds a mark in the non-data area, the mark being used to indicate that the target data in the data area is consumable.

3. The method of claim 2, wherein, The first processor adding the mark in the non-data area includes: The first processor configures a value of a target field in the non-data area as a first value, the first value being used to indicate that the target data in the data area is consumable; Or, the first processor adds a first error correction code in the non-data area, the first error correction code being obtained by inverting a second error correction code corresponding to the target data in the data area, the first error correction code being used to indicate that the target data in the data area is consumable; Or, the first processor adds the second error correction code in the non-data area, the second error correction code being used to indicate that the target data in the data area is consumable.

4. The method of claim 3, wherein, The second processor identifying the first storage area in the plurality of storage areas that is marked as consumable includes: The second processor reads a value of a target field in the non-data area, and when the value of the target field is the first value, determines that the first storage area is marked as consumable; Or, the second processor calculates a third error correction code according to the target data in the data area, and when the second error correction code in the non-data area is the same as the third error correction code after inversion, determines that the first storage area is marked as consumable; Or, the second processor calculates a third error correction code according to the target data in the data area, and when the first error correction code in the non-data area is the same as the third error correction code, determines that the first storage area is marked as consumable.

5. The method according to any one of claims 1 to 4, characterized in that, Before the first processor writes target data into the plurality of storage areas in the memory, the method further includes: The first processor marks the plurality of storage areas as non-consumable; The first processor sends a first notification message to the second processor after determining that the plurality of storage areas are marked as the non-consumable state, and the first notification message is used to inform the second processor to consume data in the storage areas marked as the consumable state in the plurality of storage areas.

6. The method of claim 5, wherein, The data processing system further includes a third processor; The method further includes: After the second processor consumes all target data in the plurality of storage areas, the first processor re-marks the plurality of storage areas as the non-consumable state; The first processor sends a second notification message to the third processor after determining that the plurality of storage areas are re-marked as the non-consumable state, and the second notification message is used to inform the third processor to consume data in the storage areas marked as the consumable state in the plurality of storage areas.

7. The method according to any one of claims 1 to 6, characterized in that, The first processor writes target data into a plurality of storage areas in the memory, including: The first processor writes a plurality of data packets corresponding to the target data into the plurality of storage areas in parallel; Or, the first processor writes the plurality of data packets corresponding to the target data into the plurality of storage areas one by one.

8. A data processing system, characterized by The data processing system includes a first processor, a second processor, and a memory; The first processor is configured to write target data into a plurality of storage areas in the memory, and each of the plurality of storage areas is configured to store the same amount of data, wherein a storage area storing the target data is marked as a consumable state. The second processor is configured to identify a first storage area in a process in which the first processor writes target data into the plurality of storage areas, the first storage area being at least one storage area marked as the consumable state in the plurality of storage areas, and consume target data in the first storage area.

9. The system of claim 8, wherein, The first storage area includes a data area and a non-data area. The first processor is further configured to add a mark in the non-data area after the data area stores the target data, and the mark is used to indicate that the target data in the data area is in the consumable state.

10. The system of claim 9, wherein, The first processor is configured to: Configure a value of a target field in the non-data area as a first value, and the first value is used to indicate that the target data in the data area is in the consumable state; Or, add a first error correction code in the non-data area, the first error correction code being obtained by inverting a second error correction code corresponding to the target data in the data area, and the first error correction code is used to indicate that the target data in the data area is in the consumable state; Or, add the second error correction code corresponding to the target data in the data area in the non-data area, and the second error correction code is used to indicate that the target data in the data area is in the consumable state.

11. The system of claim 10, wherein, The second processor is configured to: Read a value of a target field in the non-data area, and when the value of the target field is the first value, determine that the first storage area is marked as the consumable state. Or, a third error correction code is calculated according to the target data in the data area, and when the second error correction code in the non-data area is the same as the third error correction code after being negated, it is determined that the first storage area is marked as consumable state. Or, a third error correction code is calculated according to the target data in the data area, and when the first error correction code in the non-data area is the same as the third error correction code, it is determined that the first storage area is marked as consumable state.

12. The system according to any one of claims 8 to 11, characterized in that, The first processor is further configured to: Before the first processor writes target data into a plurality of storage areas in the memory, mark the plurality of storage areas as non-consumable state; After determining that the plurality of storage areas are marked as non-consumable state, send a first notification message to the second processor, the first notification message being used to notify the second processor to consume data in storage areas marked as consumable state in the plurality of storage areas.

13. The system of claim 12, wherein, The data processing system further comprises a third processor, and the first processor is further configured to: After the second processor consumes all target data in the plurality of storage areas, re-mark the plurality of storage areas as non-consumable state; After determining that the plurality of storage areas are re-marked as non-consumable state, send a second notification message to the third processor, the second notification message being used to notify the third processor to consume data in storage areas marked as consumable state in the plurality of storage areas.

14. The system of any one of claims 8 to 13, wherein, The first processor is configured to: write a plurality of data packets corresponding to the target data into the plurality of storage areas in parallel; Or, write the plurality of data packets corresponding to the target data into the plurality of storage areas one by one.

15. A computer-readable storage medium, characterized in that, The computer program product comprises instructions which, when executed on a computing device, cause the computing device to perform the steps of the method of any one of claims 1 to 7.

16. A computer program product comprising instructions, characterized in that, The computer program product, when executed on at least one computing device, causes the at least one computing device to perform the steps of the method of any one of claims 1 to 7.

Images