Data processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a data processing method, device and equipment and storage medium, which can be applied to the technical field of near-memory computing. The data processing method comprises: dividing a to-be-processed data set into a plurality of sub-data sets according to the memory access behavior characteristics of the to-be-processed data set; determining target memory addresses of the plurality of sub-data sets according to the dynamic characteristics and static characteristics of the plurality of sub-data sets, so that the plurality of sub-data sets are respectively stored in the corresponding target memory addresses; predicting at least one target sub-data set associated with a target computing task from the plurality of sub-data sets; and accessing the at least one target sub-data set through the target memory address corresponding to the at least one target sub-data set to execute the target computing task locally.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of near memory computing, and in particular, to a data processing method and device, electronic equipment, storage medium and program product. BACKGROUND

[0002] Due to the long-term asynchronous development between memory technology and processor technology, the memory performance limitation makes it difficult for high-performance processors to perform their due functions. This memory bottleneck phenomenon is called "Memory Wall".

[0003] In order to solve the "Memory Wall" problem, the existing technology usually adopts a near memory processing (NMP) model to perform a computing task. The near memory processing model takes data as the core subject and places the computing components as close to the data storage location as possible, thereby shortening the transmission path between the data storage and the computing components, and thus reducing the memory access delay and power consumption.

[0004] However, the existing near memory processing model needs the processor to perform frequent communication to complete remote access and data transfer, which introduces large interrupt processing overhead and increases the access delay and power consumption of the remote. Alternatively, the processor will be assigned to many invalid computing tasks, resulting in a serious waste of a large amount of computing resources and memory bandwidth in the system, and a low effective utilization ratio. SUMMARY

[0005] In view of the above problems, the present disclosure provides a data processing method and device, electronic equipment, storage medium and program product.

[0006] According to a first aspect of the present disclosure, a data processing method is provided, comprising: dividing a to-be-processed data set into a plurality of sub-data sets according to the memory access behavior characteristics of the to-be-processed data set; determining target memory addresses of the plurality of sub-data sets according to the dynamic characteristics and static characteristics of the plurality of sub-data sets, so that the plurality of sub-data sets are respectively stored in the corresponding target memory addresses; predicting at least one target sub-data set associated with a target computing task from the plurality of sub-data sets; and accessing the at least one target sub-data set through the target memory address corresponding to the at least one target sub-data set, to execute the target computing task locally.

[0007] According to an embodiment of the present disclosure, the method for processing the data set comprises: obtaining data categories of a plurality of to-be-processed data in the data set; determining access behavior characteristics of the plurality of to-be-processed data according to the data categories, the access behavior characteristics comprising data volume, locality characteristic, access frequency of the to-be-processed data, and correlation between the plurality of to-be-processed data; dividing the to-be-processed data into a plurality of sub-data sets in a case where the data volume, the locality characteristic, and the access frequency of the to-be-processed data all meet a first preset condition; and dividing the plurality of to-be-processed data into the same sub-data set in a case where the correlation between the plurality of to-be-processed data meets a second preset condition; wherein each of the plurality of sub-data sets comprises at least one to-be-processed data, and each of the plurality of to-be-processed data is divided into at least one sub-data set.

[0008] According to an embodiment of the present disclosure, the method for processing the data set comprises: obtaining dynamic characteristics and static characteristics of the plurality of sub-data sets, the dynamic characteristics comprising calculation characteristics of the sub-data sets, and the static characteristics comprising data characteristics of the sub-data sets; mapping the plurality of sub-data sets into a plurality of data groups according to the calculation characteristics and the data characteristics, each of the plurality of data groups comprising at least one sub-data set, so that the plurality of data groups meet calculation load balancing or data load balancing; associating the plurality of data groups with a plurality of storage controllers respectively, so as to calculate target memory addresses of at least one sub-data set in the plurality of data groups according to a correspondence between the plurality of storage controllers and the memory; and storing the at least one sub-data set in the plurality of data groups into the corresponding target memory addresses.

[0009] According to an embodiment of the present disclosure, the method for processing the data set comprises: obtaining dynamic characteristics and static characteristics of the plurality of sub-data sets, the dynamic characteristics comprising calculation characteristics of the sub-data sets, and the static characteristics comprising data characteristics of the sub-data sets; mapping the plurality of sub-data sets into a plurality of data groups according to the calculation characteristics and the data characteristics, each of the plurality of data groups comprising at least one sub-data set, so that the plurality of data groups meet calculation load balancing or data load balancing; associating the plurality of data groups with a plurality of storage controllers respectively, so as to calculate target memory addresses of at least one sub-data set in the plurality of data groups according to a correspondence between the plurality of storage controllers and the memory; and storing the at least one sub-data set in the plurality of data groups into the corresponding target memory addresses.

[0010] According to an embodiment of the present disclosure, the method for processing the data set comprises: obtaining dynamic characteristics and static characteristics of the plurality of sub-data sets, the dynamic characteristics comprising calculation characteristics of the sub-data sets, and the static characteristics comprising data characteristics of the sub-data sets; mapping the plurality of sub-data sets into a plurality of data groups according to the calculation characteristics and the data characteristics, each of the plurality of data groups comprising at least one sub-data set, so that the plurality of data groups meet calculation load balancing or data load balancing; associating the plurality of data groups with a plurality of storage controllers respectively, so as to calculate target memory addresses of at least one sub-data set in the plurality of data groups according to a correspondence between the plurality of storage controllers and the memory; and storing the at least one sub-data set in the plurality of data groups into the corresponding target memory addresses.

[0011] According to an embodiment of the present disclosure, when the plurality of sub-data sets include N sub-data sets, the target computing task requires m specified sub-data sets, and at least one target sub-data set associated with the target computing task is predicted from the plurality of sub-data sets, the method further includes: predicting n target sub-data sets associated with the target computing task from the N sub-data sets, the n target sub-data sets including the m specified sub-data sets, and N ≥ n ≥ m.

[0012] According to an embodiment of the present disclosure, the at least one target sub-data set is accessed through a target memory address corresponding to the at least one target sub-data set to locally execute the target computing task, including: distributing the target computing task to a plurality of processors, the plurality of processors corresponding to a plurality of memory controllers; and accessing the corresponding target sub-data set in the target memory address by using the plurality of memory controllers, so that the corresponding plurality of processors locally execute the target computing task based on the corresponding target sub-data set.

[0013] According to an embodiment of the present disclosure, the to-be-processed data set includes a reference sequence and index data, and the to-be-processed data set is mapped into a plurality of sub-data sets according to the memory access behavior characteristics of the to-be-processed data set, including: dividing the reference sequence into a plurality of sub-sequence segments according to the memory access behavior characteristics of the reference sequence; constructing a plurality of sub-index data respectively according to the association of the index data and the plurality of sub-sequence segments, the plurality of sub-index data respectively corresponding to the plurality of sub-sequence segments; and mapping the plurality of sub-index data and the plurality of sub-sequence segments into the plurality of sub-data sets.

[0014] The second aspect of the present disclosure provides a data processing apparatus, including: a division module configured to divide a to-be-processed data set into a plurality of sub-data sets according to memory access behavior characteristics of the to-be-processed data set; a determination module configured to determine target memory addresses of the plurality of sub-data sets according to dynamic characteristics and static characteristics of the plurality of sub-data sets, so that the plurality of sub-data sets are respectively stored in the corresponding target memory addresses; a prediction module configured to predict at least one target sub-data set associated with a target computing task from the plurality of sub-data sets; and an access module configured to access the at least one target sub-data set through a target memory address corresponding to the at least one target sub-data set to locally execute the target computing task.

[0015] The third aspect of the present disclosure provides an electronic device, including: one or more processors; a memory configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the above method.

[0016] The fourth aspect of the present disclosure further provides a computer-readable storage medium having stored executable instructions, which are executed by a processor to cause the processor to perform the above method.

[0017] The fifth aspect of the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the above method.

[0018] The data processing method, device, equipment, medium and program product provided by the present disclosure ensure the locality of the data processing process through the data set blocking mechanism, which makes the processor complete the computing task only on the local data set. In addition, through task prediction distribution, the computing task is avoided to be distributed to the processor which does not need to participate in the calculation, and the redundancy of the data processing process is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure, taken in conjunction with the accompanying drawings, in which:

[0020] Figure 1 The system architecture diagram of the data processing method, device, equipment, medium and program product according to the embodiments of the present disclosure is schematically shown;

[0021] Figure 2 The flowchart of the data processing method according to the embodiments of the present disclosure is schematically shown;

[0022] Figure 3 The flowchart of the storage sub-data set according to the embodiments of the present disclosure is schematically shown;

[0023] Figure 4 The flowchart of the prediction sub-data set according to the embodiments of the present disclosure is schematically shown;

[0024] Figure 5 The flowchart of the prediction sub-data set according to the embodiments of the present disclosure is schematically shown;

[0025] Figure 6A The schematic diagram of the storage sub-data set according to the embodiments of the present disclosure is schematically shown;

[0026] Figure 6B The schematic diagram of the prediction sub-data set according to the embodiments of the present disclosure is schematically shown;

[0027] Figure 7 The structural block diagram of the data processing device according to the embodiments of the present disclosure is schematically shown; and

[0028] Figure 8 The block diagram of the electronic equipment suitable for implementing the data processing method according to the embodiments of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0029] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description of the embodiments, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it would be apparent to those skilled in the art that the embodiments, or one or more embodiments, can be practiced without these specific details. In other instances, well-known structures and

[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the term "including" "comprising" and the like are meant to be inclusive, but not limiting to the components, steps, operations and / or the features that are present in the described embodiments.

[0031] All terms used herein, including technical and scientific terms, have the meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of the specification, and should not be interpreted in an idealized or overly formal way.

[0032] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted to include any of one, all, or a combination thereof. For example, "a system having at least one of A, B, and C" should be interpreted to include a system having A alone, a system having B alone, a system having C alone, a system having A and B together, a system having A and C together, a system having B and C together, and / or a system having A, B, and C together, etc.

[0033] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (such as including but not limited to user personal information) involved in the technical solutions comply with the relevant legal regulations, necessary security measures are taken, and do not violate public order and good customs.

[0034] The embodiments of the present disclosure provide a data processing method, including: dividing a to-be-processed data set into a plurality of sub-data sets according to memory access behavior characteristics of the to-be-processed data set; determining target memory addresses of the plurality of sub-data sets according to dynamic characteristics and static characteristics of the plurality of sub-data sets, so that the plurality of sub-data sets are respectively stored in the corresponding target memory addresses; predicting at least one target sub-data set associated with a target computing task from the plurality of sub-data sets; and accessing the at least one target sub-data set through a target memory address corresponding to the at least one target sub-data set, to execute the target computing task locally.

[0035] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0036] Figure 1 The diagram illustrates a system architecture of a data processing method, apparatus, device, medium, and program product according to embodiments of the present disclosure.

[0037] like Figure 1 As shown, in the near-memory computing model system architecture 100, computing tasks 110 are distributed to multiple processors 120, 130, and 140. The memory controllers (MCs) 150, 160, and 170 corresponding to the processors 120, 130, and 140 access data in memory banks 180, enabling the processors 120, 130, and 140 to complete computing tasks 110 based on the accessed data. For example, memory bank 1 can be a local memory bank of processor 120, memory bank 2 can be a local memory bank of processor 130, and memory bank 3 can be a local memory bank of processor 140.

[0038] Figure 1 The number of processors, memory blocks, and memory blocks shown is for illustrative purposes only and does not limit the number of processors, memory blocks, and memory blocks in the near-memory computing model system architecture 100.

[0039] The system architecture 100 of the near-memory computing model can be a system architecture based on a near-3D Dynamic Random Access Memory (DRAM) computing model. 3D DRAM uses 3D stacking technology to stack multiple chips together and uses through-silicon vias (TSVs) or hybrid bonding (HB) technology to interconnect the multiple chips, enabling vertical data communication. In this model, the computing chip serves as the logic layer, and one or more DRAM chips serve as the storage layer.

[0040] 3D DRAM computing models can achieve near-memory computing by integrating the logic layer and the memory layer, reducing memory access latency during data processing. Furthermore, because TSV / HB technology allows the logic layer to directly access data in memory banks, 3D DRAM computing models offer excellent memory access bandwidth.

[0041] 3D DRAM computing model has a large number of spatially independent MC interfaces inside, and multiple MCs 150, 160, 170 can access data in the local bank through the MC interface. When multiple MCs 150, 160, 170 work at the same time, a huge internal access bandwidth will be generated. Multiple processors 120, 130, 140 corresponding to multiple MCs 150, 160, 170 are multiple processors inside the 3D DRAM.

[0042] Multiple MCs 150, 160, 170 can also access data in the remote bank through the interconnection network of the logic layer. For example, MC 150 accesses data in memory block 2 through the interconnection network of the logic layer. Since multiple MCs 150, 160, 170 are physically far apart on the wafer, a large communication delay between MCs can occur, so the delay of MC 150, 160, 170 accessing the remote bank through the interconnection network of the logic layer will be much larger than the delay of MC 150, 160, 170 directly accessing data in the local bank, and MC 150, 160, 170 accessing the remote bank will also introduce additional data communication overhead.

[0043] For example, the data set required by the computing task 110 can be distributed in different banks of the memory 180, and the computing task 110 is allocated to different processors 120, 130, 140 for parallel computing of global data. When the processor 120, 130, 140 completes the assigned computing task, it can access data in the local bank through the corresponding MC 150, 160, 170, respectively, or communicate with other processors through the interconnection network to access global data. For example, processor 120 accesses data in memory block 1 through corresponding MC 150, and processor 120 communicates with processor 130 through data remote storage access or remote function call to access data in memory block 2 through MC 160.

[0044] However, when accessing global data, frequent communication is required between processors to complete remote access and data transfer. Frequent communication will introduce a large interrupt processing overhead to the processor, and data transfer in the interconnection network will also increase the delay and power consumption of accessing the remote MC. In addition, although the 3D DRAM can realize inter-wafer interconnection, the communication bandwidth it can provide is likely to be much smaller than the memory access bandwidth provided by the on-chip MC, so the communication bandwidth of the processor accessing the remote bank through the interconnection network is limited.

[0045] For example, the data set required by the computing task 110 is divided into multiple independent sub-data sets and allocated to different banks, forming local banks of the MCs 150, 160, 170 respectively, which are independently accessed by the MCs 150, 160, 170, and the processors 120, 130, 140 respectively perform the computing task by data in the local banks. The computing task 110 is broadcast to all the processors 120, 130, 140 based on the broadcast model, and the processors 120, 130, 140 start corresponding calculations on the local data managed by the corresponding MCs 150, 160, 170 after receiving the computing task 110.

[0046] However, each computing task 110 usually involves a small number of sub-data sets, and each processor 120, 130, 140 receiving the computing task 110 does not necessarily need to process the computing task 110. But the broadcast model will send each computing task 110 to all processors 120, 130, 140 in order to ensure the accuracy and integrity of the calculation result, which makes some of the processors 120, 130, 140 be allocated a lot of invalid tasks, causing redundancy in the data processing process, and thus a large amount of computing resources and memory bandwidth in the 3D DRAM computing model are wasted, and the effective utilization ratio is low, which cannot fully exert the potential of the 3D DRAM.

[0047] Through the embodiments of the present disclosure, the sub-data sets required by the computing task are predicted based on the association between the computing task and the multiple sub-data sets, so that the computing task is only executed by the processors corresponding to the required sub-data sets, thereby ensuring high memory parallelism and good locality of the computing process, and fully exerting the advantages of high bandwidth and low latency of the 3D DRAM. The high memory parallelism ensures that the bandwidth provided by the multiple independent MCs can be fully utilized, and the locality ensures that the processors access data in the local banks through the local MCs as much as possible, and remote access is reduced as much as possible to avoid communication overhead and delay.

[0048] Figure 2 A flowchart of a data processing method according to an embodiment of the present disclosure is schematically shown.

[0049] As Figure 2 shown, the data processing method includes operations S210-S240.

[0050] In operation S210, the to-be-processed data set is divided into multiple sub-data sets according to the memory access behavior characteristics of the to-be-processed data set.

[0051] In the embodiments of the present disclosure, the to-be-processed data set is a data set related to a field involved in a target computing task. For example, the to-be-processed data set is data that should be stored in the memory before the computing task starts, such as a graph structure in a graph computing task, network structure data in a neural network, reference sequence and its index data in a genetic sequence matching, and the like.

[0052] The memory access behavior characteristics of the to-be-processed data in the to-be-processed data set are analyzed to perform coarse-grained blocking on the data set, and a plurality of independent sub-data sets are obtained. The plurality of independent sub-data sets can be processed in parallel by a plurality of processors.

[0053] In operation S220, target memory addresses of the plurality of sub-data sets are determined according to the dynamic characteristics and the static characteristics of the plurality of sub-data sets, so that the plurality of sub-data sets are respectively stored in the corresponding target memory addresses.

[0054] In the embodiments of the present disclosure, according to the dynamic characteristics and the static characteristics of the plurality of sub-data sets, the plurality of sub-data sets can be reordered and clustered, so as to determine the data layout of all sub-data sets in the 3D DRAM storage layer, and realize the spatial mapping of the sub-data sets to the 3D DRAM storage layer.

[0055] The sub-data sets located in the 3D DRAM storage layer are respectively accessed by a plurality of independent MCs, and each MC corresponds to a local data of the MC.

[0056] In operation S230, at least one target sub-data set associated with the target computing task is predicted from the plurality of sub-data sets.

[0057] In the embodiments of the present disclosure, a reconfigurable predictor can be used to predict the sub-data set involved in each input target computing task, determine at least one target sub-data set associated with the target computing task, and thus the target computing task can be distributed to the processor corresponding to the at least one target sub-data set. The processor corresponding to the at least one target sub-data set associated with the target computing task processes the target computing task, which can realize high memory parallelism and locality of the computing process, fully exert the advantages of high bandwidth and low latency of the 3D DRAM, and reduce communication overhead and latency.

[0058] In operation S240, the at least one target sub-data set is accessed through the target memory address corresponding to the at least one target sub-data set, so as to execute the target computing task locally.

[0059] According to the at least one target sub-data set predicted, the processor corresponding to the at least one target sub-data set associated with the target computing task executes the target computing task based on the local data, obtains a computing result, and can write the computing result back to a pre-defined address space through an on-chip network.

[0060] When each processor performs the target computing task based on the local data, each processor only needs to access the local data through the corresponding MC, without accessing the data between the remote MCs through the interconnection resources or any data interaction with other processors. The multiple processors can access the local data through the corresponding MCs in parallel, and complete different computing tasks in parallel, thereby improving the computing efficiency.

[0061] In the embodiments of the present disclosure, in operation S210, the operation of dividing the to-be-processed data set into multiple sub-data sets according to the memory access behavior characteristics of the to-be-processed data set can include: respectively acquiring data categories of multiple to-be-processed data in the to-be-processed data set; respectively determining memory access behavior characteristics of the multiple to-be-processed data according to the data categories, the memory access behavior characteristics including data quantity, locality characteristic, memory access frequency of the to-be-processed data, and correlation between the multiple to-be-processed data; in the case that the data quantity, the locality characteristic, and the memory access frequency of the to-be-processed data all meet the first preset condition, dividing the to-be-processed data into the multiple sub-data sets; and in the case that the correlation between the multiple to-be-processed data meets the second preset condition, dividing the multiple to-be-processed data into the same sub-data set.

[0062] For example, each of the multiple sub-data sets includes at least one to-be-processed data, and each of the multiple to-be-processed data is divided into at least one sub-data set.

[0063] The data category is a data type divided according to different record information, for example, nodes and edges in a graph structure belong to different data categories, and sequence information and index information in gene sequence matching belong to different data categories. The memory access behavior characteristics corresponding to the data category can include memory access characteristics within a single data category and correlation characteristics between multiple data categories.

[0064] For example, the memory access characteristics within a single data category can include data quantity size, locality characteristic, and memory access frequency characteristic, etc. The data quantity size includes the total quantity of to-be-processed data of the data category and the average data quantity of each access to the to-be-processed data of the data category in the target computing task. The locality characteristic is whether the data required by the target computing task is continuously accessed in the time domain or the space domain. The memory access frequency characteristic is the frequency of the to-be-processed data of the data category being accessed in the execution of the target computing task.

[0065] The correlation characteristics between multiple data categories are the association and dependency of the memory access behaviors between the to-be-processed data of multiple data categories in the target computing task. For example, whether the to-be-processed data of different data categories will be accessed at the same time, or the access of the to-be-processed data of one data category depends on the to-be-processed data of other data categories.

[0066] For a single data category, the data category with large data volume, weak memory locality, and high memory access frequency can be a key category. The first preset condition can be that the data volume is greater than a preset value, the memory locality is less than a preset value, and the memory access frequency is greater than a preset value. In the case where the data volume is large, the memory locality is weak, and the memory access frequency is high, the to-be-processed data is divided into a plurality of different sub-data sets, so that the to-be-processed data can be allocated to sufficient parallel access bandwidth after subsequent data set mapping is completed.

[0067] For to-be-processed data with small data volume and low memory access frequency, the to-be-processed data can be divided into fewer sub-data sets, so that the to-be-processed data is allocated to less access bandwidth after subsequent data set mapping is completed, and effective bandwidth is allocated to to-be-processed data with large data volume, weak memory locality, and high memory access frequency.

[0068] For example, corresponding weights can be set for different memory access behavior characteristics, and the weights are determined by specific application requirements. For example, in a gene sequence matching algorithm, the index data set has the characteristics of large data volume, large single access data volume, random access, and high memory access frequency, so the index data can be divided into all sub-data sets to ensure parallel access bandwidth.

[0069] For multiple data categories, to-be-processed data of data categories with strong correlation and dependency can be divided into the same one or more sub-data sets, so that the to-be-processed data can be allocated to the same memory area after subsequent data mapping is completed, thereby ensuring the locality and avoiding communication between sub-data sets. The second preset condition can be that the correlation and dependency between a plurality of to-be-processed data are greater than a preset value.

[0070] To-be-processed data of data categories with weak correlation can be divided into different sub-data sets. For example, in a gene sequence matching algorithm, reference sequence data fragments and index information corresponding to the reference sequence data fragments have strong correlation, and the reference sequence data fragments and the corresponding index information should be divided into the same sub-data set.

[0071] The dependency between the plurality of sub-data sets obtained by the dividing is weak, and the sub-data sets with dependency can be merged into larger sub-data sets, thereby ensuring that each sub-data set can independently complete the calculation after receiving a coarse-grained target calculation task, and does not need to depend on other sub-data sets during execution of a single target calculation task.

[0072] The plurality of sub-datasets have differences, so that the target computing task can be accurately allocated to fewer sub-datasets through prediction. The differences are related to specific application fields, for example, in genetic sequence matching, different reference sequence fragments have differences in base sequence distribution. When an input sequence is matched with a reference sequence, the sub-dataset related to the input sequence in the reference sequence fragment can be predicted through base information, so that the corresponding matching task can be completed by the input sequence.

[0073] Through the data set blocking mechanism, the locality of the data processing process is ensured, so that the processor only needs to complete the computing task on the local data set. Through task prediction distribution, the computing task is avoided to be distributed to the processor which does not need to participate in the calculation, and the wide redundancy problem of the data processing process is reduced.

[0074] Figure 3 A flowchart for storing sub-datasets is schematically shown according to an embodiment of the present disclosure.

[0075] As shown in Figure 3 Operation S220 determines target memory addresses of the plurality of sub-datasets according to dynamic characteristics and static characteristics of the plurality of sub-datasets, so that the plurality of sub-datasets are respectively stored in corresponding target memory addresses, which can include operations S321-S324.

[0076] In operation S321, dynamic characteristics and static characteristics of the plurality of sub-datasets are respectively acquired.

[0077] For example, the dynamic characteristics include computing characteristics of the sub-datasets, and the static characteristics include data characteristics of the sub-datasets. The computing characteristics include average bandwidth requirements and average computing loads of using the sub-datasets, etc. The data characteristics include data size of the sub-datasets. The average bandwidth requirements can be determined by the memory access frequency of all types of data in the sub-datasets, and the average computing load can be determined by static analysis of the target computing task under typical input data, and the data size can be determined by summing all types of data in the sub-datasets.

[0078] In operation S322, the plurality of sub-datasets are mapped into a plurality of data groups according to the computing characteristics and the data characteristics, so that the plurality of data groups satisfy computing load balancing or data load balancing.

[0079] For example, each data group in the plurality of data groups includes at least one sub-dataset.

[0080] In operation S323, the plurality of data groups are respectively associated with a plurality of storage controllers, so as to calculate target memory addresses of at least one sub-dataset in the plurality of data groups according to corresponding relationships between the plurality of storage controllers and the memory.

[0081] At operation S324, at least one sub-data set in the plurality of data sets is stored in a corresponding target memory address.

[0082] In the embodiments of the present disclosure, the plurality of sub-data sets is divided into a plurality of data sets by reorganizing and clustering the sub-data sets. The plurality of sub-data sets is reordered according to the computing features and the data features, and the plurality of sub-data sets is combined into a group. Each group of sub-data sets can correspond to a memory space managed by a single MC in the 3D DRAM, and a target computing task related to the group of sub-data sets is completed by the same processor corresponding to the MC.

[0083] When grouping the sub-data sets, different grouping optimization targets can be selected according to actual needs. For example, the total computing load of the sub-data sets in different groups can be similar to achieve computing load balancing among the plurality of data sets. Or the total bandwidth demand of the sub-data sets in different groups can be similar to achieve bandwidth utilization balancing among the plurality of data sets.

[0084] The spatial mapping of the sub-data sets to the DRAM can map the grouped sub-data sets according to the correspondence between the groups and the MCs of the 3D DRAM, and calculate the memory addresses corresponding to the to-be-processed data of various data categories of each sub-data set in the data group, so that the processor can access the to-be-processed data of the sub-data sets in the group through the corresponding memory addresses.

[0085] In the embodiments of the present disclosure, at operation S220, the target memory addresses of the plurality of sub-data sets are determined according to the dynamic features and the static features of the plurality of sub-data sets, so that the plurality of sub-data sets are respectively stored in the corresponding target memory addresses. The method can further include: respectively acquiring the storage capacities of the plurality of storage controllers in the memory; and under the constraint of the storage capacities, respectively associating the plurality of data sets with the plurality of storage controllers, so that the data amount of the data set is less than or equal to the storage capacity corresponding to the associated storage controller.

[0086] For example, each group of sub-data sets can be mapped in the memory space managed by a single MC in the 3D DRAM, so the sum of the data amounts corresponding to all the sub-data sets in each data group needs to be less than the storage space capacity corresponding to the single MC of the 3D DRAM, so as to ensure that each sub-data set can be successfully stored in the memory.

[0087] In the embodiments of the present disclosure, when the data set does not change, only one block processing is required for the to-be-processed data set, and the plurality of sub-data sets obtained by the block processing can be multiplexed by the plurality of target computing tasks.

[0088] Figure 4 A flowchart for predicting sub-data sets according to the embodiments of the present disclosure is schematically shown.

[0089] As Figure 4As shown, operation S230 of predicting at least one target sub-data set associated with the target computing task from the plurality of sub-data sets can include operations S431-S433.

[0090] In operation S431, the operation logic of the target computing task is obtained.

[0091] In operation S432, the correlation degrees of the plurality of sub-data sets and the operation logic are respectively calculated.

[0092] In operation S433, at least one target sub-data set is determined from the plurality of sub-data sets according to the correlation degrees.

[0093] In the embodiments of the present disclosure, the prediction process can be implemented by a reconfigurable predictor based on certain prediction logic. The prediction logic of the reconfigurable predictor can be reconfigured by writing configuration information and the like to implement the optimal task prediction logic required in different application fields. For example, in the application of gene sequence matching, the prediction logic can be the similarity between the input sequence and the reference sequence fragment, and the sub-data set in which all the reference sequence fragments with a similarity exceeding a threshold are located is taken as the prediction result.

[0094] The optimal task prediction logic of the reconfigurable predictor can ensure high accuracy and low redundancy of the prediction process, and the calculation speed is fast. For example, the predictor excludes part of irrelevant sub-data sets from the plurality of sub-data sets according to the optimal task prediction logic, and distributes the computing task prediction to at least one target sub-data set that can be related.

[0095] For example, the prediction process of the target computing task can be a one-to-many mapping process. The plurality of sub-data sets include N sub-data sets, m specified sub-data sets are required to execute the target computing task, n target sub-data sets associated with the target computing task are predicted from the N sub-data sets, the n target sub-data sets include the m specified sub-data sets, and N≥n≥m.

[0096] For example, the n sub-data sets contain the m specified sub-data sets to ensure high accuracy of the calculation result. The number n of target sub-data sets is close to the number m of specified sub-data sets, so that when the target computing task is executed, too many irrelevant sub-data sets are not introduced, so as to avoid redundant calculation and memory access overhead and reduce redundancy.

[0097] In the embodiments of the present disclosure, based on the near-memory computing requirements in application fields, the optimal task prediction logic can be determined according to the specific algorithm characteristics and data set characteristics, so that the near-memory computing model provided by the present disclosure can be transplanted into the corresponding application scenarios, improving the universality.

[0098] Figure 5 A flowchart of predicting a sub-data set according to an embodiment of the present disclosure is schematically shown.

[0099] As Figure 5 shown, operation S240 can include operation S541 and operation S542.

[0100] In operation S541, the target computing task is distributed to a plurality of processors corresponding to a plurality of memory controllers.

[0101] In operation S542, the corresponding target sub-data set in the target memory address is accessed by the plurality of memory controllers, and the corresponding plurality of processors locally execute the target computing task based on the corresponding target sub-data set.

[0102] In the embodiments of the present disclosure, the computing task can be delivered to the corresponding processor by the chip input end, and the task delivery can be achieved through a bus, a network on chip, or the like.

[0103] The processor can complete the target computing task based on the local data in the local bank. When each processor runs the target computing task, it only needs to access the local data in the target memory address through the corresponding MC, and does not access the data between remote MCs through the interconnection resource or perform any data interaction with other processors. The plurality of processors can access the corresponding MC in parallel and complete different target computing tasks in parallel.

[0104] For a plurality of different target computing tasks, the plurality of target computing tasks can be input into the 3D DRAM in a coarse-grained pipeline manner, and the prediction, distribution, and computing processes are sequentially performed by the predictor, the network on chip, and the processor to complete the complete computing process of the target computing task.

[0105] After the processor completes the target computing task, the computing result of the target computing task obtained by the processor can be written into a specified result storage address. The address can be a certain piece of space in the 3D DRAM, or an external other storage system. The process of result delivery can also be completed by the network on chip.

[0106] The present disclosure also provides a data processing method applied to a genetic sequence matching scenario, which combines Figure 6A and Figure 6B schematically illustrates the data processing method. Figure 6A A schematic diagram of a storage sub-data set according to an embodiment of the present disclosure is schematically shown. Figure 6B A schematic diagram of a prediction sub-data set according to an embodiment of the present disclosure is schematically shown.

[0107] In this embodiment, the dataset to be processed includes a given reference sequence R and the corresponding index data I. The target computation task is to input a set of short sequences {r1, r2, ..., r...} i , ...r n}, for each short sequence r i Map one or more positions onto a reference sequence R such that the subsequence at the corresponding position in R and r i The similarity is the highest.

[0108] Analysis of the memory access characteristics of the reference sequence R and the index data I reveals that the reference sequence R has a relatively small data volume, strong locality, and relatively low memory access frequency, while the index data I has a large data volume, poor memory access locality, and high memory access frequency. There is a strong correlation and dependency between the reference sequence R and the index data I; each matching task always accesses the index data I first, and then accesses the corresponding reference sequence R based on the access result.

[0109] The process of dividing the reference sequence R and index data I into multiple subsets may include: dividing the reference sequence into multiple subsequence segments based on the memory access behavior characteristics of the reference sequence; constructing multiple sub-index data based on the correlation between the index data and the multiple sub-sequence segments, with each sub-index data corresponding to a different sub-sequence segment; and mapping the multiple sub-index data and the multiple sub-sequence segments to multiple subsets.

[0110] like Figure 6A As shown, based on the memory access characteristics of the reference sequence R and the index data I, the reference sequence R and the index data I can be coarsely segmented, directly dividing the reference sequence R into multiple subsequence segments {R1, R2, ..., R...}. j , ...R m The global index data I cannot be directly linearly divided; however, it can be divided by processing each subsequence R. i Reconstructing the index yields the block-based index data {I1, I2, ..., I...} i , ...I m}, where I i For R j The index data, where m is a positive integer, i≤m, j≤m.

[0111] I i and R j Their memory access characteristics are complementary, for I i and R j Access between them is dependent, therefore I can i and R i The data are divided into the same subset. All data subsets can be {(I1, R1), (I2, R2), ..., I...} i R j ), …(Im R m Let S be the denote... ij =(I i R j All sub-datasets S after partitioning ij They are independent of each other and can complete the matching task in parallel, and because R... j The subset S has different base sequences. ij The differences between them facilitate prediction by subsequent predictors.

[0112] For all subsets S i Reorganization and clustering are performed because all subsets S ij The included data are similar because each subset S ij The total amount of data is basically the same, but each subset S ij The bandwidth requirements and computing load may differ.

[0113] For example, all subsets S can be evaluated based on computational load. ij Recombination and clustering are performed. The input sequence is sampled to obtain q input sequences {r}. p1 r p2 …r pq}, where q is a positive integer. Given q input sequences {r} p1 r p2 …r pq The dataset S is matched against R, and the total number of alignment tasks assigned to each subsequence segment of R is used as the computational load metric for each subsequence segment of R. The entire subset S is then matched against R. ij The clusters are divided into multiple data groups, each containing one or more subsets. A constraint is that the total size of the subsets within each data group does not exceed the DRAM space size corresponding to each MC.

[0114] To minimize the difference in total computational cost between multiple data sets, multiple subsets S are processed in an optimization problem-solving manner. ij Clustering results in multiple data groups. For example, using the linear bin packing algorithm, M groups are obtained, denoted as {C1, C2, ..., C...}. i , ...C M}, where M is a positive integer and i ≤ M. Let C... i The data is mapped to the address space corresponding to the i-th MC in the 3D DRAM, thus completing the spatial mapping of the data.

[0115] like Figure 6B As shown, the predictor is used for each input sequence r to be matched. i As a candidate subset, prediction is performed, and the prediction result is i. n Target subdatasets For example, the predictor can predict r i according to the similarity between the input sequence r j and the reference sequence R i . Whether r j should be distributed to the processor where R j is located is calculated. For example, the number of hits of the seed in r i in R j can be quickly calculated using an algorithm such as a Bloom filter, and the total number of hits is used as a similarity measure. When the similarity measure is greater than a preset threshold, r i will be distributed to R j . After the prediction is completed, r i is delivered to all processors where R j is located through the network on a chip. At the same time, the predictor can process the next input sequence, so that the prediction and matching of multiple input sequences are completed in a pipelined manner.

[0116] All processors that receive the input sequence r i independently and in parallel complete the matching task in the target computing task, and each processor accesses the local sub-data set S ij through the corresponding MC to complete sequence alignment work on r i and R j . After the matching is completed, the processor transmits the alignment result through the interconnection network to the external storage system through the network on a chip, saves the computing result, and waits to receive the next input sequence.

[0117] In the embodiments of the present disclosure, the data processing method based on the near-memory computing model provided by the present disclosure is data-centric, has high computing and memory access parallelism and locality, and can eliminate redundancy. In addition, the data processing method provided by the present disclosure is suitable for various application fields, can comprehensively consider the spatial distribution characteristics of data sets and computing tasks, and has flexibility and high efficiency.

[0118] Based on the above data processing method, the present disclosure further provides a data processing apparatus. The data processing apparatus will be described in detail below. Figure 7

[0119] Figure 7 The structure block diagram of the data processing apparatus according to the embodiments of the present disclosure is schematically shown.

[0120] As shown in Figure 7 , the data processing apparatus 700 of this embodiment includes a division module 710, a determination module 720, a prediction module 730, and an access module 740.

[0121] The dividing module 710 is configured to divide the to-be-processed data set into a plurality of sub-data sets according to the memory access behavior characteristics of the to-be-processed data set. In an embodiment, the dividing module 710 can be configured to perform operation S210 described above, and details are not repeated here.

[0122] The determining module 720 is configured to determine, according to the inventory task list, the asset attributes of the to-be-inventoried assets, the asset attributes including known assets and unknown assets. In an embodiment, the determining module 720 can be configured to perform operation S220 described above, and details are not repeated here.

[0123] The predicting module 730 is configured to predict at least one target sub-data set associated with the target computing task from the plurality of sub-data sets. In an embodiment, the predicting module 730 can be configured to perform operation S230 described above, and details are not repeated here.

[0124] The accessing module 740 is configured to access the at least one target sub-data set through the target memory address corresponding to the at least one target sub-data set, so as to execute the target computing task locally. In an embodiment, the accessing module 740 can be configured to perform operation S240 described above, and details are not repeated here.

[0125] According to an embodiment of the present disclosure, any one or more of the dividing module 710, the determining module 720, the predicting module 730, and the accessing module 740 can be combined in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of other modules, and implemented in one module. According to an embodiment of the present disclosure, at least one of the dividing module 710, the determining module 720, the predicting module 730, and the accessing module 740 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, etc. hardware or firmware, or any one of software, hardware and firmware or any appropriate combination of any of them. Alternatively, at least one of the dividing module 710, the determining module 720, the predicting module 730, and the accessing module 740 can be at least partially implemented as a computer program module which can perform corresponding functions when running.

[0126] Figure 8 A block diagram of an electronic device suitable for implementing the data processing method according to an embodiment of the present disclosure is schematically shown.

[0127] As Figure 8As shown, the electronic device 800 according to embodiments of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 802 or loaded into a random access memory (RAM) 803 from a storage section 808. The processor 801 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chip set, and / or a dedicated microprocessor (e.g., an application specific integrated circuit (ASIC)), and so on. The processor 801 can also include an on-board memory for cache use. The processor 801 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present disclosure.

[0128] In the RAM 803, various programs and data required for the operation of the electronic device 800 are stored. The processor 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. The processor 801 performs various operations of the method processes according to embodiments of the present disclosure by executing the programs in the ROM 802 and / or the RAM 803. Note that the programs can also be stored in one or more memories other than the ROM 802 and the RAM 803. The processor 801 can also perform various operations of the method processes according to embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0129] According to embodiments of the present disclosure, the electronic device 800 can further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 can further include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as necessary. A removable recording medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 810 as necessary, so that a computer program read therefrom is installed into the storage section 808 as necessary.

[0130] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which when executed, implement the method according to embodiments of the present disclosure.

[0131] According to an embodiment of the present disclosure, the computer readable storage medium can be a nonvolatile computer readable storage medium, for example, can include but not limited to: a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a portable compact disc read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer readable storage medium can include the ROM 802 and / or the RAM 803 described above and / or one or more memory other than the ROM 802 and the RAM 803.

[0132] Embodiments of the present disclosure also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the data processing method provided by the embodiments of the present disclosure.

[0133] The above functions defined in the system / device of the embodiments of the present disclosure are performed when the computer program is executed by the processor 801. According to an embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by computer program modules.

[0134] In one embodiment, the computer program can rely on tangible storage media such as optical storage media, magnetic storage media, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of signals on a network medium, and be downloaded and installed through the communication part 809, and / or be installed from the detachable medium 811. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to: wireless, wired, etc., or any appropriate combination thereof.

[0135] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or be installed from the detachable medium 811. When the computer program is executed by the processor 801, the above functions defined in the system of the embodiments of the present disclosure are performed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0136] According to embodiments of the present disclosure, program code of the computer program for performing the methods provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, and can be implemented in a computer program product. Specifically, the computer program can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. The programming language includes, but is not limited to, Java, C++, python, “C” language, or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, and partly on a remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider (ISP).

[0137] The computer program product of the present disclosure can be a computer program product, which is a machine-readable medium (or computer readable medium) having stored therein a sequence of instructions executable by a machine such as a personal digital assistant (PDA), a laptop, a desktop computer, or a server. Alternatively, the computer program product can be a propagated signal per se generated by using the program code, and the program code can be stored in a computer readable medium in a server, a database, and the like. The service provider of the present disclosure can provide a routine for implementing the present disclosure to a user or client who accesses the service via a network such as the Internet. The program code can be provided to the user terminal via the network while the user is logged on to the service provider's system. According to a preferred embodiment of the present disclosure, the program code can be downloaded into the user's system from the service provider's system as a part of a service provided by the service provider when the user is using the service, and thus the present disclosure can be implemented in the user's system. The program code can be stored in the memory of the user's system (RAM, for example) and is ready to be executed by the processor(s) of the computer to execute the acts of the method provided by the embodiments of the present disclosure.

[0138] It will be appreciated by persons skilled in the art that features of various embodiments and / or claims of the present disclosure can be combined or / and integrated with one another, even though such combinations or integrations are not expressly disclosed in the present disclosure. In particular, features of various embodiments and / or claims of the present disclosure can be combined and / or integrated with one another without departing from the spirit and teachings of the present disclosure. All such combinations and / or integrations are within the scope of the present disclosure.

[0139] The above describes embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims

1. A data processing method, comprising: Based on the memory access behavior characteristics of the dataset to be processed, the dataset to be processed is divided into multiple sub-datasets, including: obtaining the data categories of multiple data to be processed in the dataset to be processed; determining the memory access behavior characteristics of the multiple data to be processed according to the data categories, wherein the memory access behavior characteristics include the data volume, locality characteristics, memory access frequency, and correlation between the multiple data to be processed; if the data volume, locality characteristics, and memory access frequency of the data to be processed all meet a first preset condition, the data to be processed is divided into multiple sub-datasets; if the correlation between the multiple data to be processed meets a second preset condition, the multiple data to be processed is divided into the same sub-dataset; wherein each sub-dataset in the multiple sub-datasets includes at least one data to be processed, and each data to be processed in the multiple data to be processed is divided into at least one sub-dataset; Based on the dynamic and static characteristics of the multiple subsets, the target memory addresses of the multiple subsets are determined, so that the multiple subsets are stored in their respective target memory addresses. Predict at least one target subset of data from the plurality of subsets that is associated with the target computation task; and The at least one target subset is accessed via the target memory address corresponding to the at least one target subset to execute the target computation task locally.

2. The data processing method according to claim 1, wherein, The dynamic and static features of the multiple subsets of data are used as a basis. Determining the target memory addresses of the plurality of subsets of data, such that each subset of data is stored in its corresponding target memory address, includes: The dynamic features and static features of the multiple subsets are obtained respectively. The dynamic features include the computational features of the subsets, and the static features include the data features of the subsets. Based on the computational features and the data features, the plurality of subset datasets are mapped into a plurality of data groups, each of the plurality of data groups including at least one subset dataset, so that the plurality of data groups satisfy computational load balancing or data load balancing. The plurality of data groups are associated with a plurality of storage controllers, respectively, to calculate the target memory address of at least one subset of the plurality of data groups based on the correspondence between the plurality of storage controllers and memory; and Store at least one subset of the multiple data groups into the corresponding target memory address.

3. The data processing method according to claim 2, wherein, Associating the plurality of data groups with the plurality of storage controllers includes: Obtain the storage capacity of each of the plurality of storage controllers in memory; Under the constraint of the storage capacity, the multiple data groups are associated with multiple storage controllers respectively, such that the data volume of the data group is less than or equal to the storage capacity corresponding to the associated storage controller.

4. The data processing method according to claim 1, wherein, The step of predicting at least one target subset of data associated with the target computation task from the plurality of subsets includes: Obtain the computational logic of the target computation task; Calculate the correlation degree between the multiple subsets of data and the computational logic, respectively; and Based on the correlation degree, at least one target sub-dataset is determined from the plurality of sub-datasets.

5. The data processing method according to claim 4, wherein, The plurality of subsets includes N subsets, and m specified subsets are required to perform the target computation task. The step of predicting at least one target subset associated with the target computation task from the plurality of subsets further includes: From the N subsets of data, predict n target subsets associated with the target computation task, wherein the n target subsets include the m specified subsets, and N ≥ n ≥ m.

6. The data processing method according to claim 2, wherein, The step of accessing the at least one target subset of data points through the target memory address corresponding to the at least one target subset of data points to execute the target computation task locally includes: The target computing task is distributed to multiple processors, each corresponding to a multiple memory controller; and The multiple storage controllers access the corresponding target subset in the target memory address, so that the multiple processors respectively execute the target computing task locally based on the corresponding target subset.

7. The data processing method according to claim 2, wherein, The dataset to be processed includes a reference sequence and index data. The step of mapping the dataset to be processed into multiple sub-datasets based on its memory access behavior characteristics includes: Based on the memory access behavior characteristics of the reference sequence, the reference sequence is divided into multiple sub-sequence segments; Based on the correlation between the index data and the plurality of sub-sequence segments, a plurality of sub-index data are constructed, each sub-index data corresponding to one of the plurality of sub-sequence segments; and The multiple sub-index data and the multiple sub-sequence fragments are mapped into multiple sub-datasets.

8. A data processing apparatus, comprising: A partitioning module is used to partition the dataset to be processed into multiple sub-datasets based on the memory access behavior characteristics of the dataset to be processed. This includes: obtaining the data categories of multiple data sets to be processed in the dataset; determining the memory access behavior characteristics of the multiple data sets to be processed based on the data categories, wherein the memory access behavior characteristics include the data volume, locality characteristics, memory access frequency, and correlation between the multiple data sets to be processed; partitioning the data sets to be processed into multiple sub-datasets if the data volume, locality characteristics, and memory access frequency all meet a first preset condition; and partitioning the multiple data sets to be processed into the same sub-dataset if the correlation between the multiple data sets to be processed meets a second preset condition. Each sub-dataset in the multiple sub-datasets includes at least one data set to be processed, and each data set to be processed is partitioned into at least one sub-dataset. The determining module is used to determine the target memory address of the multiple sub-data sets based on the dynamic and static characteristics of the multiple sub-data sets, so that the multiple sub-data sets are respectively stored in the corresponding target memory address; A prediction module is configured to predict at least one target subset of data from the plurality of subsets that is associated with the target computation task; and An access module is used to access the at least one target subset of data through the target memory address corresponding to the at least one target subset of data, so as to execute the target computing task locally.

9. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images