Data stream parallel acceleration processing method and system in chip design

Abstract

The invention provides a data stream parallel acceleration processing method and system in chip design, and relates to the technical field of chip design, and the method comprises the steps: partitioning a data stream through a preset scale, building a computing resource pool with graded computing power and a storage resource pool with graded access speed, and achieving the optimal distribution of storage resources and calculation tasks; dividing parallel processing units, and determining a data processing sequence; setting a shared cache area to establish a distributed cache synchronization network, and realizing synchronization control based on a cache identifier and a Lamport logic clock mechanism; and generating a resource competition degree matrix, and rearranging the task according to the access popularity map. According to the method, the data stream processing efficiency and the resource utilization rate of the chip are improved, and resource competition conflicts are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of chip design, and particularly relates to a data flow parallel acceleration processing method and system in chip design. BACKGROUND

[0002] With the rapid development of chip design and application fields, data flow processing has become a core problem in modern chip design. In particular, in computing-intensive application scenarios such as big data, artificial intelligence, and image processing, efficient data flow parallel acceleration processing methods are crucial to improving chip performance. Traditional data flow processing usually adopts a single pipeline architecture or a simple parallel computing model, which is difficult to meet the growing complex computing needs.

[0003] Current data flow processing technology in chip design is usually based on a fixed computing resource allocation strategy, which statically allocates data blocks to computing units for processing. Although this method can provide certain performance improvement in specific application scenarios, with the increase of data size and computing complexity, existing technologies face many challenges and limitations.

[0004] The main defects and deficiencies existing in the prior art are embodied in the following aspects: the existing data flow processing method lacks effective hierarchical management of computing resources and storage resources, and cannot dynamically adjust the resource allocation strategy according to the data access characteristics and computing task complexity, resulting in low resource utilization and limited processing efficiency. In the traditional parallel processing architecture, the data synchronization and cache management mechanism between adjacent processing units is imperfect, and lacks an efficient cache consistency control strategy, which easily causes data competition and conflict, affecting system stability and processing speed. The existing technology is difficult to effectively deal with the resource competition problem between parallel processing units, and lacks a dynamic monitoring and task rearrangement mechanism for the competition state. When multiple processing units simultaneously access shared resources, it often leads to system performance degradation, and even serious problems such as deadlock and starvation. SUMMARY

[0005] The embodiments of the present application provide a data flow parallel acceleration processing method and system in chip design, which can solve the problems in the prior art.

[0006] In a first aspect of the embodiments of the present application, a data flow parallel acceleration processing method in chip design is provided, comprising:

[0007] dividing the data flow into a plurality of to-be-processed data blocks according to a preset size;

[0008] establish a computing resource pool and a storage resource pool, perform computing power grading on computing units in the computing resource pool, and organize progressive flow water scheduling according to processing capacity rising rates; grade the storage resource pool according to access speeds, and based on access accelerations of to-be-processed data blocks and key computing paths, realize allocation of the to-be-processed data blocks among different access speed storage levels and scheduling of computing tasks among different computing power computing units; and divide corresponding storage spaces for each computing unit in the corresponding storage layer;

[0009] divide each computing unit into a plurality of parallel processing units, determine a data processing order based on the time sequence dependency relationship, and distribute to-be-processed data blocks to corresponding parallel processing units according to the processing order;

[0010] establish a distributed cache synchronization network by setting a shared cache area between adjacent parallel processing units, allocate a unique cache identifier for each shared cache area, establish a cache access priority sequence using the cache identifier, and based on the cache access priority sequence, synchronize control of read-write states of cache data through a Lamport logical clock timestamp mechanism and an atomic lock operation;

[0011] generate a resource competition degree matrix between parallel processing units and calculate a competition load value, when the competition load value exceeds a preset competition saturation threshold, rearrange computing tasks based on an access heat map, and coordinate execution orders of computing tasks in a group based on the Lamport logical clock timestamp.

[0012] perform computing power grading on computing units in the computing resource pool, and organize progressive flow water scheduling according to processing capacity rising rates, including:

[0013] test computing flux of the computing unit under different instruction types, and generate instruction execution characteristics of the computing unit;

[0014] based on the instruction execution characteristics, construct an I / O performance cube of the computing unit, calculate a spatial vector of each computing unit in the I / O performance cube, and the spatial vector forms a performance trajectory curve with load changes;

[0015] calculate Hausdorff distances between performance trajectory curves of different computing units, divide computing units with the Hausdorff distances lower than a preset performance similarity threshold into the same performance level, and obtain a plurality of performance computing unit groups;

[0016] based on the performance trajectory curve slope of the computing unit, calculate the processing capacity rising rate in each computing unit group, organize the computing units into a progressive flow pipeline according to the rising rate, and allocate the computing unit with the highest rising rate to the end of the progressive flow pipeline;

[0017] Real-time monitoring of performance curves of each stage of the pipeline, when the performance of a stage of the pipeline appears jitter, calculating the first derivative of the performance curve, calculating the performance decay inflection point according to the derivative trend, migrating the task to the standby computing unit before reaching the decay inflection point.

[0018] Classifying the storage resource pool according to access speed, based on the access acceleration of the to-be-processed data block and the key computing path, realizing the allocation of the to-be-processed data block among different access speed storage levels and the scheduling of the computing task among different computing power computing units, including:

[0019] Calculating the standard deviation of the adjacent access time of the to-be-processed data block to obtain the time volatility rate, and extracting the periodic access feature vector of the to-be-processed data block according to the Fourier transform result of the time volatility rate;

[0020] Statistically counting the number of access transitions between to-be-processed data blocks to generate a Markov transition matrix, and performing eigenvalue decomposition on the Markov transition matrix to obtain the steady-state access probability and convergence time of the to-be-processed data block;

[0021] Based on the periodic access feature vector, steady-state access probability and convergence time, extracting the access mode features of the to-be-processed data block through principal component analysis, calculating the Euclidean distance of the access mode features of the to-be-processed data block, and aggregating the to-be-processed data blocks with a Euclidean distance less than a dynamic clustering radius to form a data block group;

[0022] Statistically counting the access delay and access times of the data block group in each storage layer, using the access times as the abscissa and the access delay as the ordinate, constructing an access performance curve using polynomial fitting, calculating the first derivative of the access performance curve to obtain the access acceleration, and dynamically distributing the data block group to the corresponding storage layer based on the extreme point of the access acceleration;

[0023] Setting the to-be-processed data block as a graph node, setting the dependency relationship between the data blocks as a directed edge, and setting the data flow transition frequency as an edge weight to obtain an initial computing graph, extracting a strongly connected component in the initial computing graph, calculating the maximum flow minimum cut of the strongly connected component to obtain a key computing path set, and distributing a corresponding task subset to different computing power computing units based on the key computing path set.

[0024] Extracting a strongly connected component in the initial computing graph, calculating the maximum flow minimum cut of the strongly connected component to obtain a key computing path set, and distributing a corresponding task subset to different computing power computing units based on the key computing path set, including:

[0025] performing a depth-first search in the initial calculation graph obtains a strongly connected component, determining a source point and a sink point based on edge weights within the strongly connected component, searching for a reachable path from the source point to the sink point in a residual network of the strongly connected component in an iterative manner, obtaining a maximum flow minimum cut when there is no reachable path from the source point to the sink point in the residual network, and the edge set corresponding to the maximum flow minimum cut is the critical calculation path set;

[0026] statistically obtaining the path depth of each path in the critical calculation path set, obtaining the path criticality by weighting the path depth, the sum of node weights on the path, and the sum of edge weights, calculating the ratio of the calculation capability of each calculation unit to the calculation demand of the path, and generating the matching degree of the path and the calculation unit in combination with the physical distance between the calculation unit and the path node;

[0027] based on the path criticality and the matching degree, the task subset corresponding to the critical calculation path is allocated to different calculation units.

[0028] A distributed cache synchronization network is established by setting a shared cache area between adjacent parallel processing units, a unique cache identifier is allocated to each shared cache area, a cache access priority sequence is established using the cache identifier, and the read-write state of cache data is synchronized and controlled through a Lamport logical clock timestamp mechanism and an atomic lock operation based on the cache access priority sequence, including:

[0029] The shared cache areas are connected to each other through cache data synchronization channels to form a distributed cache synchronization network, and a globally unique cache identifier is generated by multiplying the horizontal position index of the shared cache area in the distributed cache synchronization network by the number of columns and adding the vertical direction index position;

[0030] For each data item in each shared cache area, the current system clock time is obtained as an initial timestamp, when a read-write operation occurs to the data, the initial timestamp of the data item is compared with the maximum timestamp of the adjacent shared cache area, and the larger value plus one is selected as the Lamport logical clock timestamp of the data item;

[0031] The cache identifier and the Lamport logical clock timestamp are weighted and calculated to generate a data access priority sequence;

[0032] According to the data access priority sequence from high to low, the atomic lock of each data item is obtained in sequence to control the data access authority, when it is detected that there is a read operation on the data item, the atomic lock blocks the subsequent write operation until all read operations are completed, and then the write operation is performed;

[0033] The Lamport logical clock timestamp is updated after the data read / write operation is completed, and the atomic lock is released to allow subsequent data access operations.

[0034] A resource competition degree matrix between parallel processing units is generated, and a competition load value is calculated, and when the competition load value exceeds a preset competition saturation threshold, the computing tasks are rearranged based on an access heat map, and the execution order of the computing tasks in a group is coordinated based on the Lamport logical clock timestamp, including:

[0035] The number of cache access overlaps between each pair of parallel processing units is counted, and an access conflict matrix is generated, the Euclidean distance between feature vectors of each pair of parallel processing units is calculated, and a performance difference matrix is generated, the access conflict matrix and the performance difference matrix are weighted and combined to generate a resource competition degree matrix, and the competition load value of each parallel processing unit is calculated through the resource competition degree matrix;

[0036] An access counter is set in a shared cache area to record the number of times and time intervals that each data block is accessed by different parallel processing units in a unit time window, the heat value of the data block is calculated according to the product of the number of accesses and the time interval, the heat value is arranged in the order of the physical address of the data block, and a heat map reflecting the data access mode is generated;

[0037] When the competition load value of any parallel processing unit exceeds a preset competition saturation threshold, a data block group with a heat value difference less than a preset heat difference threshold and a continuous address is identified from the heat map, the computing tasks accessing these data block groups are marked as related tasks with data locality, and the related tasks are reassigned to the same computing unit group for execution;

[0038] A Lamport logical clock timestamp is assigned to each computing task in a computing unit group according to the heat value of the data block accessed by the computing task, the execution priority of the task in the group is determined based on the timestamp, and the task with the smallest timestamp is executed preferentially.

[0039] In a second aspect of the embodiment of the application, a data flow parallel acceleration processing system in chip design is provided, including:

[0040] A first unit is configured to divide the data flow into a plurality of to-be-processed data blocks according to a preset scale;

[0041] The second unit is configured to establish a computing resource pool and a storage resource pool, perform computing power classification on computing units in the computing resource pool, and organize progressive flow scheduling according to processing capacity rising rates; the storage resource pool is classified according to access speeds, and distribution of to-be-processed data blocks among different access speed storage levels and scheduling of computing tasks among different computing power computing units are realized based on access accelerations of the to-be-processed data blocks and key computing paths; a corresponding storage space is divided for each computing unit in a corresponding storage layer;

[0042] The third unit is configured to divide each computing unit into a plurality of parallel processing units, determine a data processing sequence based on the time sequence dependency relationship, and distribute to-be-processed data blocks into corresponding parallel processing units according to the processing sequence.

[0043] The fourth unit is configured to establish a distributed cache synchronization network by setting shared cache areas between adjacent parallel processing units, allocate a unique cache identifier for each shared cache area, establish a cache access priority sequence using the cache identifier, and perform synchronization control on read and write states of cache data based on the cache access priority sequence, a Lamport logical clock timestamp mechanism, and an atomic lock operation.

[0044] The fifth unit is configured to generate a resource competition degree matrix between parallel processing units and calculate a competition load value, rearrange computing tasks based on an access heat map when the competition load value exceeds a preset competition saturation threshold, and coordinate execution sequences of computing tasks in a group based on the Lamport logical clock timestamp.

[0045] A third aspect of the embodiment of the application,

[0046] An electronic device is provided, comprising:

[0047] A processor;

[0048] A memory for storing processor-executable instructions;

[0049] The processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0050] A fourth aspect of the embodiment of the application,

[0051] A computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0052] The beneficial effects of the application are as follows:

[0053] The application realizes progressive flow scheduling of computing tasks and reasonable allocation of data by dividing the data stream into multiple data blocks to be processed, establishing hierarchical computing resource pools and storage resource pools, thereby improving the efficiency and throughput of data processing in chip design.

[0054] The distributed cache synchronization network is established by setting a shared cache area between adjacent parallel processing units, and the read-write state of cache data is synchronously controlled by using the Lamport logical clock timestamp mechanism and atomic lock operation, thereby solving the data consistency problem in the parallel computing process of multiple processing units and ensuring the accuracy of the computing result.

[0055] The application dynamically monitors the load state of the resource competition degree matrix, and when the competition load value exceeds the preset threshold, the task is rearranged, and through the optimization based on the access heat map and the coordination mechanism of the Lamport logical clock timestamp, the resource competition problem in the parallel processing process is effectively alleviated, and the stability and reliability of the data stream parallel processing in chip design are improved. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 The figure is a flowchart of the data stream parallel acceleration processing method in the chip design of the embodiment of the application.

[0057] Figure 2 The figure is a performance trajectory curve comparison diagram.

[0058] Figure 3 The figure is a Markov transition matrix and steady-state probability convergence diagram. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical scheme and advantages of the embodiment of the application clearer, the technical scheme in the embodiment of the application will be described clearly and completely below in combination with the drawings in the embodiment of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0060] The technical scheme of the application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in some embodiments.

[0061] Figure 1 The figure is a flowchart of the data stream parallel acceleration processing method in the chip design of the embodiment of the application, as shown in Figure 1 The method comprises:

[0062] The data stream is divided into multiple data blocks to be processed according to a preset scale.

[0063] establishing a computing resource pool and a storage resource pool, classifying computing units in the computing resource pool according to computing power, and organizing progressive flow water scheduling according to processing capacity rising rate; classifying the storage resource pool according to access speed, and realizing allocation of to-be-processed data blocks among different access speed storage levels and scheduling of computing tasks among different computing power computing units based on access acceleration of the to-be-processed data blocks and key computing paths; and dividing corresponding storage space for each computing unit in the corresponding storage layer;

[0064] dividing each computing unit into a plurality of parallel processing units, determining data processing order based on the time sequence dependency relationship, and distributing to-be-processed data blocks to corresponding parallel processing units according to the processing order;

[0065] establishing a distributed cache synchronization network by setting a shared cache area between adjacent parallel processing units, allocating a unique cache identifier for each shared cache area, establishing a cache access priority sequence using the cache identifier, and synchronously controlling read and write states of cache data based on the cache access priority sequence, a Lamport logical clock timestamp mechanism and an atomic lock operation;

[0066] generating a resource competition degree matrix between parallel processing units and calculating a competition load value, rearranging computing tasks based on an access heat map when the competition load value exceeds a preset competition saturation threshold, and coordinating execution order of computing tasks in a group based on the Lamport logical clock timestamp.

[0067] In an optional implementation, classifying computing units in the computing resource pool according to computing power and organizing progressive flow water scheduling according to processing capacity rising rate includes:

[0068] testing computing flux of a computing unit under different instruction types, and generating instruction execution characteristics of the computing unit;

[0069] based on the instruction execution characteristics, constructing an I / O performance cube of the computing unit, calculating a spatial vector of each computing unit in the I / O performance cube, and forming a performance trajectory curve with the spatial vector changing with load;

[0070] calculating Hausdorff distances between performance trajectory curves of different computing units, dividing computing units with a Hausdorff distance lower than a preset performance similarity threshold into the same performance level, and obtaining a plurality of performance computing unit groups;

[0071] based on a performance trajectory curve slope of a computing unit in each computing unit group, calculating a processing capacity rising rate, organizing computing units into a progressive flow water line according to the rising rate, and distributing a computing unit with the highest rising rate to the end of the progressive flow water line.

[0072] Real-time monitoring of the performance curve of each stage of the pipeline, when the performance jitter of a stage of the pipeline occurs, the first derivative of the performance curve is calculated, and the performance decay inflection point is calculated according to the derivative trend. Before reaching the decay inflection point, the task is migrated to the standby computing unit.

[0073] For each computing unit, a preset instruction set test program is executed, which contains different types of instructions such as integer operations, floating point operations, vector operations, and matrix operations. During the test, the execution time, throughput, delay, and other parameters of each type of instruction are recorded. For example, for a computing unit A, the integer operation throughput is measured to be 10000MIPS, the floating point operation throughput is measured to be 8500MFLOPS, and the vector operation throughput is measured to be 12000MVOPS. Through these test data, the instruction execution characteristic report of each computing unit can be generated, providing basic data for subsequent analysis.

[0074] The I / O performance cube is a three-dimensional space model, and the three dimensions represent input throughput, output throughput, and processing delay. The performance parameters of each computing unit are mapped into this three-dimensional space to form a space vector. For example, the space vector of computing unit B is (9500, 9000, 2.5), indicating that its input throughput is 9500MB / s, output throughput is 9000MB / s, and processing delay is 2.5ms. As the load increases, the performance parameters of the computing unit will change, and these changes form a performance trajectory curve in the I / O performance cube. By gradually increasing the load (from 10% to 100%, with a step size of 5%), the performance parameters at each load point are recorded to generate a complete performance trajectory curve. For example, the performance vector of computing unit C at a load of 50% is (7800, 7600, 3.2), and at a load of 80% it becomes (6200, 6000, 5.1).

[0075] The Hausdorff distance is an index for measuring the similarity between two curves. For two performance trajectory curves A and B, the Euclidean distance between their space vectors at each load point is calculated, and the maximum value is taken as their Hausdorff distance. For example, the Hausdorff distance between computing units D and E is 42.3, and the Hausdorff distance between D and F is 156.8. A performance similarity threshold is set in advance, such as 50. When the Hausdorff distance between two computing units is less than this threshold, they are considered to be similar in performance and can be divided into the same performance level. In this way, all computing units are divided into multiple performance computing unit groups. In actual application, a resource pool containing 100 computing units is divided into 5 performance level groups, each containing 10-25 computing units.

[0076] The slope of the performance trajectory curve reflects the speed of performance improvement of the computing unit as the load increases. The slope of each computing unit performance curve in different load intervals is calculated, and the average value is taken as the processing capacity rise rate. For example, the average rise rate of computing unit G is 0.85, and the average rise rate of computing unit H is 1.22. According to these rise rates, the computing units are organized into a progressive pipeline. The higher the processing capacity rise rate, the faster the performance of the computing unit improves as the load increases, so the computing unit with the highest rise rate is assigned to the end of the progressive pipeline. For example, for a pipeline composed of computing units G, H, and I, if their rise rates are 0.85, 1.22, and 0.93 respectively, they will be organized in the order of G→I→H. This arrangement ensures that the processing capacity gradually increases as the computing task flows through the pipeline, avoiding bottlenecks in subsequent stages.

[0077] The performance curve of each stage of the pipeline is monitored in real time to detect performance jitter and perform task migration. Performance jitter refers to a sudden drop in the performance of a computing unit, which is caused by hardware failure, resource competition, or other reasons. The first derivative of the performance curve is calculated to identify the trend of performance change. A positive to negative derivative indicates that performance is starting to decline, and an increasing absolute value of the derivative indicates that the decline is accelerating. The performance decay inflection point is calculated according to the derivative trend, which is the point in time when performance begins to decline sharply. For example, the first derivative of the performance curve of computing unit J changes from 0.02 to -0.05 at time point t = 157 seconds, and decreases to -0.25 at t = 163 seconds, and the performance decay inflection point is judged to be t = 160 seconds. When performance jitter is detected in a stage of the pipeline, the task will be migrated to a backup computing unit before the decay inflection point is reached (e.g. 3 seconds in advance). This predictive migration mechanism avoids migrating tasks only after performance has declined significantly, significantly improving stability and response speed.

[0078] Through the above method, the computing units in the computing resource pool can be accurately classified in terms of computing power, and an efficient progressive flow scheduling can be organized according to the processing capacity rise rate, while having the ability to predict performance jitter and migrate tasks, greatly improving the utilization efficiency of computing resources and the stability of the system.

[0079] Figure 2For the performance trajectory curve comparison diagram, the diagram shows the comprehensive performance index comparison of the progressive pipelining scheduling method proposed by the application and the traditional scheduling algorithm under different loads. As can be clearly observed from the diagram, the application divides the computing units into three performance levels through performance trajectory modeling and Hausdorff distance grading algorithm, and each level presents obvious hierarchy and stability in performance. The computing units of performance level 1 of the application always maintain the highest comprehensive performance index in the entire load range, which gradually decreases from 96.5 points at low load to 78.5 points at high load, but is still significantly better than the traditional method. The computing units of performance level 2 and level 3 also maintain good performance consistency, which embodies the effectiveness of the grading strategy of the application. In contrast, the performance of the traditional FIFO scheduling algorithm decreases sharply under high load, from 85.5 points to 46.2 points, and the performance of the traditional shortest job first algorithm decreases more severely, from 78.5 points to 35.0 points. The advantage of the application lies in that through accurate performance trajectory analysis and intelligent grading mechanism, even in the extreme case of load reaching 100%, the application can still maintain relatively stable performance output, effectively avoiding the performance collapse of the traditional scheduling method under high load, and proving the significant technical advantage of the progressive pipelining scheduling architecture in the distributed computing environment.

[0080] In an optional embodiment, the storage resource pool is classified according to access speed, and based on the access acceleration of the to-be-processed data block and the key computing path, the allocation of the to-be-processed data block among different access speed storage levels and the scheduling of the computing task among different computing power computing units are implemented, including:

[0081] The standard deviation of the adjacent access time of the to-be-processed data block is calculated to obtain a time fluctuation rate, and a periodic access feature vector of the to-be-processed data block is extracted according to the Fourier transform result of the time fluctuation rate;

[0082] The number of access transitions between the to-be-processed data blocks is counted to generate a Markov transition matrix, and the eigenvalue decomposition of the Markov transition matrix is performed to obtain a steady-state access probability and a convergence time of the to-be-processed data block;

[0083] Based on the periodic access feature vector, the steady-state access probability and the convergence time, the access mode feature of the to-be-processed data block is extracted through principal component analysis, the Euclidean distance of the access mode feature of the to-be-processed data block is calculated, and the to-be-processed data blocks with an Euclidean distance less than a dynamic clustering radius are aggregated to form a data block group;

[0084] The access delay and the number of accesses of the data block group in each storage layer are counted, the number of accesses is taken as the abscissa, the access delay is taken as the ordinate, a polynomial fitting is adopted to construct an access performance curve, a first derivative of the access performance curve is calculated to obtain an access acceleration, and the data block group is dynamically allocated to a corresponding storage layer based on an extreme point of the access acceleration.

[0085] The data block to be processed is set as a graph node, the dependency relationship between the data blocks is set as a directed edge, and the data flow conversion frequency is set as an edge weight to obtain an initial calculation graph. Strongly connected components in the initial calculation graph are extracted, a maximum flow minimum cut of the strongly connected components is calculated to obtain a key calculation path set, and a corresponding task subset is allocated to different computing units based on the key calculation path set.

[0086] Taking a data block as an example, assuming that its adjacent access time sequence is [2ms, 5ms, 3ms, 8ms, 2ms, 5ms, 3ms], the time fluctuation rate of the data block is 2.16ms by calculating the standard deviation of these values. Then, the time fluctuation rate is subjected to Fourier transform to extract periodic access features. After the transform, a frequency domain representation is obtained, for example, the main frequency components are [0.25, 0.5], and the corresponding periods are 4ms and 2ms, and the extracted periodic access feature vector is [4ms, 2ms, 0.25, 0.5], which contains the period values and corresponding weights.

[0087] Meanwhile, by monitoring the access transfer between data blocks, the number of access transfers between data blocks is counted. For example, for data blocks A, B and C, the number of times of access from A to B is 30 times, the number of times of access from A to C is 20 times, the number of times of access from B to A is 25 times, and so on. A Markov transition matrix is generated from these statistical data. For the above example, if the total number of accesses is 50 times for A, 40 times for B, and 35 times for C, then the data in one row of the transition matrix can be represented as: the probability of A to A is 0, the probability of A to B is 0.6, and the probability of A to C is 0.4. The eigenvalues [1, 0.8, 0.6] and the corresponding eigenvectors are obtained by eigenvalue decomposition of the transition matrix. The eigenvector corresponding to the eigenvalue 1 is normalized to [0.4, 0.35, 0.25] after normalization, which represents the steady-state access probability of data blocks A, B and C. The convergence time is calculated to be 10 iterations from the eigenvalue, which represents the number of access transfers required to reach the steady state.

[0088] Figure 3Fig. 4 is a schematic diagram of Markov transition matrix and steady-state probability convergence. The diagram shows the visualization results of the data block access pattern feature extraction and dynamic clustering algorithm based on principal component analysis proposed in the present application. The heat map adopts a 10x10 matrix form, and both the horizontal and vertical coordinates are data block numbers A to J. The color depth represents the Euclidean distance of the access pattern feature between any two data blocks. Dark blue represents a small distance (high similarity), and red represents a large distance (low similarity). The present application obtains the time volatility rate by calculating the standard deviation of the adjacent access time of the data block, extracts the periodic access feature vector by Fourier transform of the time volatility rate, generates the Markov transition matrix by counting the access transition times between data blocks, and obtains the steady-state access probability and convergence time by eigenvalue decomposition. The Euclidean distance is calculated after the features are processed by the principal component analysis method. From the diagram, it can be observed that the combinations of data blocks A and B, data blocks C and D, data blocks E and F, etc. present dark blue, indicating that their Euclidean distance is less than the dynamic clustering radius 2.0, and thus they are aggregated into the same data block group. Other data block pairs present lighter or red areas, indicating that the distance is large and should be allocated to different groups. The innovation of the present application lies in the combination of time domain and frequency domain analysis, Markov chain modeling, and principal component analysis, etc. mathematical methods, which realizes the accurate quantification and intelligent grouping of the data block access pattern, provides a scientific basis for subsequent storage level allocation and computing task scheduling, and significantly improves the resource configuration efficiency and access performance of the distributed storage system.

[0089] The periodic access feature vector includes the average access period and the access period standard deviation. The steady-state access probability includes the access probability in the high load, medium load, and low load states. The convergence time represents the time required to reach the steady-state access probability. For example, for data block A, the extracted original features are: average access period 4 ms, period standard deviation 2 ms, high load state access probability 0.25, medium load state access probability 0.5, low load state access probability 0.4, and convergence time 10 ms.

[0090] These features are combined into a feature vector. For data block A, the feature vector is [4, 2, 0.25, 0.5, 0.4, 10]. The feature vector is processed by the principal component analysis method: first, the feature data is standardized to eliminate the dimension effect; then the feature covariance matrix is calculated, and the eigenvalues and eigenvectors are solved; the first k principal components with a cumulative contribution rate of 85% are selected to obtain the reduced feature vector. For example, the reduced feature vector of data block A is [2.5, 1.8, 0.6], the reduced feature vector of data block B is [2.8, 1.5, 0.5], and the reduced feature vector of data block C is [4.2, 3.1, 1.2].

[0091] The Euclidean distance between data blocks A and B is 1.2, the Euclidean distance between A and C is 3.5, and the Euclidean distance between B and C is 2.8. The clustering radius is dynamically adjusted based on the current load state, and the current dynamic clustering radius is set to 2.0. Data blocks with a Euclidean distance less than the dynamic clustering radius are aggregated into a group: since the Euclidean distance between data blocks A and B is 1.2, which is less than the clustering radius 2.0, A and B are aggregated into one data block group; and C has a Euclidean distance greater than 2.0 from A and B, so C forms a data block group alone.

[0092] For data block group AB, the average access delay in the high-speed storage layer is 0.5 ms, the access frequency is 1000 times; the average access delay in the medium-speed storage layer is 2 ms, the access frequency is 500 times; and the average access delay in the low-speed storage layer is 10 ms, the access frequency is 100 times. Taking the access frequency as the horizontal coordinate and the access delay as the vertical coordinate, a polynomial fitting is used to construct the access performance curve, and the curve expression is obtained. The first derivative of the expression is calculated to obtain the access acceleration curve. The extreme points of the access acceleration curve are found, for example, extreme values occur at access frequencies of 300 times and 800 times, which divide the access frequency space into three segments: 0-300 times, 300-800 times, and more than 800 times. According to the actual access frequency of the data block group, it is dynamically allocated to the corresponding storage layer. For example, the access frequency of AB group is 1000 times, which is greater than 800 times, so it is allocated to the high-speed storage layer.

[0093] Data block A needs to be processed first before processing B, so a directed edge from A to B is added. The data flow frequency between data blocks is set as the edge weight, for example, the data flow frequency from A to B is 50 times per second, so the weight of this edge is 50. In this way, the initial computation graph is constructed.

[0094] Strongly connected components, i.e., sets of mutually dependent data blocks, are extracted from the initial computation graph. For example, if A depends on B, B depends on C, and C depends on A, then A, B, and C form a strongly connected component. For each strongly connected component, the maximum flow and minimum cut are calculated to find the critical computation path. The specific operation is to add a source point and a sink point, connect the source point to all nodes with an in-degree of 0, and connect all nodes with an out-degree of 0 to the sink point, and then use the maximum flow and minimum cut algorithm. For example, for a strongly connected component containing nodes A, B, C, and D, the minimum cut set calculated is edges {(A, B), (C, D)}, indicating that these two edges form the critical computation path.

[0095] Tasks on the critical path are assigned to high-performance computing units, and tasks on non-critical paths are assigned to ordinary computing units. In actual deployment, if there are four computing units A, B, C, and D with decreasing performance, the tasks A-B on the critical computing path can be assigned to computing unit A, the tasks C-D on the critical computing path can be assigned to computing unit B, and the remaining tasks can be assigned to computing units C and D.

[0096] Through the above method, intelligent storage layering and computing task scheduling based on data access patterns are realized, and the overall performance and resource utilization are effectively improved.

[0097] In an optional implementation, strong connected components in the initial computing graph are extracted, a maximum flow minimum cut of the strong connected components is calculated to obtain a critical computing path set, and a corresponding task subset is assigned to different computing units based on the critical computing path set, including:

[0098] A depth-first search is performed in the initial computing graph to obtain strong connected components, source points and sink points are determined based on edge weights in the strong connected components, an accessible path from the source points to the sink points is searched in a residual network of the strong connected components in an iterative manner, a maximum flow minimum cut is obtained when there is no accessible path from the source points to the sink points in the residual network, and an edge set corresponding to the maximum flow minimum cut is the critical computing path set;

[0099] The number of nodes included in each path in the critical computing path set is counted to obtain a path depth, the path depth, a sum of node weights on the path, and a sum of edge weights are weighted to obtain a path criticality, a ratio of a computing capability of each computing unit to a path computing demand is calculated, a matching degree of the path and the computing unit is generated in combination with a physical distance between the computing unit and the path node;

[0100] A task subset corresponding to the critical computing path is assigned to different computing units based on the path criticality and the matching degree.

[0101] Starting from any node in the graph, continuously deepening along the edges until it cannot continue, and then backtracking to the nearest node with unvisited neighbors to continue searching. By recording the discovery time and completion time of the nodes, and using an auxiliary stack structure, strong connected components can be effectively identified. In actual implementation, for a graph containing 100 nodes, the time complexity of this algorithm is O(node number + edge number).

[0102] After determining the strongly connected components, the source and sink nodes need to be determined in each strongly connected component. The source node is chosen as the node with the largest out-degree, and the sink node is chosen as the node with the largest in-degree. If there are multiple nodes with the same largest out-degree or in-degree, the node with the largest weight is chosen. For example, in a strongly connected component containing 10 nodes, if node A has an out-degree of 8 and a weight of 50, and node B has an out-degree of 6 and a weight of 70, node A is chosen as the source node.

[0103] A residual network is initialized with a capacity equal to the edge weights in the original graph. In each iteration, a breadth-first search is used to find an augmenting path from the source node to the sink node in the residual network. Along the augmenting path, the residual capacities of the edges on the path are decreased, and the capacities of the reverse edges are increased. When no more augmenting paths can be found, the algorithm terminates, and the edges between the set of nodes reachable from the source node and the set of nodes not reachable from the source node form the minimum cut, which is the set of critical computation paths.

[0104] In an example, a strongly connected component contains nodes {A, B, C, D, E} and edge set {(A, B, 5), (B, C, 3), (C, D, 4), (D, E, 2), (E, A, 3), (B, E, 6)}, where the third number in the parentheses represents the edge weight. Node A is chosen as the source node, and node D is chosen as the sink node. Through iterative computation, the maximum flow is 7, and the minimum cut edge set is {(B, C), (E, A)}, which form the set of critical computation paths.

[0105] The path depth is the number of nodes contained in the path. For example, the path A→B→C has a depth of 3. The path node weight sum is the cumulative value of the weights of all nodes on the path. If the weights of A, B, and C are 10, 15, and 20, respectively, the node weight sum is 45. The edge weight sum is calculated similarly. The path criticality is calculated using the weighted formula: criticality = 0.4 × path depth + 0.3 × node weight sum + 0.3 × edge weight sum. The weight coefficients can be adjusted according to specific application requirements.

[0106] The matching degree of the computing unit and the path takes into account two factors: the computing power and demand ratio and the physical distance. The computing power and demand ratio is defined as the processing speed of the computing unit divided by the total computation demand of the path. The physical distance is the average physical distance from the computing unit to the nodes of the path, which can be the actual distance or network delay. The matching degree calculation formula is: matching degree = 0.7 × power and demand ratio + 0.3 × (1 / normalized physical distance).

[0107] For example, assuming that there is a computing unit X with a processing speed of 1000 MIPS, and the computing requirement of path P is 500 MIPS, the capability requirement ratio is 2. The average physical distance of computing unit X to each node of path P is 5 ms, and the normalized distance is 0.5, so the matching degree is 0.7*2+0.3*(1 / 0.5)=1.4+0.6=2.0.

[0108] In the task allocation process, the critical computing paths are arranged in descending order of path criticality, and the computing units are arranged in descending order of matching degree. For the path with the highest criticality, the computing unit with the highest matching degree is preferentially allocated. If the remaining computing capacity of the computing unit is insufficient, the computing unit with the second highest matching degree is allocated. This cycle is repeated until all critical paths are allocated.

[0109] In an actual allocation case, assuming that there are three critical computing paths P1, P2 and P3 with criticalities of 10, 8 and 6 respectively, and three computing units X, Y and Z with matching degrees of 2.0, 1.8 and 1.5 respectively with P1. According to the algorithm, P1 is allocated to unit X, P2 is allocated to unit Y, and P3 is allocated to unit Z. If the computing capacity of unit X is insufficient to process all the tasks of P1, part of the tasks of P1 will be allocated to unit Y.

[0110] Through the above method, the application realizes efficient task allocation based on a computing graph, fully utilizes heterogeneous computing resources, and improves the overall performance of the system.

[0111] In an optional implementation, a distributed cache synchronization network is established by setting shared cache areas between adjacent parallel processing units, a unique cache identifier is allocated to each shared cache area, a cache access priority sequence is established using the cache identifier, and the read and write states of cache data are synchronously controlled based on the cache access priority sequence through a Lamport logical clock timestamp mechanism and an atomic lock operation, including:

[0112] The shared cache areas are connected to each other through cache data synchronization channels to form a distributed cache synchronization network, and a globally unique cache identifier is generated by multiplying the horizontal position index of the shared cache area in the distributed cache synchronization network by the column number and adding the vertical direction index position;

[0113] For each data item in each shared cache area, the current system clock time is obtained as an initial timestamp, and when a read or write operation occurs on the data, the initial timestamp of the data item is compared with the maximum timestamp of the adjacent shared cache area, and the larger value plus one is selected as the Lamport logical clock timestamp of the data item;

[0114] The cache identifier and the Lamport logical clock timestamp are weighted to generate a data access priority sequence.

[0115] According to the sequence of the data access priority from high to low, the atomic lock of each data item is acquired in turn to control the data access right, and when detecting that there is a read operation on the data item, the atomic lock blocks the subsequent write operation until all read operations are completed, and then the write operation is executed;

[0116] After completing the data read and write operation, the Lamport logical clock timestamp is updated, and the atomic lock is released to allow subsequent data access operations.

[0117] A plurality of shared cache regions are established between the plurality of parallel processing units, and the shared cache regions are connected to each other through cache data synchronization channels to form a distributed cache synchronization network. Each shared cache region has a specific horizontal position index and a vertical position index in the distributed network. For example, in a 4x4 array of processing units, each processing unit can have a unique coordinate position (x, y), where x represents the horizontal position index (0≤x≤3) and y represents the vertical position index (0≤y≤3).

[0118] In order to assign a globally unique cache identifier, the horizontal position index of the shared cache region in the distributed cache synchronization network is multiplied by the number of columns, and then added to the vertical direction index position. For example, for a processing unit at position (2, 1), if the total number of columns is 4, its cache identifier is calculated as 2x4+1=9. In this way, each shared cache region is assigned a unique identifier starting from 0. In the above example of a 4x4 array, the generated cache identifier ranges from 0 to 15.

[0119] Suppose a data item A has a system clock time of 1000 milliseconds at initialization, then this value is set as the initial timestamp of A. When a read or write operation occurs on data item A, the initial timestamp of this data item is compared with the maximum timestamp of the same data item in the adjacent shared cache region. For example, if the initial timestamp of data item A is 1000, and the timestamps of the same data item in the adjacent four cache regions are 1002, 998, 1005 and 997 respectively, then the maximum timestamp is 1005. Selecting the larger value 1005+1, we get 1006 as the new Lamport logical clock timestamp of data item A.

[0120] The Lamport logical clock timestamp is multiplied by a large weight factor, and then added to the cache identifier. For example, using a weight factor of 10000, for a data item with a cache identifier of 9 and a Lamport timestamp of 1006, the priority value is 1006x10000+9=10060009. This ensures that data items with larger timestamps have higher access priority, and in the case of the same timestamp, data items with larger cache identifiers have higher priority.

[0121] If the priorities of the three data items are 10060009, 10050012 and 10060007 respectively, the atomic lock of the data item with the priority of 10060009 is firstly tried to be acquired, then the data item with the priority of 10060007, and finally the data item with the priority of 10050012.

[0122] A read counter is maintained, and the initial value is 0. Each time a new read operation request arrives, the read counter is incremented by 1; and after the read operation is completed, the read counter is decremented by 1. When the read counter is greater than 0, all write operation requests are blocked. For example, if the read counter of data item B is 3 at a certain time, it indicates that there are 3 concurrent read operations being performed, and if a write operation request arrives at this time, the request will be blocked until the read counter becomes 0.

[0123] After all read operations are completed, the write operation is executed. The write operation adopts a mutual exclusion lock mechanism to ensure that only one write operation can access the data item at the same time. For example, when data item C needs to be updated, the mutual exclusion lock of C is firstly acquired, the write operation is executed, the data value is updated, and then the mutual exclusion lock is released.

[0124] For a read operation, the timestamp is incremented by 1; and for a write operation, the timestamp is incremented by 2. For example, if the current timestamp of data item D is 1020, after a read operation is completed, the timestamp is updated to 1021; and if a write operation is completed, the timestamp is updated to 1022. The update of the timestamp ensures the causal consistency of the operations, and the timestamp increment of the write operation is greater than that of the read operation, which reflects the greater impact of the write operation on the system state.

[0125] Finally, the atomic lock is released to allow subsequent data access operations. After all operations of a specific data item are completed, the latest timestamp of the data item is broadcast to the adjacent shared cache area through the cache data synchronization channel, to ensure the data consistency in the entire distributed cache synchronization network. After the adjacent cache area receives the updated timestamp, the timestamp information of the corresponding data item in the respective cache is updated, to provide a basis for subsequent data access decisions.

[0126] Through the above mechanism, the present application realizes the cache data synchronization control in a distributed environment, ensures the data consistency and efficient access, and solves the problem of data synchronization between parallel processing units.

[0127] In an optional implementation, a resource contention degree matrix between parallel processing units is generated, and a contention load value is calculated, when the contention load value exceeds a preset contention saturation threshold, the computing tasks are rearranged based on an access heat map, and the execution order of the computing tasks in a group is coordinated based on the Lamport logical clock timestamp, which includes:

[0128] counting the number of cache access overlaps between each pair of parallel processing units to generate an access conflict matrix; calculating the Euclidean distance between the feature vectors of each pair of parallel processing units to generate a performance difference matrix; and combining the access conflict matrix and the performance difference matrix by weighting to generate a resource contention degree matrix, and calculating the contention load value of each parallel processing unit through the resource contention degree matrix;

[0129] setting an access counter in the shared cache region to record the number of times each data block is accessed by different parallel processing units within a unit time window and the time interval, calculating the heat value of the data block according to the product of the number of accesses and the time interval, and arranging the heat values in order of the physical addresses of the data blocks to generate a heat map reflecting the data access pattern;

[0130] when the contention load value of any parallel processing unit exceeds a preset contention saturation threshold, identifying a data block group with a heat value difference less than a preset heat difference threshold and a continuous address from the heat map, marking a computing task accessing these data block groups as a related task with data locality, and reassigning the related task to the same computing unit group for execution;

[0131] assigning a Lamport logical clock timestamp to each computing task in the computing unit group in order of the heat values of the data blocks accessed by the computing task, determining the execution priority of the tasks in the group based on the timestamp, and executing the task with the smallest timestamp first.

[0132] First, a resource contention degree matrix between parallel processing units needs to be generated. This process is achieved by constructing an access conflict matrix and a performance difference matrix, and then combining the two by weighting. Specifically, the number of cache access overlaps between each pair of parallel processing units is counted. For example, for a system with 4 processing units (PU0, PU1, PU2, PU3), the cache address sequence accessed by each processing unit is recorded within an observation window, such as PU0 accessing address sequence [0x1000, 0x2000, 0x3000] and PU1 accessing address sequence [0x2000, 0x4000, 0x5000]. By comparing these sequences, it can be determined that PU0 and PU1 have 1 access overlap (address 0x2000). Fill the number of overlaps of all processing unit pairs into the matrix to obtain the access conflict matrix. In this example, the following access conflict matrix is formed: PU0-PU1: 1 time, PU0-PU2: 2 times, PU0-PU3: 0 times, PU1-PU2: 1 time, PU1-PU3: 2 times, PU2-PU3: 3 times.

[0133] Simultaneously, the Euclidean distance between the performance feature vectors of each pair of parallel processing units is calculated. The feature vectors contain parameters such as the processing unit's computing power, cache size, and memory access latency. For example, the feature vector of PU0 is [3.2GHz, 256KB, 20ns], the feature vector of PU1 is [3.2GHz, 256KB, 20ns], and the feature vector of PU2 is [2.8GHz, 128KB, 35ns]. By calculating the Euclidean distance between these vectors, a performance difference matrix is ​​obtained. In the example above, the performance difference between PU0 and PU1 is 0, while the performance difference between PU0 and PU2 is relatively large, approximately 140.1.

[0134] The access conflict matrix and performance difference matrix are weighted and combined to generate a resource contention matrix. The weighting coefficients are adjusted according to the actual application scenario, such as an access conflict weight of 0.7 and a performance difference weight of 0.3. This combination allows for a more comprehensive assessment of resource contention between processing units. For example, although PU0 and PU1 have 1 access conflict and 0 performance difference, their overall contention is 0.7×1 + 0.3×0 = 0.7; while PU2 and PU3 have 3 access conflicts and, assuming a performance difference of 50, their overall contention is 0.7×3 + 0.3×50 = 16.1.

[0135] The contention load value is equal to the sum of the contention levels between this processing unit and all other processing units. In the example above, the contention load value of PU0 is 0.7 + 2.8 + 0 = 3.5; the contention load value of PU1 is 0.7 + 1.4 + 9.5 = 11.6; the contention load value of PU2 is 2.8 + 1.4 + 16.1 = 20.3; and the contention load value of PU3 is 0 + 9.5 + 16.1 = 25.6.

[0136] Data block 0x1000 is accessed 10 times by PU0 within a 100ms time window, with an average time interval of 10ms; it is accessed 5 times by PU1, with an average time interval of 20ms. The heat value of the data block is calculated by multiplying the number of accesses by the time interval. In this example, the heat value of data block 0x1000 is 10×10 + 5×20 = 200. The heat values ​​of all data blocks are arranged in order of their physical addresses to form a heat map. For example, data block 0x1000 has a heat value of 200, 0x1100 has a heat value of 150, 0x1200 has a heat value of 160, 0x1300 has a heat value of 155, and 0x2000 has a heat value of 300, etc.

[0137] When the competition load value of any parallel processing unit exceeds a preset competition saturation threshold (e.g., set to 20), a task rearrangement mechanism is triggered. According to the heat map, a data block group with a heat value difference less than a preset heat difference threshold (e.g., set to 10) and continuous addresses is identified. For example, the heat values of data blocks 0x1100 to 0x1300 are 150, 160, and 155, respectively, and their differences are all less than 10, and the addresses are continuous, so they are identified as a data block group. The computing tasks that access these data block groups are marked as related tasks with data locality.

[0138] The tasks T1, T2, and T3 that access the data blocks 0x1100 to 0x1300 originally dispersed on the PUs 1 and 2 are reassigned to the computing unit group CG1 (including the PUs 0 and 1) for execution. This reassignment reduces the cache contention between different processing units and improves the cache hit rate.

[0139] To coordinate the execution order of tasks within the computing unit group, a Lamport logical clock timestamp is assigned to each computing task according to the size order of the heat values of the data blocks accessed by the task. The larger the heat value, the smaller the timestamp assigned. In the above example, task T1 accesses data block 0x1100 (heat value 150), task T2 accesses data block 0x1200 (heat value 160), and task T3 accesses data block 0x1300 (heat value 155), so T2 obtains the smallest timestamp 1, T3 obtains timestamp 2, and T1 obtains timestamp 3. According to these timestamps, the execution priority of the tasks is determined, and T2 is executed first, then T3, and finally T1.

[0140] Through this task scheduling method based on the resource competition degree matrix and the access heat map, the resource competition between parallel processing units can be effectively reduced, the cache utilization efficiency can be improved, and thus the overall computing performance can be significantly improved. In actual tests, compared with traditional scheduling methods, this method can reduce cache conflicts by about 35% and increase throughput by about 25%.

[0141] The data flow parallel acceleration processing system in the chip design of the embodiment of the application comprises:

[0142] A first unit is configured to divide the data flow into a plurality of to-be-processed data blocks according to a preset scale.

[0143] The second unit is configured to establish a computing resource pool and a storage resource pool, perform computing power classification on computing units in the computing resource pool, and organize progressive flow scheduling according to processing capacity rising rates; the storage resource pool is classified according to access speeds, and distribution of to-be-processed data blocks among different access speed storage levels and scheduling of computing tasks among different computing power computing units are implemented based on access accelerations of the to-be-processed data blocks and key computing paths; a corresponding storage space is divided for each computing unit in a corresponding storage layer;

[0144] The third unit is configured to divide each computing unit into a plurality of parallel processing units, determine a data processing sequence based on the time sequence dependency relationship, and distribute to-be-processed data blocks into corresponding parallel processing units according to the processing sequence.

[0145] The fourth unit is configured to establish a distributed cache synchronization network by setting a shared cache area between adjacent parallel processing units, allocate a unique cache identifier for each shared cache area, establish a cache access priority sequence using the cache identifier, and perform synchronization control on read-write states of cache data based on the cache access priority sequence, a Lamport logical clock timestamp mechanism, and an atomic lock operation.

[0146] The fifth unit is configured to generate a resource competition degree matrix between parallel processing units and calculate a competition load value, rearrange computing tasks based on an access heat map when the competition load value exceeds a preset competition saturation threshold, and coordinate execution sequences of computing tasks in a group based on the Lamport logical clock timestamp.

[0147] In a third aspect of the embodiments of the present application, an electronic device is provided, comprising:

[0148] a processor;

[0149] a memory for storing processor-executable instructions;

[0150] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0151] In a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0152] The present application can be a method, device, system and / or computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions loaded thereon for performing various aspects of the present application.

[0153] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for data stream parallel acceleration processing in chip design, characterized in that, The method comprises the following steps: dividing the data stream into a plurality of data blocks to be processed according to a preset scale; establishing a computing resource pool and a storage resource pool, classifying computing units in the computing resource pool according to computing power, organizing progressive flow water scheduling according to the rising rate of processing capacity, classifying the storage resource pool according to access speed, and based on the access acceleration of the data block to be processed and the key computing path, realizing the distribution of the data block to be processed among different access speed storage levels and the scheduling of the computing task among different computing power computing units; and dividing the corresponding storage space for each computing unit in the corresponding storage layer; dividing each computing unit into a plurality of parallel processing units, determining the data processing order based on the time sequence dependency relationship, and distributing the data block to be processed to the corresponding parallel processing unit according to the processing order; by setting a shared cache area between adjacent parallel processing units to establish a distributed cache synchronization network, assigning a unique cache identifier to each shared cache area, establishing a cache access priority sequence using the cache identifier, and synchronously controlling the read-write state of the cache data through the Lamport logical clock timestamp mechanism and atomic lock operation based on the cache access priority sequence; generating a resource competition degree matrix between parallel processing units and calculating a competition load value, when the competition load value exceeds a preset competition saturation threshold, rearranging the computing task based on the access heat map, and coordinating the execution order of the computing task in the group based on the Lamport logical clock timestamp.

2. The method of claim 1, wherein, classifying computing units in the computing resource pool according to computing power, organizing progressive flow water scheduling according to the rising rate of processing capacity, which comprises: testing the computing flux of the computing unit under different instruction types to generate the instruction execution characteristics of the computing unit; based on the instruction execution characteristics, constructing an I / O performance cube of the computing unit, calculating the spatial vector of each computing unit in the I / O performance cube, and the spatial vector forms a performance trajectory curve with the change of load; calculating the Hausdorff distance between the performance trajectory curves of different computing units, dividing the computing units with a Hausdorff distance lower than a preset performance similarity threshold into the same performance level to obtain a plurality of performance computing unit groups; based on the slope of the performance trajectory curve of the computing unit, calculating the rising rate of processing capacity in each computing unit group, and organizing the computing units into a progressive flow pipeline according to the rising rate, and assigning the computing unit with the highest rising rate to the end of the progressive flow pipeline; real-time monitoring the performance curve of each level of the flow pipeline, when the performance of a certain level of the flow pipeline fluctuates, calculating the first derivative of the performance curve, calculating the performance decay inflection point according to the derivative trend, and migrating the task to a standby computing unit before reaching the decay inflection point.

3. The method of claim 1, wherein, classifying the storage resource pool according to access speed, based on the access acceleration of the data block to be processed and the key computing path, realizing the distribution of the data block to be processed among different access speed storage levels and the scheduling of the computing task among different computing power computing units, which comprises: calculating the standard deviation of the adjacent access time of the data block to be processed to obtain the time fluctuation rate, and extracting the periodic access feature vector of the data block to be processed according to the Fourier transform result of the time fluctuation rate; The number of access transitions between the to-be-processed data blocks is counted to generate a Markov transition matrix, and eigenvalue decomposition is performed on the Markov transition matrix to obtain a steady-state access probability and a convergence time of the to-be-processed data blocks; Based on the periodic access eigenvector, the steady-state access probability and the convergence time, an access mode feature of the to-be-processed data blocks is extracted through principal component analysis, the Euclidean distance of the access mode feature of the to-be-processed data blocks is calculated, and to-be-processed data blocks with a Euclidean distance less than a dynamic clustering radius are aggregated to form a data block group; The access delay and the number of accesses of the data block group in each storage layer are counted, the number of accesses is used as the abscissa, the access delay is used as the ordinate, a polynomial fitting is used to construct an access performance curve, a first derivative of the access performance curve is calculated to obtain an access acceleration, and the data block group is dynamically allocated to a corresponding storage layer based on an extreme point of the access acceleration; The to-be-processed data blocks are set as graph nodes, the dependency relationship between the data blocks is set as a directed edge, and the data flow transition frequency is set as an edge weight to obtain an initial computation graph, a strongly connected component in the initial computation graph is extracted, a maximum flow minimum cut of the strongly connected component is calculated to obtain a set of critical computation paths, and a corresponding task subset is allocated to different computing units based on the set of critical computation paths.

4. The method of claim 3, wherein, The strongly connected component in the initial computation graph is extracted, the maximum flow minimum cut of the strongly connected component is calculated to obtain a set of critical computation paths, and a corresponding task subset is allocated to different computing units based on the set of critical computation paths, including: Depth-first search is performed in the initial computation graph to obtain a strongly connected component, a source point and a sink point are determined based on the edge weight in the strongly connected component, an accessible path from the source point to the sink point is searched in a residual network of the strongly connected component in an iterative manner, a maximum flow minimum cut is obtained when there is no accessible path from the source point to the sink point in the residual network, and an edge set corresponding to the maximum flow minimum cut is the set of critical computation paths; The number of nodes included in each path in the set of critical computation paths is counted to obtain a path depth, and the path depth, the sum of node weights on the path, and the sum of edge weights are weighted to obtain a path criticality; the ratio of the computing capacity of each computing unit to the path computing demand is calculated, and the matching degree of the path and the computing unit is generated in combination with the physical distance between the computing unit and the path node; The task subset corresponding to the critical computation path is allocated to different computing units based on the path criticality and the matching degree.

5. The method of claim 1, wherein, A distributed cache synchronization network is established by setting a shared cache area between adjacent parallel processing units, a unique cache identifier is allocated to each shared cache area, a cache access priority sequence is established using the cache identifier, and the read-write state of cache data is synchronously controlled through a Lamport logical clock timestamp mechanism and an atomic lock operation based on the cache access priority sequence, including: The shared cache areas are connected to each other through cache data synchronization channels to form a distributed cache synchronization network, and a global unique cache identifier is generated according to the horizontal position index of the shared cache area in the distributed cache synchronization network multiplied by the number of columns and added to the vertical direction index position; For each data item in each shared cache area, an initial timestamp is obtained as the current system clock time, and when a read or write operation occurs on the data, the initial timestamp of the data item is compared with the maximum timestamp of the adjacent shared cache area, and the larger value plus one is selected as the Lamport logical clock timestamp of the data item; The cache identifier and the Lamport logical clock timestamp are weighted and calculated to generate a data access priority sequence; According to the data access priority sequence from high to low, the atomic lock of each data item is obtained in sequence to control the data access permission, and when it is detected that there is a read operation on the data item, the atomic lock blocks the subsequent write operation until all read operations are completed, and then the write operation is performed; After completing the data read and write operation, the Lamport logical clock timestamp is updated, and the atomic lock is released to allow subsequent data access operation.

6. The method of claim 1, wherein, A resource competition degree matrix between parallel processing units is generated, and a competition load value is calculated, and when the competition load value exceeds a preset competition saturation threshold, the computing tasks are rearranged based on an access heat map, and the execution order of the computing tasks in the group is coordinated based on the Lamport logical clock timestamp, including: The number of cache access overlaps between each pair of parallel processing units is counted to generate an access conflict matrix, the Euclidean distance between the feature vectors of each pair of parallel processing units is calculated to generate a performance difference matrix, the access conflict matrix and the performance difference matrix are weighted and combined to generate a resource competition degree matrix, and the competition load value of each parallel processing unit is calculated through the resource competition degree matrix; An access counter is arranged in the shared cache area to record the number of times and time intervals that each data block is accessed by different parallel processing units within a unit time window, the heat value of the data block is calculated according to the product of the number of accesses and the time interval, and the heat value is arranged in the order of the physical address of the data block to generate a heat map reflecting the data access mode; When the competition load value of any parallel processing unit exceeds the preset competition saturation threshold, a data block group with a heat value difference less than a preset heat difference threshold and a continuous address is identified from the heat map, the computing tasks accessing these data block groups are marked as related tasks with data locality, and the related tasks are reassigned to the same computing unit group for execution; Each computing task in the computing unit group is assigned a Lamport logical clock timestamp in the order of the heat value of the accessed data block, the execution priority of the task in the group is determined based on the timestamp, and the task with the smallest timestamp is executed preferentially.

7. A dataflow parallel acceleration processing system in a chip design for implementing the method of any one of claims 1-6, characterized by The method comprises the following steps: A first unit is configured to divide a data stream into a plurality of to-be-processed data blocks according to a preset scale; The second unit is configured to establish a computing resource pool and a storage resource pool, perform computing power classification on computing units in the computing resource pool, and organize progressive flow scheduling according to processing capacity rising rates; the storage resource pool is classified according to access speeds, and based on access accelerations of to-be-processed data blocks and key computing paths, distribution of the to-be-processed data blocks among different access speed storage levels and scheduling of computing tasks among different computing power computing units are implemented; a corresponding storage space is divided for each computing unit in a corresponding storage layer; The third unit is configured to divide each computing unit into a plurality of parallel processing units, determine a data processing order based on the time sequence dependency relationship, and distribute to-be-processed data blocks into corresponding parallel processing units according to the processing order; The fourth unit is configured to establish a distributed cache synchronization network by setting a shared cache area between adjacent parallel processing units, allocate a unique cache identifier for each shared cache area, establish a cache access priority sequence using the cache identifier, and perform synchronization control on read and write states of cache data based on the cache access priority sequence through a Lamport logical clock timestamp mechanism and an atomic lock operation; The fifth unit is configured to generate a resource competition degree matrix between parallel processing units and calculate a competition load value, rearrange computing tasks based on an access heat map when the competition load value exceeds a preset competition saturation threshold, and coordinate execution orders of computing tasks in a group based on the Lamport logical clock timestamp.

8. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method of any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by the processor, implement the method of any one of claims 1 to 6.

Images