Data processing method, system and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method, system and device, electronic equipment and a storage medium, relates to the technical field of computers, and aims to at least solve the problems of operation resource waste, low operation efficiency, poor real-time performance, low operation precision, poor adaptability and the like in related technologies. The data processing method comprises the following steps: determining a dynamic threshold value corresponding to current initial data; dividing the current initial data into at least one current data subset; and for each current data subset, determining a non-zero element set of valid data of the current data subset based on a dynamic threshold, the valid data of the current data subset further comprising an index information set, and the index information set is used for representing index information of each non-zero element in the non-zero element set in the current data subset.

Description

Technical Field

[0001] The present disclosure relates to, but is not limited to, the field of computer technology, and in particular to a data processing method, system and device, electronic device, and storage medium. Background Art

[0002] In related technologies, in scenarios such as deep learning, graph computing, and scientific computing, the default approach is to perform full operations on all data (e.g., dynamic data, highly sparse tensors, etc.). However, some special values ​​in the data (e.g., zero values, redundant values, invalid values, etc.) do not contribute to the computational results, leading to problems such as wasted computing resources, low computational efficiency, and poor real-time performance. Alternatively, some methods initially employ fixed thresholds or static rules to identify and filter special values. This not only makes it difficult to adapt to varying data distributions and task requirements, but also easily leads to over- or under-filtering, resulting in subsequent problems such as reduced computational accuracy and wasted resources. Summary of the Invention

[0003] Embodiments of the present disclosure provide a data processing method, system and device, electronic device, storage medium, and computer program product.

[0004] The technical solution of the embodiment of the present disclosure is implemented as follows: The present disclosure provides a data processing method, including: Determine the dynamic threshold corresponding to the current initial data; Dividing the current initial data into at least one current data subset; For each current data subset, based on the dynamic threshold, the non-zero element set of the valid data of the current data subset is determined. The valid data of the current data subset also includes the non-zero element set. The index information set is used to characterize the index information of each non-zero element in the non-zero element set in the current data subset.

[0005] An embodiment of the present disclosure provides a data processing system, including a first data processing unit, wherein: The first data processing unit is used to: determine a dynamic threshold corresponding to the current initial data; divide the current initial data into at least one current data subset; and for each current data subset, determine a non-zero element set of valid data of the current data subset based on the dynamic threshold, wherein the valid data of the current data subset also includes an index information set, and the index information set is used to represent the index information of each non-zero element in the non-zero element set in the current data subset.

[0006] An embodiment of the present disclosure provides a data processing device, including: A first determination module is used to determine a dynamic threshold corresponding to the current initial data; A partitioning module, configured to partition the current initial data into at least one current data subset; The second determination module is used to determine, for each current data subset, a non-zero element set of valid data of the current data subset based on a dynamic threshold, where the valid data of the current data subset also includes an index information set, which is used to characterize the index information of each non-zero element in the non-zero element set in the current data subset.

[0007] An embodiment of the present disclosure provides an electronic device, including a processor and a memory, wherein the memory stores a computer program that can be run on the processor, and the above method is implemented when the processor executes the computer program.

[0008] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above method is implemented.

[0009] An embodiment of the present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, the above method is implemented.

[0010] In the embodiment of the present disclosure, on the one hand, the effective data of the data subset is determined by a dynamic threshold determined in real time, which reduces the possibility of insufficient filtering or excessive filtering compared to the use of fixed thresholds or static rules for identification and filtering in related technologies, thereby not only improving the subsequent calculation accuracy but also improving resource utilization, and being able to adapt to changes in different data distributions and task requirements, improving the compatibility and adaptability of data processing. At the same time, compared to the full amount of calculation in related technologies, since the present disclosure only calculates on the effective data, it not only improves the utilization of calculation resources but also improves the calculation efficiency, and also improves the real-time performance of data calculation; on the other hand, the effective data of the data subset only records each Compared with recording the global index of each non-zero value in the initial data, the local index of the non-zero value in the data subset, first, because each data subset only maintains the internal index, the index maintenance cost is reduced; second, in the subsequent search, since the search is only performed in the local range, the search is simplified while also reducing the difficulty and complexity of the search; finally, in the subsequent memory access, since the access can be performed only in the local memory, unnecessary global memory access is reduced, and the locality of data access is improved while also reducing storage and computing costs; on the other hand, since the data processing method can be used to process image data, sparse tensors, time series data, etc., it has good versatility and expands the application scope of the data processing method.

[0011] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0013] Figure 1 A schematic diagram of the implementation process of a data processing method provided in the embodiment of the present disclosure Figure 1 ; Figure 2 A schematic diagram of the implementation process of a data processing method provided in the embodiment of the present disclosure Figure 2 ; Figure 3 A schematic diagram of the implementation process of a data processing method provided in the embodiment of the present disclosure Figure 3 ; Figure 4 A schematic diagram of the implementation process of a data processing method provided in the embodiment of the present disclosure Figure 4 ; Figure 5 A schematic diagram of the structure of a data processing system provided in an embodiment of the present disclosure Figure 1 ; Figure 6 A schematic diagram of the structure of a data processing system provided in an embodiment of the present disclosure Figure 2 ; Figure 7 A schematic diagram of the structure of a data processing system provided in an embodiment of the present disclosure Figure 3 ; Figure 8 A schematic diagram of the implementation process of a data processing method provided in the embodiment of the present disclosure Figure 5 ; Figure 9 A schematic diagram of the structure of a data processing device provided in an embodiment of the present disclosure; Figure 10 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting the present disclosure. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0015] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0016] In the following description, the terms "first\second\third" are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art in the art of the present disclosure. The terms used herein are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0018] In related technologies, GPUs (Graphics Processing Units) are widely used to accelerate large-scale data operations, such as matrix operations and convolution operations.

[0019] In scenarios such as deep learning, graph computing, and scientific computing, GPUs typically use highly parallel threads to perform full operations on data (e.g., dynamic data, highly sparse tensors, etc.). However, some special values ​​in the data (e.g., zero values, redundant values, invalid values, etc.) do not contribute to the operation results. For example, in the CUDA (Compute Unified Device Architecture, a parallel computing platform and programming model) programming framework, operations such as matrix multiplication and convolution are performed using full data operations, ignoring the existence of zero values. Although the result of zero value operations is still zero, it consumes a large amount of computing resources and storage bandwidth, resulting in problems such as wasted computing resources, low computing efficiency, and poor real-time performance. Alternatively, some methods will first use fixed thresholds, static rules, etc. to identify and filter special values. However, the use of fixed thresholds or static rules is difficult to cope with changes in different data distributions and task requirements, and is prone to the possibility of misfiltering, missed filtering, over-filtering or insufficient filtering. At the same time, when facing noise or changes in data distribution, it lacks flexibility and is difficult to dynamically adjust according to real-time data, which in turn affects subsequent computing efficiency and accuracy, reduces computing accuracy, and brings about problems such as waste of resources.

[0020] The embodiment of the present disclosure provides a data processing method. On the one hand, the valid data of a data subset is determined by a dynamic threshold determined in real time. Compared with the related art that adopts fixed thresholds or static rules for identification and filtering, the possibility of insufficient filtering or excessive filtering is reduced, thereby not only improving the subsequent calculation accuracy but also improving resource utilization, and being able to adapt to changes in different data distributions and task requirements, thereby improving the compatibility and adaptability of data processing. At the same time, compared with the related art that performs full calculations, since the present disclosure only performs calculations on valid data, not only improving the utilization of calculation resources but also improving calculation efficiency, but also improving the real-time performance of data calculations; on the other hand, the valid data of the data subset is only Compared with recording the global index of each non-zero value in the initial data, recording the local index of each non-zero value in the data subset, first, because each data subset only maintains the internal index, the index maintenance cost is reduced; second, in subsequent searches, since the search is only performed in a local range, the search is simplified while also reducing the difficulty and complexity of the search; finally, in subsequent memory accesses, since the access can be performed only in the local memory, unnecessary global memory access is reduced, and the locality of data access is improved while also reducing storage and computing costs; on the other hand, since the data processing method can process image data, sparse tensors, time series data, etc., it has good versatility and expands the application scope of the data processing method.

[0021] The methods provided in the embodiments of the present disclosure can be performed by electronic devices, which can be various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), etc., or can be implemented as servers. The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0022] Below, the technical solutions in the embodiments of the present disclosure will be clearly and completely described in conjunction with the drawings in the embodiments of the present disclosure.

[0023] Figure 1 A schematic diagram of the implementation process of a data processing method provided in the embodiment of the present disclosure Figure 1 ,like Figure 1 As shown, the detection method includes steps S11 to S13, wherein: Step S11: Determine the dynamic threshold corresponding to the current initial data.

[0024] Here, the initial data (including the current initial data and other initial data mentioned below) can be any suitable data. For example, image data, sparse tensors, time series data, etc. The initial data may contain special values, such as zero values, redundant values, invalid values, etc. In some embodiments, in multi-round iterative scenarios such as weight updates and optimization operations in deep learning, the current initial data can be a sparse tensor in the current iteration.

[0025] The dynamic threshold value can be any suitable value and is used to identify and filter special values ​​in the initial data to obtain valid data of the initial data. Special values ​​are not included in the valid data of the initial data. In some embodiments, different initial data can correspond to the same or different dynamic threshold values.

[0026] The dynamic threshold may be determined in any appropriate manner.

[0027] In some implementations, a correspondence between various initial data and various thresholds may be pre-established, and based on the correspondence, the dynamic threshold corresponding to the current initial data may be obtained.

[0028] In some embodiments, the dynamic threshold corresponding to the current initial data can be determined by a neural network model. The neural network model can be any suitable model capable of implementing this function. For example, a large model, a visual language model, a MobileNet based on depthwise separable convolution, a TinyMLP optimized using pruning and quantization techniques, etc. During implementation, the current initial data is input into the neural network model to obtain the dynamic threshold.

[0029] In some embodiments, the dynamic threshold can be determined according to the features corresponding to the current initial data through a mapping function or a neural network model. The mapping function or neural network model can be any suitable function or model that can achieve this function. For example, the dynamic threshold is generated by a fully connected layer. The features corresponding to the current initial data may include but are not limited to the features of the current initial data, the features of the first data obtained after preprocessing the current initial data, etc. Preprocessing may include but is not limited to at least one of outlier processing, standardization processing, and normalization processing. Outlier processing refers to removing outliers from the data. Standardization processing refers to converting each data into a dimensionless value without order of magnitude difference to ensure consistency and comparability. Normalization processing refers to mapping the data feature values ​​to a specific interval so that different data are in the same scale range for easy comparison and subsequent processing.

[0030] In some embodiments, data features may include, but are not limited to, global features of the data, local features of the data, and the like. During implementation, data features may be extracted using any suitable feature extraction method, such as histograms, principal component analysis, CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), and Transformer converters.

[0031] In some implementations, the dynamic threshold corresponding to the current initial data may be determined by the following formula (1-1), namely: (1-1); in, Refers to the dynamic threshold corresponding to the initial data, represents the mapping function of the neural network, Indicates the features corresponding to the initial data.

[0032] In some embodiments, the dynamic threshold can be determined based on the threshold adjustment amplitude determined by the trained agent and the dynamic threshold corresponding to the initial data of the target round iteration. The target round iteration can be the iteration before the current iteration, for example, the previous iteration, the first iteration, etc. The method for determining the dynamic threshold can include, but is not limited to, a first sum value, a weighted first sum value, etc. The first sum value refers to the sum of the threshold adjustment amplitude and the dynamic threshold corresponding to the initial data of the target round iteration. For example, the first sum value is used as the dynamic threshold corresponding to the current initial data.

[0033] In some embodiments, the following formula (1-2) can be used to determine the Dynamic threshold corresponding to the initial data in the round iteration ,Right now: (1-2); in, It refers to the The dynamic threshold corresponding to the initial data in the round iteration, Indicates in The threshold adjustment amount determined by the trained agent in the round iteration.

[0034] Step S12: Divide the current initial data into at least one current data subset.

[0035] Here, the current initial data includes at least one row and at least one column. For example, 8 rows and 16 columns, 12 rows and 8 columns, etc. In some embodiments, the current initial data can be divided according to a pre-set division strategy to obtain various current data subsets. The division strategy may include but is not limited to a division step size, a division method, etc. The division step size can be any suitable step size, for example, 2 rows and 2 columns, 3 rows and 1 column, etc. During implementation, the row and column values ​​in the division step size can be the same or different. The division method may include but is not limited to uniform division, uneven division, custom division, random division, etc. For example, the current initial data is divided according to 3 rows and 3 columns. For another example, the current initial data is evenly divided.

[0036] The current data subset includes at least one row and at least one column. For example, 2 rows and 2 columns, 3 rows and 4 columns, etc. The sizes of the current data subsets can be the same or different. In implementation, the number of data subsets is determined based on the size of the current data subset and the partitioning strategy. For example, assuming the initial tensor , is the number of rows, is the number of columns, Divide into sizes The data subset has a total of A subset of data.

[0037] In some embodiments, if the size of a data subset is smaller than the partition step size, a padding operation may be performed on the data subset to make the size of the data subset equal to the partition step size. For example, if the partition step size is 3 rows and 2 columns, and the size of a data subset is 2 rows and 2 columns, a zero padding operation may be performed on the data subset to make the size of the data subset 3 rows and 2 columns.

[0038] Step S13: For each current data subset, based on the dynamic threshold, determine the non-zero element set of valid data of the current data subset, the valid data of the current data subset also includes an index information set, and the index information set is used to represent the index information of each non-zero element in the non-zero element set in the current data subset.

[0039] Here, the set of non-zero elements The index information set may include but is not limited to a row index information set. , column index information set Etc. During implementation, the position of each non-zero element in the current data subset can be determined through the index information set.

[0040] Row Index Information Set The row index information set includes at least one row index element, which may include but is not limited to the row index of the corresponding non-zero element in the current data subset, the index in the non-zero element set, etc. For example, the row index information set includes the row indexes of all non-zero elements in the current data subset in the current data subset. For another example, the row index information set includes the index of the first non-zero element in each row of the current data subset in the non-zero element set or the column index information set. In some embodiments, the row index information set may also include a special row index, which is used to record the total number of non-zero elements contained in the non-zero element set.

[0041] Column index information set The column index information set includes at least one column index element, which may include but is not limited to the column index of the corresponding non-zero element in the current data subset, the index in the non-zero element set, etc. For example, the column index information set includes the column indexes of all non-zero elements in the current data subset in the current data subset. For another example, the column index information set includes the index of the first non-zero element in each column of the current data subset in the non-zero element set or the row index information set. In some embodiments, the column index information set may also include a special column index, which is used to record the total number of non-zero elements contained in the non-zero element set.

[0042] The non-zero element set may be determined in any suitable manner.

[0043] In some embodiments, the set of non-zero elements of the valid data can be determined based on each element in the current data subset and a dynamic threshold. For example, if the element is greater than the dynamic threshold, the element is considered a non-zero element; otherwise, the element is eliminated. For another example, if the difference between the element and the dynamic threshold is not within the threshold range, the element is considered a non-zero element; otherwise, the element is eliminated.

[0044] In some embodiments, the set of non-zero elements of the valid data can be determined based on the clustering results of the current data subset and a dynamic threshold. The clustering results can be obtained using a clustering algorithm such as K-means, K-MEDOIDS, or DBSCAN. For example, for any element in any cluster of the clustering results, if the distance between the element and the cluster center of the cluster in which the element resides is greater than the dynamic threshold, the element is considered a non-zero element; otherwise, the element is removed.

[0045] In some implementations, a sparse object set corresponding to the current data subset may be determined based on a dynamic threshold; and a set of non-zero elements of valid data in the current data subset may be determined based on at least one sparse object in the sparse object set. The sparse objects include the corresponding non-zero elements and the index of the non-zero elements in the current data subset.

[0046] The sparse object set may be determined in any appropriate manner.

[0047] In some embodiments, the sparse object set can be determined based on each element in the current data subset and a dynamic threshold. For example, if an element is greater than the dynamic threshold, the element is considered a non-zero element corresponding to a sparse object; otherwise, the element is removed. For another example, if the difference between an element and the dynamic threshold is not within the threshold range, the element is considered a non-zero element corresponding to a sparse object; otherwise, the element is removed.

[0048] In some embodiments, the sparse object set can be determined based on the clustering result corresponding to the current data subset and the dynamic threshold. For example, for any element in any cluster of the clustering result, if the distance between the element and the cluster center of the cluster where the element is located is greater than the dynamic threshold, then the element is regarded as a non-zero element corresponding to a sparse object, otherwise, the element needs to be eliminated. The clustering result corresponding to the current data subset can be the clustering result of the current data subset, the clustering result of the target data subset, etc. The clustering result can be obtained by using clustering algorithms such as K-means, K-MEDOIDS, DBSCAN, etc. The target data subset is based on the current data subset and the change index set. The change index set includes at least one change index, each change index being used to indicate that the element at the corresponding position is different from the element at that position in the target round iteration. For example, it changes from a zero value to a non-zero value. For another example, it is a different non-zero value. The change index set is determined based on the current data subset and the data subset of the target round iteration. The target round iteration can be the iteration before the current round iteration, for example, the previous round iteration, the first round iteration, etc. The position of the data subset of the target round iteration in the initial data of the target round iteration is the same as the position of the current data subset in the current initial data. During implementation, the size and partitioning strategy of the initial data can be the same in each round iteration, so that the size of the data subset at the same position in each round iteration is the same. For example, if the initial data of each round iteration is 9 rows and 9 columns, and the partitioning step size is 3 rows and 3 columns, then each round iteration includes 9 data subsets, and the first data subset in each round iteration is the first 3 rows and first 3 columns of the initial data of the corresponding round iteration.

[0049] The index information set may be determined in any appropriate manner.

[0050] In some embodiments, the index information set may be determined simultaneously with the determination of the non-zero element set. In some embodiments, the index information set may be determined based on the index of each non-zero element in the non-zero element set in the current data subset. In some embodiments, the index information set may be determined based on the index of each non-zero element in the non-zero element set in the current data subset and the index of each non-zero element in the non-zero element set. In some embodiments, the index information set may be determined based on each sparse object in the sparse object set corresponding to the current data subset.

[0051] In some embodiments, the set of non-zero elements may be determined first, and then the index information set may be determined by other means. For example, the index information set may be determined based on the change index set and the index information set in the valid data of the previous current data subset. The previous current data subset is a data subset obtained by dividing the previous current initial data. The previous current initial data may be the initial data in the target round iteration. For example, the union of the change index set and the index information set in the valid data of the previous current data subset may be used as the index information set.

[0052] In some implementations, because a sparse tensor often contains a large fraction of zero elements, a special compression format can be used to store the valid data of the sparse tensor to conserve memory and improve computational efficiency. The compression format can be any suitable format, such as COO (Coordinate List), CSR (Compressed Sparse Row), or CSC (Compressed Sparse Column). In practice, different compression formats can correspond to the same or different valid data.

[0053] For example, for COO and CSR, each non-zero element in the current data subset is added to the non-zero element set in sequence according to the row value. Therefore, the non-zero element sets of the two are the same, but for COO, the row index information set is used to record the row index of each non-zero element in the current data subset, while for CSR, the row index information set is used to record the index of the first non-zero element of each row in the current data subset in the non-zero element set or column index information set and record the total number of non-zero elements contained in the non-zero element set. Therefore, the row index information sets of the two are different.

[0054] For another example, for CSR and CSC, CSR adds each non-zero element in the current data subset to the non-zero element set in sequence according to the row value, while CSC adds each non-zero element in the current data subset to the non-zero element set in sequence according to the column value. Therefore, the non-zero element sets of the two are different; for CSR, the row index information set is used to record the index of the first non-zero element in each row of the current data subset in the non-zero element set or the column index information set and the total number of non-zero elements contained in the non-zero element set, while for CSC, the row index information set is used to record the row index of each non-zero element in the current data subset. Therefore, the row index information sets of the two are different; for CSR, the column index information set is used to record the column index of each non-zero element in the current data subset, while for CSC, the column index information set is used to record the index of the first non-zero element in each column of the current data subset in the non-zero element set or the row index information set and the total number of non-zero elements contained in the non-zero element set. Therefore, the column index information sets of the two are different.

[0055] For example, if the current data subset , the corresponding valid data can be: For COO, V COO , R COO , C COO ; For CSR, V CSR , R CSR , C CSR ; For CSC, V CSC , R CSC , C CSC ; Among them, R CSR and C CSC The last element 6 in the , is used to represent the set of non-zero elements The total number of nonzero elements contained in .

[0056] In some embodiments, the first thread block corresponding to the current data subset can be first determined, and then the current data subset can be loaded into the first shared memory corresponding to the first thread block. Based on the threads in the first thread block and a dynamic threshold, at least a set of non-zero elements of valid data of the current data subset can be determined. Different current data subsets can correspond to the same or different first thread blocks. In some embodiments, the first thread block can be selected based on the load of each thread block, the utilization rate of each thread block, randomness, the order in which each thread block is used, etc. Different first thread blocks can correspond to the same or different first shared memories. During implementation, two different first thread blocks can reuse the same shared memory to improve resource utilization. Alternatively, each first thread block corresponds to a first shared memory to reduce interference between data. In some embodiments, the index information set can be determined simultaneously with the determination of the non-zero element set.

[0057] In some implementations, at least two current data subsets may be processed in parallel to obtain valid data of the at least two current data subsets, thereby improving data processing efficiency.

[0058] In some implementations, each current data subset may also be processed serially to obtain valid data for each current data subset.

[0059] In the embodiment of the present disclosure, on the one hand, the effective data of the data subset is determined by a dynamic threshold determined in real time, which reduces the possibility of insufficient filtering or excessive filtering compared to the use of fixed thresholds or static rules for identification and filtering in related technologies, thereby not only improving the subsequent calculation accuracy but also improving resource utilization, and being able to adapt to changes in different data distributions and task requirements, improving the compatibility and adaptability of data processing. At the same time, compared to the full amount of calculation in related technologies, since the present disclosure only calculates on the effective data, it not only improves the utilization of calculation resources but also improves the calculation efficiency, and also improves the real-time performance of data calculation; on the other hand, the effective data of the data subset only records each Compared with recording the global index of each non-zero value in the initial data, the local index of the non-zero value in the data subset, first, because each data subset only maintains the internal index, the index maintenance cost is reduced; second, in the subsequent search, since the search is only performed in the local range, the search is simplified while also reducing the difficulty and complexity of the search; finally, in the subsequent memory access, since the access can be performed only in the local memory, unnecessary global memory access is reduced, and the locality of data access is improved while also reducing storage and computing costs; on the other hand, since the data processing method can be used to process image data, sparse tensors, time series data, etc., it has good versatility and expands the application scope of the data processing method.

[0060] In some embodiments, the data processing method further includes steps S141 and S142, wherein: Step S141: Divide the valid data of the current data subset into at least one valid data sub-block.

[0061] Here, the valid data sub-block includes at least a portion of the valid data. In some embodiments, the valid data can be divided according to a pre-set partitioning strategy to obtain the valid data sub-blocks. The partitioning strategy may include, but is not limited to, a partitioning step size, a partitioning method, etc. The partitioning step size can be any suitable step size. The partitioning method may include, but is not limited to, uniform partitioning, uneven partitioning, custom partitioning, random partitioning, etc.

[0062] Step S142: for each valid data sub-block, determine a second thread block corresponding to the valid data sub-block; load the valid data sub-block into a second shared memory corresponding to the second thread block; use at least one thread in the second thread block to perform a parallel operation on at least one non-zero element in the valid data sub-block to obtain a calculation result of the at least one non-zero element in the valid data sub-block, and store the calculation result of the at least one non-zero element in the valid data sub-block into the second shared memory corresponding to the second thread block.

[0063] Here, the second thread block can be any suitable thread block, each of which is used to parallelize operations on corresponding valid data sub-blocks. In some embodiments, the second thread block can be a thread block in a CPU or GPU. In implementation, different valid data sub-blocks can correspond to the same or different second thread blocks.

[0064] The second thread block may be determined in any suitable manner. For example, a thread block in an idle state may be randomly selected as the second thread block. In another example, the second thread block may be determined based on the order in which the thread blocks are used. In another example, the second thread block may be selected based on a comprehensive consideration of the load and usage rate of the thread blocks.

[0065] The second shared memory can be any suitable shared memory, and different second thread blocks can correspond to the same or different second shared memories. In some embodiments, two different second thread blocks can reuse the same shared memory to improve resource utilization; alternatively, each second thread block corresponds to a second shared memory to reduce data interference. During implementation, the second shared memory is also used to store the calculation results of at least one non-zero element in the valid data sub-block. In some embodiments, the second shared memory can be located in the CPU or GPU.

[0066] In some embodiments, each non-zero element in the valid data sub-block can be assigned to a thread in the second thread block for processing, thereby achieving parallel computing using multiple threads in the second thread block. The thread performs operations such as matrix multiplication, matrix division, torque transposition, and matrix inversion on the non-zero elements to obtain a computation result for the non-zero element, and stores the computation result for the non-zero element in the second shared memory.

[0067] In the embodiment of the present disclosure, on the one hand, the valid data sub-block is pre-loaded into the second shared memory corresponding to the second thread block, so that each thread in the second thread block only accesses the data in the second shared memory for processing. Compared with reading data from the global access memory, not only the number of global memory accesses is reduced, but also the global memory access latency is reduced, thereby accelerating the processing speed and achieving the purpose of efficient parallel processing; on the other hand, the efficiency of data parallel processing is improved by having each thread in the second thread block perform operations on multiple non-zero elements at the same time. At the same time, since tasks are allocated in an intra-block parallel manner, compared with allocating tasks to threads according to element granularity, not only the possibility of load imbalance is reduced, thread allocation is optimized, but also resource utilization is maximized.

[0068] In some embodiments, the data processing method further includes steps S151 and S152, wherein: Step S151: Using the trained intelligent agent, determine the threshold adjustment range based on the state corresponding to the current initial data. The state corresponding to the current initial data is determined based on at least one of the sparsity rate corresponding to the current initial data, the dynamic threshold corresponding to the current initial data, the filtering efficiency corresponding to the current initial data, and the calculation accuracy corresponding to the current initial data.

[0069] Here, the agent is used to predict the threshold adjustment range based on the state corresponding to the current initial data. The state corresponding to the current initial data is used to describe the characteristics of the current initial data, system performance, etc.

[0070] In some embodiments, the state corresponding to the current initial data may be represented by at least one of a sparsity rate corresponding to the current initial data, a dynamic threshold corresponding to the current initial data, a filtering efficiency corresponding to the current initial data, and a computational precision corresponding to the current initial data. For example, the state corresponding to the current initial data may be represented by a sparsity rate corresponding to the current initial data and a dynamic threshold corresponding to the current initial data. For another example, the state corresponding to the current initial data may be represented by a sparsity rate corresponding to the current initial data, a dynamic threshold corresponding to the current initial data, and a computational precision corresponding to the current initial data.

[0071] The sparsity ratio corresponding to the current initial data is determined based on a second ratio between the total amount of valid data corresponding to the current initial data and the total amount of the current initial data. For example, the sparsity ratio may be the second ratio, a weighted value of the second ratio, a logarithm of the second ratio, or the like.

[0072] The filtration efficiency corresponding to the current initial data is determined based on a third ratio between the total number of non-zero elements included in the valid data corresponding to the current initial data and the total number of non-zero elements included in the current initial data. For example, the filtration efficiency corresponding to the current initial data can be the third ratio, the difference between the third ratio and a target value, a weighted value of the difference, etc. The target value can be any appropriate value, such as 1, 0.5, etc.

[0073] In some embodiments, the following formula (1-3) can be used to determine the The filtering efficiency corresponding to the initial data in the round iteration ,Right now: (1-3); in, Indicates the The valid data corresponding to the initial data in the round iteration, Indicates the Initial data in the round iteration, Indicates the number of non-zero elements.

[0074] The calculation accuracy corresponding to the current initial data is determined based on the calculation result of the valid data corresponding to the current initial data and the target calculation result. The calculation accuracy corresponding to the current initial data can be a fourth ratio, a difference between the fourth ratio and the target value, a weighted value of the difference, etc. Among them, the fourth ratio is determined based on the calculation result of the valid data corresponding to the current initial data and the target calculation result. For example, the fourth ratio can be the ratio between the calculation result of the valid data corresponding to the current initial data and the target calculation result. For another example, the fourth ratio can be the ratio between the second norm of the fourth difference and the second norm of the target calculation result, and the fourth difference refers to the difference between the calculation result of the valid data corresponding to the current initial data and the target calculation result.

[0075] In some embodiments, the following formula (1-4) can be used to determine the The calculation accuracy corresponding to the initial data in the round iteration ,Right now: (1-4); in, Indicates the The calculation result of the valid data corresponding to the initial data in the round iteration, Indicates the The target operation result of the initial data in the round iteration, represents the two-norm.

[0076] In some embodiments, for After a round of iteration, the state corresponding to the initial data can be expressed by the following formula (1-5): ,Right now: (1-5); in, Indicates the The sparsity rate corresponding to the initial data in the round iteration, Indicates the The dynamic threshold corresponding to the initial data in the round iteration, Indicates the The filtering efficiency corresponding to the initial data in the round iteration, Indicates the The calculation accuracy corresponding to the initial data in the round iteration.

[0077] The action of the agent is used to dynamically determine the threshold adjustment amplitude. The action of the agent can be expressed by the following formula (1-6) ,Right now: (1-6); in, Indicates the threshold adjustment range, Indicates the adjustment step size.

[0078] Step S152: Determine the dynamic threshold corresponding to the next initial data based on the sum of the threshold adjustment amplitude and the dynamic threshold corresponding to the current initial data.

[0079] Here, the dynamic threshold may be determined by methods including, but not limited to, a second sum value, a weighted second sum value, and the like. The second sum value refers to the sum of the threshold adjustment amplitude and the dynamic threshold value corresponding to the current initial data. For example, the second sum value is used as the dynamic threshold value corresponding to the next current initial data. In some embodiments, the dynamic threshold value corresponding to the next initial data can be obtained using the aforementioned formula (1-2).

[0080] In the embodiments disclosed herein, on the one hand, the threshold adjustment amplitude is dynamically predicted by the trained intelligent agent according to the state corresponding to the current initial data, thereby shortening the time for determining the threshold adjustment amplitude. At the same time, since the state corresponding to the initial data is determined based on feedback data such as sparsity rate, filtering efficiency, and calculation accuracy, the purpose of dynamically optimizing the dynamic threshold through the feedback mechanism is achieved, thereby ensuring the stability and accuracy of the dynamic threshold. On the other hand, the next dynamic threshold is determined according to the threshold adjustment amplitude, and the dynamic threshold can be adaptively adjusted online in a variety of scenarios, which not only improves the compatibility and adaptability of the data processing method, but also enables continuous optimization in a complex and changeable computing environment, thereby improving robustness.

[0081] In some embodiments, the data processing method further includes steps S161 to S163, wherein: Step S161: Determine the state corresponding to the training data and the action corresponding to the training data.

[0082] Here, the training data can be any suitable data, such as image data, sparse tensor data, time series data, etc. The determination of the state corresponding to the training data is similar to the determination of the state corresponding to the current initial data. When implementing this, please refer to the specific implementation of step S151 above.

[0083] The actions corresponding to the training data refer to the actions of the intelligent agent to be trained. During implementation, the actions of the intelligent agent to be trained can be defined by the aforementioned formula (1-6).

[0084] Step S162: Determine a reward function corresponding to the training data based on the computational accuracy corresponding to the training data.

[0085] Here, the reward function may be any suitable function, and the reward function may be determined in any suitable manner.

[0086] In some embodiments, the reward function is used to balance filtering efficiency and computational accuracy. Thus, the reward function can be determined based on the computational accuracy and the filtering efficiency corresponding to the training data. During implementation, the agent adjusts the dynamic threshold online to maximize filtering efficiency while maintaining computational accuracy. The computational accuracy must be no less than a pre-set accuracy threshold. .

[0087] The method for determining the reward function may include but is not limited to the difference between the operation accuracy corresponding to the training data and the filtering efficiency corresponding to the training data, the third difference, the weight of the third difference, etc. For example, the reward function may be the third difference. The third difference is the difference between the first product and the second product. The first product refers to the product between the filtering efficiency corresponding to the training data and the weight of the filtering efficiency corresponding to the training data. The second product is determined based on the operation accuracy corresponding to the training data and the weight of the operation accuracy corresponding to the training data. For example, the second product may be the product between the operation accuracy corresponding to the training data and the weight of the operation accuracy corresponding to the training data. For another example, the second product may be the product between the weight of the operation accuracy corresponding to the training data and the fifth difference. The fifth difference refers to the difference between the operation accuracy corresponding to the training data and the target value.

[0088] In some embodiments, for For each round of iteration, the reward function can be defined by the following formula (1-7): ,Right now: (1-7); in, Indicates the The filtering efficiency corresponding to the training data in the round iteration, Represents the weight of the filtering efficiency corresponding to the training data, Indicates the The computational accuracy of the training data in the round iteration, Indicates the weight of the calculation accuracy corresponding to the training data.

[0089] In some embodiments, the reward function is used to weigh the computing resource utilization, computing delay, and computing accuracy. Then, the reward function can be determined based on the computing resource utilization corresponding to the training data, the computing delay corresponding to the training data, and the filtering efficiency corresponding to the training data. The method for determining the reward function may include but is not limited to the third sum between the sixth difference and the fifth product, the weighting of the third sum, etc. For example, the reward function can be the third sum. The sixth difference refers to the difference between the third product and the fourth product. The third product refers to the product between the computing resource utilization corresponding to the training data and the weight of the computing resource utilization corresponding to the training data. The fourth product refers to the product between the computing delay corresponding to the training data and the weight of the computing delay corresponding to the training data. The fifth product refers to the product between the filtering efficiency corresponding to the training data and the weight of the filtering efficiency corresponding to the training data.

[0090] In some embodiments, for For each round of iteration, the reward function can be defined by the following formula (1-8): ,Right now: (1-8); in, Indicates the The computing resource utilization corresponding to the training data in the round iteration, Indicates the weight of the computing resource utilization corresponding to the training data, Indicates the The computational delay corresponding to the training data in the round iteration, Represents the weight of the computational delay corresponding to the training data, Indicates the The computational accuracy of the training data in the round iteration, Indicates the weight of the calculation accuracy corresponding to the training data.

[0091] Step S163: Based on the state corresponding to the training data, the action corresponding to the training data, and the reward function corresponding to the training data, the intelligent agent is subjected to reinforcement training to obtain a trained intelligent agent.

[0092] Reinforcement learning is a method of learning through the interaction between an agent and its environment. In reinforcement learning, the agent takes a series of actions, and the environment returns rewards or penalties based on the actions taken, as well as feedback on the current state. Based on the current state and previous experience, the agent selects the best action to maximize the long-term cumulative reward.

[0093] In some implementations, reinforcement learning algorithms such as DQN (Deep Q-Network), PPO (Proximal Policy Optimization), and A3C (Asynchronous Advantage Actor-Critic) can be used to learn the optimal strategy through the interaction between the agent and the environment. , in order to maximize the cumulative reward. In implementation, the state corresponding to the training data is used as the input state of the agent, and the action corresponding to the training data is used as the execution range of the agent. The training is carried out with the goal of maximizing the reward function.

[0094] In some embodiments, for The optimal strategy can be defined by the following formula (1-9) ,Right now: (1-9); in, represents the number of iterations, represents the discount factor, Indicates the The reward function in the round iteration, represents the action of the agent, Represents the state of the agent (i.e., the state corresponding to the training data).

[0095] In the disclosed embodiment, the intelligent agent is pre-trained with intensive training through training data, so that the trained intelligent agent can be used to output the optimal action (i.e., the threshold adjustment range) according to the real-time state, thereby shortening the time for determining the subsequent threshold adjustment range. At the same time, since the state corresponding to the intelligent agent is determined based on feedback data such as sparsity rate, filtering efficiency, and calculation accuracy, the purpose of dynamically optimizing the dynamic threshold through the feedback mechanism is achieved, thereby ensuring the stability and accuracy of the dynamic threshold.

[0096] In some embodiments, step S162 includes steps S1621 to S1623, wherein: Step S1621: Determine the filtering efficiency corresponding to the training data based on the valid data corresponding to the training data and the training data.

[0097] Here, the filtration efficiency may be determined in any suitable manner.

[0098] In some implementations, the ratio may be determined based on the ratio between the total amount of data included in the valid data corresponding to the training data and the total amount of data included in the training data.

[0099] In some embodiments, the filtering efficiency corresponding to the training data may be a fifth ratio, a difference between the fifth ratio and a target value, a weighted value of the difference, etc. The fifth ratio refers to the ratio between the total number of non-zero elements contained in the valid data corresponding to the training data and the total number of non-zero elements contained in the training data. For example, the filtering efficiency corresponding to the training data may be determined using the aforementioned formula (1-3).

[0100] Step S1622: Determine the calculation accuracy corresponding to the training data based on the calculation result of the valid data corresponding to the training data and the target calculation result corresponding to the training data.

[0101] Here, the calculation accuracy corresponding to the training data can be the sixth ratio, the difference between the sixth ratio and the target value, the weight of the difference, etc. For example, the calculation accuracy corresponding to the training data can be the difference between the sixth ratio and the target value. The sixth ratio is determined based on the calculation result of the valid data corresponding to the training data and the target calculation result. For example, the sixth ratio can be the ratio between the calculation result of the valid data corresponding to the training data and the target calculation result. For another example, the sixth ratio can be the ratio between the second norm of the seventh difference and the second norm of the target calculation result, where the seventh difference refers to the difference between the calculation result of the valid data corresponding to the training data and the target calculation result.

[0102] In some implementations, the computational accuracy corresponding to the training data may be determined using the aforementioned formulas (1-4).

[0103] Step S1623: Determine a reward function corresponding to the training data based on the filtering efficiency corresponding to the training data and the calculation accuracy corresponding to the training data.

[0104] Here, the reward function is used to balance filtering efficiency and computational accuracy. Methods for determining this reward function may include, but are not limited to, the difference between the computational accuracy corresponding to the training data and the filtering efficiency corresponding to the training data, a third difference, and a weighted third difference. The third difference is the difference between the first product and the second product. The first product is the product of the filtering efficiency corresponding to the training data and the weight of the filtering efficiency corresponding to the training data. The second product is determined based on the computational accuracy corresponding to the training data and the weight of the computational accuracy corresponding to the training data. In some embodiments, the reward function corresponding to the training data can be determined using the aforementioned formulas (1-7).

[0105] In the disclosed embodiment, the reward function is determined by comprehensively considering filtering efficiency and computational accuracy, emphasizing the need to balance these two factors when automatically adjusting the dynamic threshold, thereby ensuring the stability and accuracy of the dynamic threshold while implementing an intelligent adaptive optimization mechanism.

[0106] In some embodiments, step S1622 includes steps S171 to S173, wherein: Step S171: Determine a first difference between a calculation result of valid data corresponding to the training data and a target calculation result corresponding to the training data.

[0107] Step S172: Determine a first ratio based on the first difference and the target operation result corresponding to the training data.

[0108] Here, the first ratio may be determined in any suitable manner. For example, the first ratio may be the ratio between the first difference and the target operation result corresponding to the training data. In another example, the first ratio may be a weighted ratio of the first difference and the target operation result corresponding to the training data. In another example, the first ratio may be the ratio between the second norm of the first difference and the second norm of the target operation result corresponding to the training data.

[0109] Step S173: Determine the calculation accuracy corresponding to the training data based on the second difference between the first ratio and the target value.

[0110] Here, the calculation accuracy may be determined in a manner including but not limited to the second difference, weighting of the second difference, etc. For example, the second difference is used as the calculation accuracy corresponding to the training data.

[0111] In the disclosed embodiment, the operation precision is determined according to the operation result of the valid data and the target operation result, thereby improving the accuracy of the operation precision.

[0112] In some embodiments, step S1623 includes steps S181 to S183, wherein: Step S181: Determine a first product between the filtering efficiency corresponding to the training data and the weight of the filtering efficiency corresponding to the training data.

[0113] Here, the first product refers to the product of the filtering efficiency corresponding to the training data and the weight of the filtering efficiency corresponding to the training data.

[0114] Step S182: Determine a second product based on the calculation accuracy corresponding to the training data and the weight of the calculation accuracy corresponding to the training data.

[0115] Here, the second product may be determined by, but is not limited to, a product of the computational precision corresponding to the training data and a weight of the computational precision corresponding to the training data, a product of the weight of the computational precision corresponding to the training data and a fifth difference, etc. The fifth difference refers to the difference between the computational precision corresponding to the training data and a target value.

[0116] Step S183: Determine a reward function corresponding to the training data based on a third difference between the first product and the second product.

[0117] Here, the reward function may be determined in a manner including but not limited to the third difference, a weighted value of the third difference, etc. For example, the reward function may be the third difference.

[0118] In the embodiment of the present disclosure, a reward function is determined based on the filtering efficiency, the weight of the filtering efficiency, the calculation accuracy, and the weight of the calculation accuracy, thereby improving the accuracy of the reward function.

[0119] In some embodiments, step S162 includes steps S191 to S192, wherein: Step S191: Determine the computing resource utilization corresponding to the training data and the computing delay corresponding to the training data.

[0120] Here, computing resource utilization refers to the utilization of computing resources. Computing resources can be any appropriate resources used during the computing process, such as thread blocks, memory, etc.

[0121] The computing resource utilization rate may be determined in any suitable manner. During implementation, the computing resource utilization rate may be determined based on the utilization rate of at least one computing resource. Methods for determining the computing resource utilization rate may include, but are not limited to, the utilization rate of a particular computing resource, the mean / variance / mean squared deviation of the utilization rates of some computing resources, or the mean / variance / mean squared deviation of the utilization rates of all computing resources.

[0122] For example, if the computing resources include thread blocks, the thread block utilization can be determined based on the number of thread blocks used by the valid data corresponding to the training data and the total number of thread blocks, and the thread block utilization can be used as the computing resource utilization. For another example, if the computing resources include memory, the memory utilization can be determined based on the shared memory occupied by the valid data corresponding to the training data and all shared memories, and the memory utilization can be used as the computing resource utilization. For another example, the average of the thread block utilization and the memory utilization can be used as the computing resource utilization.

[0123] Operational delay primarily refers to the delay during the operation. During the operation of training data, the actual operation duration of the training data can be pre-recorded. Based on the actual operation duration and the target operation duration, the operation delay can be obtained. Methods for determining the operation delay may include, but are not limited to, the difference between the actual operation duration and the target operation duration, the weighting of the difference, etc. For example, the difference can be used as the operation delay.

[0124] Step S192: Determine a reward function corresponding to the training data based on the computing resource utilization corresponding to the training data, the delay corresponding to the training data, and the computing accuracy corresponding to the training data.

[0125] Here, the reward function may be determined by, but is not limited to, a third sum between the sixth difference and the fifth product, a weighted third sum, and the like. For example, the reward function may be the third sum. The sixth difference refers to the difference between the third product and the fourth product. The third product refers to the product of the computing resource utilization corresponding to the training data and the weight of the computing resource utilization corresponding to the training data. The fourth product refers to the product of the computing delay corresponding to the training data and the weight of the computing delay corresponding to the training data. The fifth product refers to the product of the filtering efficiency corresponding to the training data and the weight of the filtering efficiency corresponding to the training data.

[0126] In the disclosed embodiment, the reward function is determined based on computing resource utilization, computing latency, and computing accuracy, emphasizing the need to balance these three factors when automatically adjusting the dynamic threshold, while implementing an intelligent adaptive optimization mechanism to ensure the stability and accuracy of the dynamic threshold.

[0127] In some embodiments, step S193 includes steps S1931 to S1934, wherein: Step S1931: Determine a third product between the computing resource utilization corresponding to the training data and the weight of the computing resource utilization corresponding to the training data.

[0128] Here, the third product refers to the product between the resource utilization and the weight of the resource utilization.

[0129] Step S1932: Determine a fourth product between the operation delay corresponding to the training data and the weight of the operation delay corresponding to the training data.

[0130] Here, the fourth product refers to the product between the operation delay and the weight of the operation delay.

[0131] Step S1933: Determine the fifth product between the calculation accuracy corresponding to the training data and the weight of the calculation accuracy corresponding to the training data.

[0132] Here, the fifth product refers to the product of the operation precision and the weight of the operation precision.

[0133] Step S1934: Determine a reward function corresponding to the training data based on the third product, the fourth product, and the fifth product.

[0134] Here, the reward function may be determined by, but is not limited to, a third sum between the sixth difference and the fifth product, a weighted third sum, or the like. For example, the reward function may be the third sum. The sixth difference refers to the difference between the third product and the fourth product.

[0135] In the disclosed embodiment, the reward function is determined based on resource utilization, operation delay, operation accuracy, and weights of various factors, thereby improving the accuracy of the reward function.

[0136] Figure 2 A schematic diagram of the implementation process of a data processing method provided in the embodiment of the present disclosure Figure 2 ,like Figure 2 As shown, the detection method includes steps S21 to S24, wherein: Step S21: pre-process the current initial data to obtain first data, where the pre-processing includes at least one of the following: standardization processing and outlier processing.

[0137] Here, outliers are values ​​that significantly deviate from other data points in a dataset. These may be data errors or real-world extremes, but they typically affect the final results. Therefore, outlier processing is required for the initial data, i.e., removing outliers from the data. In some embodiments, outlier processing can be performed using algorithms such as the Z-score and the Inter Quartile Range (IQR). Quartiles are a type of quantile in statistics. They divide all values ​​into four equal parts, arranged from smallest to largest. The values ​​at the three dividing points are quartiles, and the difference between the third quartile and the first quartile is also called the IQR. The Z-score is a statistic used to measure the degree of deviation of a data point from the data mean. It represents the relative standard distance of a data point from the mean. If the absolute value of a data point's Z-score is greater than a set threshold (e.g., 2, 3, etc.), the data point is considered an outlier.

[0138] Normalization involves converting data into dimensionless, magnitude-invariant values, ensuring that the values ​​of different features can be compared on the same scale. In some embodiments, normalization can be performed using methods such as Z-scores and Min-Max Normalization. Z-scores are based on the mean and standard deviation of the data and convert the data into a standard normal distribution with a mean of 0 and a standard deviation of 1. Min-Max Normalization scales the data by its minimum and maximum values, linearly mapping them to a fixed range (e.g., [0, 1], [-1, 1]).

[0139] In some implementations, the initial data may be processed for outliers to obtain second data, and then the second data may be normalized to obtain the first data, thereby reducing the impact of outliers on the normalized data and improving the effect of normalization.

[0140] In some embodiments, Z-scores can be used to simultaneously perform outlier processing and normalization on the initial data.

[0141] Step S22: Determine a dynamic threshold corresponding to the current initial data based on the data characteristics of the first data.

[0142] Here, the features of the first data may include, but are not limited to, global features of the first data, local features of the first data, etc. In implementation, the features of the data may be extracted using any suitable feature extraction method, such as histogram, principal component analysis, CNN, RNN, Transformer, etc.

[0143] Different data features may correspond to the same or different dynamic thresholds. The dynamic threshold may be determined in any suitable manner. In some embodiments, the data features of the first data may be input into the aforementioned formula (1-1) to obtain the dynamic threshold. In some embodiments, a correspondence between each data feature and a threshold may be pre-established. Based on this correspondence, the threshold corresponding to the data feature may be obtained, and the threshold corresponding to the data feature may be used as the dynamic threshold.

[0144] Step S23: Divide the current initial data into at least one current data subset.

[0145] Step S24: For each current data subset, determine the non-zero element set of valid data of the current data subset based on the dynamic threshold, where the valid data of the current data subset also includes an index information set, which is used to characterize the index information of each non-zero element in the non-zero element set in the current data subset.

[0146] Here, the above steps S23 to S24 correspond to the above steps S12 to S13 respectively. When implementing, please refer to the specific implementation of the above steps S12 to S13.

[0147] In the embodiments of the present disclosure, on the one hand, by performing outlier processing and / or standardization on the initial data to improve data quality, the processing volume is reduced while ensuring the accuracy and stability of subsequent results, thereby improving data processing efficiency; on the other hand, the appropriate dynamic threshold is dynamically predicted based on the characteristics of the initial data. Compared with the use of fixed thresholds in related technologies, since the dynamic threshold can be adaptively adjusted in a variety of scenarios, not only the compatibility and adaptability of the data processing method are improved, but also the possibility of misjudgment and limitations due to the use of fixed thresholds is reduced, thereby achieving the purpose of improving the reliability, accuracy and flexibility of data processing.

[0148] Figure 3 A schematic diagram of the implementation process of a data processing method provided in the embodiment of the present disclosure Figure 3 ,like Figure 3 As shown, the detection method includes steps S31 to S33, wherein: Step S31: Determine the dynamic threshold corresponding to the current initial data.

[0149] Here, the above-mentioned step S31 corresponds to the above-mentioned step S11 or the above-mentioned steps S21 to S22. When implementing, reference may be made to the specific implementation of the above-mentioned step S11 or the above-mentioned steps S21 to S22.

[0150] Step S32: Divide the current initial data into at least one current data subset.

[0151] Here, the above step S32 corresponds to the above step S12. When implementing, please refer to the specific implementation of the above step S12.

[0152] Step S33: For each current data subset, based on the dynamic threshold, determine the sparse object set corresponding to the current data subset, and based on at least one sparse object in the sparse object set corresponding to the current data subset, determine the non-zero element set of valid data of the current data subset, the valid data of the current data subset also includes an index information set, and the index information set is used to represent the index information of each non-zero element in the non-zero element set in the current data subset.

[0153] Here, the sparse object set includes at least one sparse object, and the sparse object includes a corresponding non-zero element and an index of the non-zero element in the current data subset. The index of the non-zero element in the current data subset includes a row index of the non-zero element in the current data subset and a column index of the non-zero element in the current data subset.

[0154] The sparse object set may be determined in any appropriate manner.

[0155] In some embodiments, the sparse object set can be determined based on each element in the current data subset and a dynamic threshold. For example, if an element is greater than the dynamic threshold, the element is considered a non-zero element corresponding to a sparse object; otherwise, the element is removed. For another example, if the difference between an element and the dynamic threshold is not within the threshold range, the element is considered a non-zero element corresponding to a sparse object; otherwise, the element is removed.

[0156] In some embodiments, the sparse object set can be determined based on the clustering result corresponding to the current data subset and the dynamic threshold. For example, for any element in any cluster of the clustering result, if the distance between the element and the cluster center of the cluster where the element is located is greater than the dynamic threshold, the element is regarded as a non-zero element corresponding to a sparse object, otherwise, the element needs to be eliminated. The clustering result corresponding to the current data subset can be the clustering result of the current data subset, the clustering result of the target data subset, etc. The target data subset is based on the current data subset and the change index set. It is determined that the change index set includes at least one change index, and each change index is used to indicate that the element at the corresponding position is different from the element at the position in the target round iteration.

[0157] For example, if the current data subset , and its corresponding sparse object set can be: Among them, sparse objects Including the corresponding non-zero element 1, the non-zero element 1 in The row index 0 in and the non-zero element 1 in Column index 0 in .

[0158] The index information set may include but is not limited to a row index information set. , column index information set The index information set may be determined in any appropriate manner.

[0159] In some embodiments, the index information set can be determined simultaneously with the determination of the non-zero element set. In some embodiments, the index information set can be determined based on the indices of the non-zero elements corresponding to each sparse object in the sparse object set in the current data subset. For example, the row index information set can be determined based on the row indices of the non-zero elements corresponding to each sparse object in the current data subset; the column index information set can be determined based on the column indices of the non-zero elements corresponding to each sparse object in the current data subset. In some embodiments, the index information set can be determined based on the target compression format and the indices of the non-zero elements corresponding to each sparse object in the sparse object set in the current data subset. The target compression format may include, but is not limited to, a first compression format, a second compression format, a third compression format, etc. The first compression format, the second compression format, and the third compression format are all different. For example, the first compression format may be COO, the second compression format may be CSR, and the third compression format may be CSC. In implementation, different target compression formats correspond to the same or different index information sets. For example, for COO and CSC, the row index information set is used to record the row indices of all non-zero elements in the current data subset. For CSR, the row index information set is used to record the index of the first non-zero element of each row in the current data subset in the non-zero element set or column index information set and the total number of non-zero elements contained in the non-zero element set.

[0160] In some implementations, the non-zero element set may be determined first, and then the index information set may be determined by other means, for example, based on the change index set and the index information set in the valid data of the previous current data subset.

[0161] In the embodiments of the present disclosure, on the one hand, the sparse object set corresponding to the current data subset is determined in real time based on the dynamic threshold, which reduces the possibility of insufficient filtering or excessive filtering, thereby improving the accuracy of the sparse object set; on the other hand, valid data is determined based on the sparse object set, thereby improving the accuracy of the valid data.

[0162] In some embodiments, the sparse object set corresponding to the current data subset includes at least one sparse object, and the step S33 of “determining the sparse object set corresponding to the current data subset based on the dynamic threshold” includes steps S331 and S332, wherein: Step S331: cluster the current data subset to obtain at least one cluster.

[0163] Here, a clustering algorithm such as K-means, K-MEDOIDS, or DBSCAN can be used to cluster the current data subset to obtain at least one cluster. In some embodiments, before clustering the current data subset, the current data subset can be standardized to ensure that the scale of each data is the same, thereby improving the clustering effect.

[0164] Step S332: For each element in each cluster, determine the distance between the element and the cluster center of the cluster where the element is located. When the distance is greater than a dynamic threshold, treat the element as a non-zero element corresponding to a sparse object. The sparse object includes the corresponding non-zero element and the index of the non-zero element in the current data subset.

[0165] Here, the distance can be Euclidean distance, Mahalanobis distance, etc. In implementation, since redundant data is usually located in an area with high density and very close to the cluster center, when the distance between certain elements and the cluster center of the cluster they belong to is not greater than the dynamic threshold, these elements are considered redundant and should be eliminated.

[0166] In the embodiment of the present disclosure, a dynamic threshold is used to automatically identify and select non-zero elements in each cluster, which shortens the time for determining the non-zero elements of the current data subset and improves the recognition efficiency.

[0167] In some embodiments, the step S33 of “determining the sparse object set corresponding to the current data subset based on the dynamic threshold” includes steps S341 to S343, wherein: Step S341: Determine the change index set based on the valid data of the previous current data subset and the current data subset. The previous current data subset is a data subset obtained by dividing the previous current initial data. The position of the previous current data subset in the previous current initial data is the same as the position of the current data subset in the current initial data.

[0168] Here, the previous current initial data may be the initial data in the target round iteration. The target round iteration may be the round iteration before the current round iteration, for example, the previous round iteration, the first round iteration, etc. During implementation, in each round iteration, the size and partitioning strategy of the initial data may be the same, so that in each round iteration, the size of the data subset at the same position is the same. For example, if the initial data of each round iteration is 9 rows and 9 columns, and the partitioning step size is 3 rows and 3 columns, then each round iteration includes 9 data subsets, and the second data subset in each round iteration may be the first 3 rows and the middle 3 columns of the initial data of the corresponding round iteration.

[0169] The change index set includes at least one change index, each change index being used to indicate that the element at the corresponding position is different from the element at that position in the target round iteration. During implementation, valid data from each previous data subset can be saved. Thus, the change index set can be obtained by comparing the valid data from the previous data subset with the current data subset.

[0170] For example, if in the previous iteration, the previous set of non-zero elements can be expressed as: ,in, Indicates the Rank Elements of the column, Indicates that, represents the dynamic threshold; By comparison and the current data subset , we can get the change index set : .

[0171] Step S342: Determine the target data subset based on the change index set and the current data subset.

[0172] Here, the target data subset includes at least some of the elements in the current data subset. During implementation, the elements involved in each change index of the change index set in the current data subset can be used as an element in the target data subset. For example, if the index of an element in the current data subset is the same as a change index, then the element is used as an element in the target data subset. Then, the target data subset can be expressed as: .in, Indicates the Rank Elements of the column, Indicates that, Indicates a dynamic threshold.

[0173] Step S343: Determine the sparse object set corresponding to the current data subset based on the dynamic threshold and the target data subset.

[0174] Here, the determination method of the sparse object set can be any appropriate method. In some embodiments, the sparse object set can be determined based on the elements in the target data subset and the dynamic threshold. For example, if an element is greater than the dynamic threshold, the element is treated as a non-zero element corresponding to a sparse object, otherwise, the element needs to be eliminated. For another example, if the difference between an element and the dynamic threshold is not within the threshold range, the element is treated as a non-zero element corresponding to a sparse object, otherwise, the element is eliminated. In some embodiments, the sparse object set can be determined based on the clustering result of the target data subset and the dynamic threshold. For example, for any element in any cluster of the clustering result, if the distance between the element and the cluster center of the cluster where the element is located is greater than the dynamic threshold, the element is treated as a non-zero element corresponding to a sparse object, otherwise, the element needs to be eliminated.

[0175] In the embodiment of the present disclosure, in the scenario of multiple rounds of iterations, since the changes in the initial data in each round of iteration are localized, the changed area of ​​the data is located through the change index set, and only the data in the changed area is processed to obtain valid data, without the need to fully scan and repeatedly process the entire data subset, which significantly reduces the processing cost, improves data processing efficiency and the real-time performance of non-zero data.

[0176] In some embodiments, the data processing method further includes step S35, wherein: Step S35: Determine the index information set of the valid data of the current data subset based on the change index set and the index information set in the valid data of the previous current data subset.

[0177] Here, the index information set in the valid data of the current data subset can be determined based on the union of the change index set and the index information set in the valid data of the previous current data subset. For example, the index information set in the valid data of the current data subset can be expressed as follows: .

[0178] In the embodiment of the present disclosure, on the one hand, by recording the index information set in the valid data of the current data subset, it is convenient to determine the next change index set in the next round of iteration, so as to further improve the data processing efficiency; on the other hand, since only local indexes are recorded, the index maintenance cost and storage cost are reduced, and the search efficiency of subsequent data is improved.

[0179] In some embodiments, the step S33 of “determining a set of non-zero elements of valid data of the current data subset based on at least one sparse object in the sparse object set corresponding to the current data subset” includes step S361, wherein: Step S361: Determine a non-zero element set based on the non-zero elements corresponding to each sparse object and the target compression format.

[0180] Here, the target compression format may include, but is not limited to, a first compression format, a second compression format, a third compression format, etc., wherein the first compression format, the second compression format, and the third compression format are all different. For example, the first compression format may be COO, the second compression format may be CSR, and the third compression format may be CSC.

[0181] In some embodiments, different target compression formats correspond to the same or different sets of non-zero elements. For example, for both COO and CSR, each non-zero element in the current data subset is sequentially added to the non-zero element set according to row values; therefore, the non-zero element sets of the two are the same. For another example, for CSR and CSC, CSR sequentially adds each non-zero element in the current data subset to the non-zero element set according to row values, while CSC sequentially adds each non-zero element in the current data subset to the non-zero element set according to column values; therefore, the non-zero element sets of the two are different.

[0182] In the embodiment of the present disclosure, the non-zero element set is determined according to each sparse object and the target compression format, thereby improving the accuracy of the non-zero element set of valid data.

[0183] In some embodiments, the data processing method further includes step S362, wherein: Step S362: Determine an index information set of valid data in the current data subset based on the index of the non-zero element corresponding to each sparse object in the current data subset.

[0184] Here, the index information set may include but is not limited to a row index information set, a column index information set, etc. The index information set may be determined in any appropriate manner.

[0185] In some embodiments, the row index information set may be determined based on the row index of the non-zero elements corresponding to each sparse object in the current data subset. In some embodiments, the column index information set may be determined based on the column index of the non-zero elements corresponding to each sparse object in the current data subset.

[0186] In some embodiments, different target compression formats correspond to the same or different index information sets. In some embodiments, the row index information set can be determined based on the target compression format and the index of the non-zero element corresponding to each sparse object in the non-zero element set or the column index information set. In some embodiments, the column index information set can be determined based on the target compression format and the index of the non-zero element corresponding to each sparse object in the non-zero element set or the row index information set.

[0187] During implementation, the size of the valid data of the data subset after compression by the target compression format It can be expressed by the following formula (3-1): (3-1); in, represents the size of the set of non-zero elements, Indicates the size of the row index information set, Indicates the size of the column index information set.

[0188] It can be seen that compared with the data subset, the size of the data is further reduced after using this target compression format.

[0189] In the embodiment of the present disclosure, the index information set is determined according to the index of the non-zero element corresponding to each sparse object in the current data subset, thereby improving the accuracy of the index information set of valid data.

[0190] In some embodiments, the index of the non-zero element in the current data subset includes the row index of the non-zero element in the current data subset and the column index of the non-zero element in the current data subset; step S361 includes step S3611 and / or step S3612, wherein: Step S3611: When the target compression format is the first compression format or the second compression format, determine a non-zero element set based on the row index of the non-zero element corresponding to each sparse object in the current data subset.

[0191] Here, the first compression format may be COO, and the second compression format may be CSR. In implementation, each non-zero element in the current data subset may be added to the non-zero element set in sequence according to each row index.

[0192] For example, if the current data subset The corresponding sparse object set is: When the target compression format is COO or CSR, the non-zero element set is V COO =V CSR .

[0193] Step S3612: When the target compression format is the third compression format, determine a non-zero element set based on the column index of the non-zero element corresponding to each sparse object in the current data subset.

[0194] Here, the third compression format may be CSC. In implementation, each non-zero element in the current data subset may be added to the non-zero element set in sequence according to each column index.

[0195] For example, if the current data subset The corresponding sparse object set is: , when the target compression format is CSC, the non-zero element set is: V CSC .

[0196] In the embodiments of the present disclosure, the non-zero element set is determined in different ways through different target compression formats, thereby improving the accuracy and pertinence of the non-zero element set.

[0197] In some embodiments, the index of the non-zero element in the current data subset includes the row index of the non-zero element in the current data subset and the column index of the non-zero element in the current data subset, and the index information set includes at least one of the following: a row index information set and a column index information set; step S362 includes at least one of steps S3621 to S3623, wherein: Step S3621: When the target compression format is the first compression format, determine a row index information set based on the row index of the non-zero element corresponding to each sparse object in the current data subset; determine a column index information set based on the column index of the non-zero element corresponding to each sparse object in the current data subset.

[0198] Here, the first compression format can be COO, the row index information set includes the row index of the non-zero element corresponding to each sparse object in the current data subset, and the column index information set includes the column index of the non-zero element corresponding to each sparse object in the current data subset.

[0199] For example, if the current data subset The corresponding sparse object set is: , then, when the target compression format is COO, the row index information set R COO , column index information set C COO .

[0200] Step S3622: When the target compression format is the second compression format, determine the column index information set based on the column index of the non-zero element corresponding to each sparse object in the current data subset; determine the row index information set based on the index of the non-zero element corresponding to each sparse object in the non-zero element set or the column index information set.

[0201] Here, the second compression format may be CSR, the column index information set includes the column index of the non-zero element corresponding to each sparse object in the current data subset, the row index information set includes the index of the first non-zero element in each row of the current data subset in the non-zero element set or the column index information set, and a special row index. The special row index is used to record the total number of non-zero elements included in the non-zero element set.

[0202] For example, if the current data subset The corresponding sparse object set is: , then, when the target compression format is CSR, the column index information set C CSR , since the non-zero element set V CSR , then, according to the first non-zero elements 1, 2, and 5 in these three rows, CSR or C CSR The index of the row and the total number of non-zero elements contained in the non-zero element set determine the row index information set .

[0203] Step S3623: When the target compression format is the third compression format, determine the row index information set based on the row index of the non-zero element corresponding to each sparse object in the current data subset; determine the column index information set based on the index of the non-zero element corresponding to each sparse object in the non-zero element set or the row index information set.

[0204] Here, the third compression format may be CSC, wherein the row index information set includes the row index of the non-zero element corresponding to each sparse object in the current data subset, and the column index information set includes the index of the first non-zero element in each column of the current data subset in the non-zero element set or the column index information set, as well as a special column index. The special column index is used to record the total number of non-zero elements included in the non-zero element set.

[0205] For example, if the current data subset The corresponding sparse object set is: , then, when the target compression format is CSC, the row index information set R CSC , since the non-zero element set V CSC , then, according to the first non-zero elements 1, 7, 8, 1 in these four columns in V CSC or R CSC The index of the column and the total number of non-zero elements contained in the non-zero element set determine the column index information set C CSC .

[0206] In the embodiments of the present disclosure, different target compression formats use different methods to determine the index information set, thereby improving the accuracy and pertinence of the index information set.

[0207] Figure 4 A schematic diagram of the implementation process of a data processing method provided in the embodiment of the present disclosure Figure 4 ,like Figure 4 As shown, the detection method includes steps S41 to S43, wherein: Step S41: Determine the dynamic threshold corresponding to the current initial data.

[0208] Step S42: Divide the current initial data into at least one current data subset.

[0209] Here, the above-mentioned step S41 and step S42 correspond to the above-mentioned step S11 and step S12 respectively. When implementing, please refer to the specific implementation methods of the above-mentioned step S11 and step S12.

[0210] Step S43: For each current data subset, determine a first thread block corresponding to the current data subset, load the current data subset into a first shared memory corresponding to the first thread block, and determine a non-zero element set of valid data of the current data subset based on at least one thread in the first thread block and a dynamic threshold. The valid data of the current data subset also includes an index information set, which is used to represent index information of each non-zero element in the non-zero element set in the current data subset.

[0211] Here, the first thread block can be any suitable thread block, which is used to at least determine the set of non-zero elements of valid data in the current data subset. In some embodiments, the first thread block can be a thread block in a CPU (Central Processing Unit). During implementation, different current data subsets can correspond to the same or different first thread blocks. The first thread block can be determined in any suitable manner. For example, a thread block in an idle state can be randomly selected as the first thread block. In another example, the first thread block can be determined based on the order in which each thread block is used. In another example, the first thread block can be selected based on a comprehensive evaluation of the load and utilization rate of each thread block.

[0212] The first shared memory can be any suitable shared memory, and different first thread blocks can correspond to the same or different first shared memories. In some embodiments, two different first thread blocks can reuse the same shared memory to improve resource utilization; alternatively, each first thread block corresponds to a first shared memory to reduce interference between data. The first shared memory is also used to temporarily store valid data of the current data subset to reduce latency in global memory access. In some embodiments, the first shared memory can be located in the CPU.

[0213] The threads within the first thread block can collaborate with each other to determine the set of non-zero elements of valid data for the current data subset. In some embodiments, each thread can also determine a set of index information while determining the set of non-zero elements. In implementation, the method for determining the valid data can refer to the specific implementation of step S13 or step S33 described above.

[0214] In the embodiment of the present disclosure, on the one hand, a data subset is pre-loaded into a first shared memory corresponding to a first thread block, so that each thread in the first thread block only accesses the data in the first shared memory for processing. Compared with reading data from a global access memory, this not only reduces the number of global memory accesses, but also reduces the global memory access latency, thereby accelerating the processing speed and achieving the purpose of efficient parallel processing. On the other hand, through the collaboration between the threads in the first thread block, zero-value filtering and data compression are performed on the data subset to obtain valid data of the current data subset, thereby improving the efficiency of data parallel processing. At the same time, since tasks are allocated in an intra-block parallel manner, compared with allocating tasks to threads according to element granularity, this not only reduces the possibility of load imbalance and optimizes thread allocation, but also maximizes resource utilization.

[0215] Based on the above embodiments, the present disclosure further provides a data processing system. Figure 5 A schematic diagram of the structure of a data processing system provided in an embodiment of the present disclosure Figure 1 ,like Figure 5 As shown, the data processing system 50 includes a first data processing unit 51, wherein: The first data processing unit 51 is used to: determine the dynamic threshold corresponding to the current initial data; divide the current initial data into at least one current data subset; and for each current data subset, determine the non-zero element set of valid data of the current data subset based on the dynamic threshold, the valid data of the current data subset also including an index information set, which is used to represent the index information of each non-zero element in the non-zero element set in the current data subset.

[0216] Here, the first data processing unit may be any suitable unit capable of realizing the function. In some embodiments, the first data processing unit may be located in a CPU.

[0217] The current initial data can be any suitable data, such as image data, sparse tensor data, time series data, etc. The dynamic threshold can be any suitable value, and the dynamic threshold is used to identify and filter special values ​​of the initial data. In implementation, the process of determining the dynamic threshold by the first data processing unit can refer to the specific implementation of the aforementioned step S11.

[0218] The current data subset includes at least one row and at least one column. In implementation, the process of the first data processing unit dividing the current initial data can refer to the specific implementation of the aforementioned step S12.

[0219] In some embodiments, the first data processing unit may include a preprocessing module, which is configured to preprocess the current initial data to obtain first data, wherein the preprocessing includes at least one of the following: normalization processing, outlier processing; determining a dynamic threshold corresponding to the current initial data based on data features of the first data; and dividing the current initial data into at least one current data subset.

[0220] Here, the preprocessing module can be any suitable module capable of implementing this function. In implementation, the process of preprocessing the current initial data by the preprocessing module can refer to the specific implementation of the aforementioned step S21. The characteristics of the first data can include, but are not limited to, global characteristics of the first data, local characteristics of the first data, etc. In implementation, the process of determining the dynamic threshold by the preprocessing module can refer to the specific implementation of the aforementioned step S22. The process of dividing the current initial data by the preprocessing module can refer to the specific implementation of the aforementioned step S12.

[0221] non-zero element set The index information set may include but is not limited to a row index information set. , column index information set In implementation, the process of the first data processing unit determining the valid data may refer to the specific implementation of the aforementioned step S13, step S33 or step S43.

[0222] In some embodiments, the first data processing unit is further used to: determine a first thread block corresponding to a current data subset; load the current data subset into a first shared memory corresponding to the first thread block; and determine a set of non-zero elements of valid data of the current data subset based on at least one thread in the first thread block and the dynamic threshold.

[0223] Here, the first thread block can be any suitable thread block, and the first thread block is at least used to determine the non-zero element set of valid data of the current data subset. During implementation, the process of the first data processing unit determining the first thread block can refer to the specific implementation of the aforementioned step S43.

[0224] The first shared memory may be any suitable shared memory, and different first thread blocks may correspond to the same or different first shared memories. In implementation, the process of the first data processing unit determining the first shared memory may refer to the specific implementation of the aforementioned step S43.

[0225] In some embodiments, the first data processing unit may determine the valid data index information set while determining the set of non-zero elements of the valid data. That is, the first data processing unit determines the valid data of the current data subset. The process of determining the valid data of the current data subset by the first data processing unit can be referred to the specific embodiment of the aforementioned step S43.

[0226] For example, assuming the initial tensor , divided into The optimized memory access time is It can be expressed by the following formula (5-1): (5-1); in, Represents the dimension of the data (number of rows); Indicates the number of data points (number of columns); : The number of rows in the data subset; : The number of columns in the data subset; Indicates the time it takes for a storage device (such as a hard disk or storage controller) to find a non-zero element physically or logically; Represents the time required to actually read the non-zero elements and their indices from the storage device.

[0227] It can be seen that compared to directly accessing the entire initial tensor, the time is shortened times.

[0228] In some embodiments, the first data processing unit is further used to: divide the valid data of the current data subset into at least one valid data sub-block; determine, for each valid data sub-block, a second thread block corresponding to the valid data sub-block; load the valid data sub-block into a second shared memory corresponding to the second thread block; use at least one thread in the second thread block to perform parallel operations on at least one non-zero element in the valid data sub-block to obtain an operation result of the at least one non-zero element in the valid data sub-block, and store the operation result of the at least one non-zero element in the valid data sub-block into the second shared memory corresponding to the second thread block.

[0229] Here, the valid data sub-block includes at least part of the valid data. In implementation, the process of the first data processing unit dividing the current data subset can refer to the specific implementation of the aforementioned step S141.

[0230] The second thread block may be any appropriate thread block. During implementation, the process of the first data processing unit determining the second thread block may refer to the specific implementation of the aforementioned step S142.

[0231] The second shared memory may be any suitable shared memory, and different second thread blocks may correspond to the same or different second shared memories. In implementation, the process of the first data processing unit determining the second shared memory may refer to the specific implementation of the aforementioned step S142.

[0232] The operations on non-zero elements may include but are not limited to matrix multiplication, matrix division, torque transposition, matrix inversion, etc. In implementation, the process of the first data processing unit performing operations on non-zero elements may refer to the specific implementation of the aforementioned step S142.

[0233] In some embodiments, the first data processing unit may further include a filtering module configured to determine, for each current data subset, a set of non-zero elements of valid data in the current data subset based on a dynamic threshold.

[0234] Here, the filtering module can be any suitable module that can realize the function. In implementation, the process of the filtering module determining the valid data can refer to the specific implementation of the aforementioned step S13, step S33 or step S43.

[0235] In some embodiments, the filtering module is also used to determine a change index set based on the valid data of the previous current data subset and the current data subset, where the previous current data subset is a data subset obtained by dividing the previous current initial data, and the position of the previous current data subset in the previous current initial data is the same as the position of the current data subset in the current initial data; determine a target data subset based on the change index set and the current data subset; and determine a sparse object set corresponding to the current data subset based on the dynamic threshold and the target data subset.

[0236] Here, the previous current initial data may be initial data in a target round iteration. The target round iteration may be an iteration before the current round iteration, for example, the previous round iteration, the first round iteration, etc.

[0237] The change index set includes at least one change index. During implementation, the process of the filter module determining the change index set can refer to the specific implementation of the aforementioned step S341.

[0238] The target data subset includes at least some elements in the current data subset. During implementation, the process of the filtering module determining the target data subset can refer to the specific implementation of the aforementioned step S342.

[0239] The sparse object set includes at least one sparse object. During implementation, the process of the filtering module determining the sparse object set can refer to the specific implementation of the aforementioned step S343.

[0240] In some embodiments, the filtering module is further configured to determine an index information set of valid data of the current data subset based on the change index set and an index information set in valid data of the previous current data subset.

[0241] Here, the index information set includes a row index information set and a column index information set. During implementation, the process of the filtering module determining the index information set can refer to the specific implementation of the aforementioned step S35.

[0242] In some embodiments, the data processing system also includes a second data processing unit, and the first data processing unit is further used to: adopt a pipeline transmission strategy to transmit the valid data of each current data subset to the second data processing unit respectively; the second data processing unit is used to: for each valid data of the current data subset, reconstruct the valid data of the current data subset to obtain the reconstructed valid data, perform operations on the reconstructed valid data, and obtain the operation results of the reconstructed valid data.

[0243] Here, the second data processing unit may be any suitable unit capable of implementing this function. In some embodiments, the second data processing unit may be located in a GPU.

[0244] The pipeline transmission strategy essentially involves the first data processing unit transmitting the i-th current data subset while the second data processing unit is processing the i-1-th current data subset, thereby concealing the transmission duration. During implementation, the optimized transmission duration is determined based on the transmission and processing times. This optimized transmission duration is the larger of the two.

[0245] In some embodiments, the first data processing unit may further include a data transmission module, wherein the data transmission module is configured to adopt a pipeline transmission strategy to transmit valid data of each current data subset to the second data processing unit.

[0246] In some embodiments, the size of the valid data can be determined based on the sparsity rate of the current data subset and the size of the current data subset, and then the transmission duration of the current data subset can be determined based on the size of the valid data and the bandwidth. The sparsity rate of the current data subset represents the proportion of zero-valued elements in the current data subset. During implementation, the sparsity rate of the current data subset is determined based on the ratio between the number of zero-valued elements in the current data subset and the number of elements in the current data subset. If the sparsity rate is larger, it indicates that most of the elements in the data subset are zero-valued elements; conversely, if the sparsity rate is smaller, it indicates that most of the elements in the data subset are non-zero-valued elements.

[0247] In some implementations, the size of the valid data of the data subset may include, but is not limited to, the values ​​of non-zero elements, index information, metadata, and the like.

[0248] In some embodiments, the size of the valid data of the data subset is It can be defined by the following formula (5-2): (5-2); in, represents the size of the non-zero elements in the data subset, The size of the row index information set representing the data subset, The size of the column index information set representing the subset of data.

[0249] For example, for CSR, if for , for , for , represents the sparsity rate of the data subset, represents the row size of the data subset, Represents the column size of the data subset, then the size of the valid data of the data subset is: .

[0250] In some embodiments, the transmission duration of the single data subset is It can be defined by the following formula (5-3): (5-3); in, Indicates the size of the valid data of the data subset, Indicates the bus bandwidth between the host and GPU. Buses include, but are not limited to, PCTE and NVLink.

[0251] The operation time of a data subset mainly refers to the time it takes for the data subset to be operated on the GPU. The operation time is mainly determined based on the complexity of the data subset and the number of operations that the GPU can perform per second. The complexity of a data subset mainly refers to the number of operations that need to be performed on the data subset. In some embodiments, the complexity of the data subset can be determined based on the size of the data subset and the operation method. The operation method may include but is not limited to matrix multiplication, matrix division, torque transposition, matrix inversion, etc. The number of operations that the GPU can perform per second may include but is not limited to FLOPS (Floating-point Operations Per Second), etc.

[0252] In some embodiments, the computation time for the single data subset is It can be defined by the following formula (5-4): (5-4); in, represents the complexity of a subset of data, Indicates the number of operations that the GPU can perform per second.

[0253] In some embodiments, the second data processing unit may reconstruct the received valid data according to a target compression format. In implementation, the target compression format may be sent by the first data processing unit or read from a configuration file that includes at least the target compression format.

[0254] In some implementations, the received valid data may be reconstructed using the following formula (5-5), namely: (5-5); in, Indicates the row index information of valid data. Indicates column index information of valid data. , The set of non-zero elements representing valid data.

[0255] The operation result of the reconstructed valid data may include the operation result of each non-zero element in the valid data. In some embodiments, multiple threads may be used to perform operations on multiple non-zero elements simultaneously to obtain the operation result of each non-zero element.

[0256] Figure 6 A schematic diagram of the structure of a data processing system provided in an embodiment of the present disclosure Figure 2 ,like Figure 6 As shown, the data processing system includes a first data processing unit 51 and a second data processing unit 52, wherein: The first data processing unit 51 is configured to determine a dynamic threshold corresponding to the current initial data; divide the current initial data into at least one current data subset; determine, for each current data subset, valid data of the current data subset based on the dynamic threshold; and adopt a pipeline transmission strategy to transmit the valid data of each current data subset to the second data processing unit. The second data processing unit 52 is configured to reconstruct the valid data of each current data subset to obtain reconstructed valid data, and perform operations on the reconstructed valid data to obtain operation results of the reconstructed valid data.

[0257] In the embodiment of the present disclosure, on the one hand, the first data processing unit only transmits valid data, reducing the bandwidth occupied by invalid data, thereby significantly shortening the data transmission time and significantly reducing the communication overhead, improving the transmission efficiency and optimizing the system throughput; on the other hand, through the pipeline transmission strategy, data operations are performed while transmitting data, which effectively improves the computing efficiency. At the same time, since only valid data is operated, the computing overhead is reduced and the computing performance is improved.

[0258] In some embodiments, the second data processing unit is further used to: divide the reconstructed valid data into at least one data sub-block; determine, for each data sub-block, a third thread block corresponding to the data sub-block; load the data sub-block into a third shared memory corresponding to the third thread block; use at least one thread in the third thread block to perform parallel operations on at least one non-zero element in the data sub-block to obtain a calculation result of the at least one non-zero element in the data sub-block, and store the calculation result of the at least one non-zero element in the data sub-block into the third shared memory corresponding to the third thread block.

[0259] Here, the data sub-block includes at least part of the reconstructed valid data. The division of the reconstructed valid data can refer to the specific implementation of the aforementioned step S141.

[0260] The third thread block can be any suitable thread block, each of which is used to parallelize operations on corresponding data sub-blocks. In some embodiments, the third thread block can be a thread block in a GPU. In implementation, different data sub-blocks can correspond to the same or different third thread blocks.

[0261] The third thread block may be determined in any suitable manner. In implementation, the third thread block may be determined in the manner described in the aforementioned step S142 .

[0262] The third shared memory can be any suitable shared memory, and different third thread blocks can correspond to the same or different third shared memories. In some embodiments, two different third thread blocks can reuse the same shared memory to improve resource utilization; alternatively, each third thread block corresponds to a third shared memory to reduce interference between data. During implementation, the third shared memory is also used to store the calculation results of at least one non-zero element in the data sub-block. In some embodiments, the third shared memory can be located in the GPU.

[0263] In some embodiments, each non-zero element in the data sub-block can be assigned to a thread in the third thread block for processing, thereby implementing parallel computing using multiple threads in the third thread block. The thread performs operations such as matrix multiplication, matrix division, torque transposition, and matrix inversion on the non-zero elements to obtain a computation result for the non-zero element, and stores the computation result for the non-zero element in the third shared memory.

[0264] In the embodiment of the present disclosure, on the one hand, the data sub-blocks are pre-loaded into the third shared memory corresponding to the third thread block, so that each thread in the third thread block only accesses the data in the third shared memory for processing. Compared with reading data from the global access memory, not only the number of global memory accesses is reduced, but also the global memory access latency is reduced, thereby accelerating the processing speed and achieving the purpose of efficient parallel processing. On the other hand, the efficiency of data parallel processing is improved by having each thread in the third thread block perform operations on multiple non-zero elements at the same time. At the same time, since tasks are allocated in an intra-block parallel manner, compared with allocating tasks to threads according to element granularity, not only the possibility of load imbalance is reduced and thread allocation is optimized, but also resource utilization is maximized.

[0265] In some embodiments, the data processing system also includes an optimization unit, which is used to use the trained intelligent agent to determine the threshold adjustment amplitude based on the state corresponding to the current initial data, where the state corresponding to the current initial data is determined based on the sparsity rate corresponding to the current initial data, the dynamic threshold corresponding to the current initial data, the filtering efficiency corresponding to the current initial data and / or the calculation accuracy corresponding to the current initial data; and determine the dynamic threshold corresponding to the next initial data based on the sum of the threshold adjustment amplitude and the dynamic threshold corresponding to the current initial data.

[0266] Here, the optimization unit may be any appropriate unit capable of realizing this function.

[0267] The agent is used to predict the threshold adjustment range based on the state corresponding to the current initial data. The state corresponding to the current initial data is used to describe the characteristics of the current initial data, system performance, etc. In implementation, the process of the optimization unit determining the threshold adjustment range can be referred to the specific implementation of the aforementioned step S151.

[0268] The dynamic threshold corresponding to the next initial data is used to determine the effective data of the next initial data. In implementation, the process of the optimization unit determining the dynamic threshold corresponding to the next initial data can refer to the specific implementation of the aforementioned step S152.

[0269] In some embodiments, the optimization unit is also used to: determine the state corresponding to the training data and the action corresponding to the training data; determine the reward function corresponding to the training data based on the calculation accuracy corresponding to the training data; and perform reinforcement training on the intelligent agent based on the state corresponding to the training data, the action corresponding to the training data, and the reward function corresponding to the training data to obtain the trained intelligent agent.

[0270] Here, the training data may be any appropriate data. In implementation, the process of the optimization unit determining the state and action corresponding to the training data may refer to the specific implementation of the aforementioned step S161.

[0271] The reward function may be any suitable function. In implementation, the process of determining the reward function by the optimization unit may refer to the specific implementation of the aforementioned step S162.

[0272] Reinforcement learning is a method of learning through interaction between an agent and its environment. During implementation, the process of training the agent by the optimization unit can refer to the specific implementation of the aforementioned step S163.

[0273] Figure 7 A schematic diagram of the structure of a data processing system provided in an embodiment of the present disclosure Figure 3 ,like Figure 7 As shown, the data processing system includes a first data processing unit 51, a second data processing unit 52 and an optimization unit 53. The first data processing unit 51 includes a pre-processing module 511, a filtering module 512 and a data transmission module 513, wherein: The preprocessing module 511 is used to clean the original tensor to obtain an initial tensor (corresponding to the aforementioned initial data); perform outlier processing and standardization on the initial tensor to obtain a target tensor (corresponding to the aforementioned first data); determine the dynamic threshold corresponding to the target tensor based on the characteristics of the target tensor; divide the target tensor into multiple sub-tensors (corresponding to the aforementioned data subsets); wherein, cleaning the original tensor is mainly a process of correcting, completing and optimizing the original tensor to remove noise, missing values, inconsistencies, etc. in the original tensor to ensure the accuracy and consistency of the data.

[0274] The filtering module 512 is used to cluster each sub-tensor using the K-means clustering algorithm to obtain at least one cluster. Then, for each element of each cluster, if the distance between the element and the cluster center of the cluster in which the element is located is greater than the dynamic threshold, the element is treated as a non-zero element and the CSR is used to store the valid data of the sub-tensor. During implementation, the dynamic threshold is used to dynamically identify and eliminate zero values ​​or redundant data in each sub-tensor, and only retain valid data. The data transmission module 513 is used to transmit the valid data of each sub-tensor to the second data processing unit 52 using a pipeline transmission strategy, which significantly reduces the communication overhead; The second data processing unit 52 is configured to reconstruct each sub-tensor according to the CSR to obtain each reconstructed sub-tensor, and implement task allocation and computation based on MUSA (Moore Threads Unified System Architecture) to obtain computational results for each sub-tensor, thereby optimizing thread allocation and memory access patterns and improving data processing performance. During implementation, the second data processing unit may first divide the reconstructed sub-tensor into multiple data sub-blocks, determine a third thread block corresponding to each data sub-block, load the data sub-block into a third shared memory corresponding to the third thread block, utilize at least one thread in the third thread block to perform parallel computation on at least one non-zero element in the data sub-block, obtain a computational result for the at least one non-zero element, and store the computational result for the at least one non-zero element in the data sub-block in the third shared memory corresponding to the third thread block. The optimization unit 53 is used to use a reinforcement learning algorithm such as DQN or PPO to dynamically predict the threshold adjustment range based on key performance indicators such as filtering efficiency and calculation accuracy through the trained intelligent agent; adjust the next dynamic threshold based on the threshold adjustment range to improve the overall performance of the system.

[0275] Figure 8 A schematic diagram of the implementation process of a data processing method provided in the embodiment of the present disclosure Figure 5 ,like Figure 8 As shown, the detection method includes steps S801 to S808, wherein: Step S801: The first data processing unit performs outlier processing and normalization processing on the input tensor (corresponding to the aforementioned current initial data) to obtain a target tensor (corresponding to the aforementioned first data); Here, the input tensor can be normalized by the min-max normalization method.

[0276] Step S802: The first data processing unit uses CNN to extract features of the target tensor, and generates a dynamic threshold corresponding to the input tensor in combination with the fully connected layer, or uses the dynamic threshold output by the optimization unit as the dynamic threshold corresponding to the input tensor; Step S803: The first data processing unit divides the target tensor into at least one sub-tensor; Step S804: The first data processing unit performs parallel filtering and compression processing on each sub-tensor to obtain valid data of each sub-tensor; Here, filtering primarily removes or ignores zero values ​​in the subtensors, retaining only non-zero values ​​that contribute to the computational result, significantly reducing computational complexity and memory overhead. In some implementations, during multiple iterations, the index set can be changed to identify areas of data change, reducing repetitive operations, improving processing efficiency, and further reducing communication overhead and transmission latency, significantly optimizing system throughput. Compression refers to efficiently compressing the filtered data according to the target compression format to further reduce the data volume.

[0277] Step S805: The first data processing unit adopts a pipeline transmission strategy to transmit valid data of each sub-tensor to the second data processing unit; Step S806: The second data processing unit reconstructs the received valid data to obtain reconstructed valid data, and divides the reconstructed valid data into multiple data sub-blocks; Step S807: The second data processing unit uses multiple threads in the thread block corresponding to the data sub-block to perform parallel operations on multiple non-zero elements at the same time, and stores the operation results in the shared memory corresponding to the thread block; Step S808: The optimization unit uses the trained intelligent agent to determine the threshold adjustment amplitude according to the state corresponding to the input tensor, and determines the dynamic threshold corresponding to the next input tensor according to the threshold adjustment amplitude and the dynamic threshold corresponding to the input tensor.

[0278] Here, the optimization unit can determine the threshold adjustment range based on performance indicators such as filtering efficiency, calculation accuracy, calculation resource utilization, calculation delay, etc. During implementation, each performance indicator can be selected based on the parameters in the reward function used during the agent training process.

[0279] Below is the input tensor The specific processing process of the data processing method disclosed herein is described as follows: (1) Input tensor Perform outlier processing and standardization; The mean of the input tensor and standard deviation They are:

[0280]

[0281] Since each element of the input tensor is within the threshold range Therefore, there are no outliers in this input tensor.

[0282] Normalize the input tensor to get the target tensor for .

[0283] (2) Using dynamic threshold to adjust the input tensor Filter to get the input tensor Sparse object set for: .

[0284] (3) Convert sparse object sets into CSR The input tensor The set of non-zero elements of for ; The input tensor Row index information set for ; The input tensor Column index information set for ; The compressed data size is 12. Compared with the size of the original tensor (3*3), the compression ratio is about 1.33 (12 / 9).

[0285] (4) Through the PCIE channel, the input tensor The effective data is transferred to the GPU; (5) GPU uses MUSA to process the input tensor The effective data is used for parallel operation, and the shared memory is used to store the operation results of each thread block, which significantly improves the operation efficiency.

[0286] In the embodiment of the present disclosure, on the one hand, the effective data of the data subset is determined by a dynamic threshold determined in real time, which reduces the possibility of insufficient filtering or excessive filtering compared to the use of fixed thresholds or static rules for identification and filtering in related technologies, thereby not only improving the subsequent calculation accuracy but also improving resource utilization, and being able to adapt to changes in different data distributions and task requirements, improving the compatibility and adaptability of data processing. At the same time, compared to the full amount of calculation in related technologies, since the present disclosure only calculates on the effective data, it not only improves the utilization of calculation resources but also improves the calculation efficiency, and also improves the real-time performance of data calculation; on the other hand, the effective data of the data subset only records each Compared with recording the global index of each non-zero value in the initial data, the local index of the non-zero value in the data subset, first, because each data subset only maintains the internal index, the index maintenance cost is reduced; second, in the subsequent search, since the search is only performed in the local range, the search is simplified while also reducing the difficulty and complexity of the search; finally, in the subsequent memory access, since the access can be performed only in the local memory, unnecessary global memory access is reduced, and the locality of data access is improved while also reducing storage and computing costs; on the other hand, since the data processing method can be used to process image data, sparse tensors, time series data, etc., it has good versatility and expands the application scope of the data processing method.

[0287] Based on the above embodiments, the present disclosure provides a data processing device. Figure 9 A schematic diagram of the structure of a data processing device provided in an embodiment of the present disclosure is shown in FIG. Figure 9 As shown, the data processing device 90 includes a first determination module 91, a division module 92 and a second determination module 93, wherein: The first determining module 91 is configured to determine a dynamic threshold corresponding to the current initial data; The partitioning module 92 is configured to partition the current initial data into at least one current data subset; The second determination module 93 is used to determine, for each current data subset, a set of non-zero elements of the valid data of the current data subset based on a dynamic threshold. The valid data of the current data subset also includes an index information set, which is used to characterize the index information of each non-zero element in the non-zero element set in the current data subset.

[0288] In some embodiments, the first determination module 91 is further used to: preprocess the current initial data to obtain first data, where the preprocessing includes at least one of the following: standardization processing and outlier processing; and determine the dynamic threshold corresponding to the current initial data based on the data characteristics of the first data.

[0289] In some embodiments, the second determination module 93 is further used to: determine the sparse object set corresponding to the current data subset based on a dynamic threshold; and determine the non-zero element set of valid data of the current data subset based on at least one sparse object in the sparse object set corresponding to the current data subset.

[0290] In some embodiments, the sparse object set corresponding to the current data subset includes at least one sparse object; the second determination module 93 is also used to: perform clustering processing on the current data subset to obtain at least one cluster cluster; for each element in each cluster cluster, determine the distance between the element and the cluster center of the cluster cluster where the element is located, and when the distance is greater than a dynamic threshold, treat the element as a non-zero element corresponding to a sparse object, and the sparse object includes the corresponding non-zero element and the index of the non-zero element in the current data subset.

[0291] In some embodiments, the second determination module 93 is also used to: determine the change index set based on the valid data of the previous current data subset and the current data subset, the previous current data subset is a data subset after the previous current initial data is divided, and the position of the previous current data subset in the previous current initial data is the same as the position of the current data subset in the current initial data; determine the target data subset based on the change index set and the current data subset; determine the sparse object set corresponding to the current data subset based on the dynamic threshold and the target data subset.

[0292] In some embodiments, the second determining module 93 is further configured to determine an index information set of valid data of the current data subset based on the change index set and an index information set in valid data of the previous current data subset.

[0293] In some embodiments, the second determination module 93 is further used to: determine a non-zero element set based on the non-zero elements corresponding to each sparse object and the target compression format; and determine an index information set based on the index of the non-zero elements corresponding to each sparse object in the current data subset.

[0294] In some embodiments, the index of the non-zero element in the current data subset includes the row index of the non-zero element in the current data subset and the column index of the non-zero element in the current data subset; the second determination module 93 is also used to: when the target compression format is the first compression format or the second compression format, determine the non-zero element set based on the row index of the non-zero element corresponding to each sparse object in the current data subset; when the target compression format is the third compression format, determine the non-zero element set based on the column index of the non-zero element corresponding to each sparse object in the current data subset.

[0295] In some embodiments, the index of the non-zero element in the current data subset includes the row index of the non-zero element in the current data subset and the column index of the non-zero element in the current data subset, and the index information set includes at least one of the following: a row index information set and a column index information set; the second determination module 93 is further used to: when the target compression format is the first compression format, determine the row index information set based on the row index of the non-zero element corresponding to each sparse object in the current data subset; determine the column index information set based on the column index of the non-zero element corresponding to each sparse object in the current data subset; in the target compression format, determine the row index information set based on the row index of the non-zero element corresponding to each sparse object in the current data subset; in the target compression format, determine the column index information set based on the row index of the non-zero element corresponding to each sparse object in the current data subset; in the target compression format, determine the row ... When the target compression format is the second compression format, the column index information set is determined based on the column index of the non-zero element corresponding to each sparse object in the current data subset; the row index information set is determined based on the index of the non-zero element corresponding to each sparse object in the non-zero element set or the column index information set; when the target compression format is the third compression format, the row index information set is determined based on the row index of the non-zero element corresponding to each sparse object in the current data subset; the column index information set is determined based on the index of the non-zero element corresponding to each sparse object in the non-zero element set or the row index information set.

[0296] In some embodiments, the second determination module 93 is further used to: determine the first thread block corresponding to the current data subset; load the current data subset into the first shared memory corresponding to the first thread block; and determine the set of non-zero elements of valid data of the current data subset based on at least one thread in the first thread block and a dynamic threshold.

[0297] In some embodiments, the data processing device also includes a third determination module, which is used to: divide the valid data of the current data subset into at least one valid data sub-block; determine the second thread block corresponding to the valid data sub-block for each valid data sub-block; load the valid data sub-block into the second shared memory corresponding to the second thread block; use at least one thread in the second thread block to perform parallel operations on at least one non-zero element in the valid data sub-block to obtain the operation result of at least one non-zero element in the valid data sub-block, and store the operation result of at least one non-zero element in the valid data sub-block in the second shared memory corresponding to the second thread block.

[0298] In some embodiments, the data processing device also includes a fourth determination module, which is used to: use the trained intelligent agent to determine the threshold adjustment amplitude based on the state corresponding to the current initial data, the state corresponding to the current initial data is determined based on at least one of the sparsity rate corresponding to the current initial data, the dynamic threshold corresponding to the current initial data, the filtering efficiency corresponding to the current initial data, and the calculation accuracy corresponding to the current initial data; determine the dynamic threshold corresponding to the next initial data based on the sum of the threshold adjustment amplitude and the dynamic threshold corresponding to the current initial data.

[0299] In some embodiments, the data processing device also includes a fifth determination module, which is used to: determine the state corresponding to the training data and the action corresponding to the training data; determine the reward function corresponding to the training data based on the calculation accuracy corresponding to the training data; and perform reinforcement training on the intelligent agent based on the state corresponding to the training data, the action corresponding to the training data, and the reward function corresponding to the training data to obtain a trained intelligent agent.

[0300] In some embodiments, the fifth determination module is further used to: determine the filtering efficiency corresponding to the training data based on the valid data corresponding to the training data and the training data; determine the calculation accuracy corresponding to the training data based on the calculation result of the valid data corresponding to the training data and the target calculation result corresponding to the training data; determine the reward function corresponding to the training data based on the filtering efficiency corresponding to the training data and the calculation accuracy corresponding to the training data.

[0301] In some embodiments, the fifth determination module is also used to: determine a first difference between the operation result of the valid data corresponding to the training data and the target operation result corresponding to the training data; determine a first ratio based on the first difference and the target operation result corresponding to the training data; and determine the operation accuracy corresponding to the training data based on a second difference between the first ratio and the target value.

[0302] In some embodiments, the fifth determination module is further used to: determine a first product between the filtering efficiency corresponding to the training data and the weight of the filtering efficiency corresponding to the training data; determine a second product based on the calculation accuracy corresponding to the training data and the weight of the calculation accuracy corresponding to the training data; and determine a reward function corresponding to the training data based on a third difference between the first product and the second product.

[0303] In some embodiments, the fifth determination module is further used to: determine the computing resource utilization corresponding to the training data and the computing delay corresponding to the training data; and determine the reward function corresponding to the training data based on the computing resource utilization corresponding to the training data, the computing delay corresponding to the training data, and the computing accuracy corresponding to the training data.

[0304] In some embodiments, the fifth determination module is further used to: determine a third product between the computing resource utilization corresponding to the training data and the weight of the computing resource utilization corresponding to the training data; determine a fourth product between the computing delay corresponding to the training data and the weight of the computing delay corresponding to the training data; determine a fifth product between the computing accuracy corresponding to the training data and the weight of the computing accuracy corresponding to the training data; and determine the reward function corresponding to the training data based on the third product, the fourth product and the fifth product. The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present disclosure, please refer to the description of the method embodiment of the present disclosure for understanding.

[0305] It should be noted that in the embodiments of the present disclosure, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present disclosure, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The software product is stored in a storage medium and includes a number of instructions for enabling an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk. In this way, the embodiments of the present disclosure are not limited to any specific combination of hardware and software.

[0306] An embodiment of the present disclosure provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the above method is implemented when the processor executes the computer program.

[0307] The present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements the above method when executed by a processor. The computer-readable storage medium may be transient or non-transient.

[0308] The present disclosure provides a computer program product comprising a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, some or all of the steps of the above-described method are implemented. The computer program product may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK).

[0309] It should be noted that Figure 10 A hardware entity diagram of an electronic device provided in an embodiment of the present disclosure, such as Figure 10 As shown, the hardware entity of the electronic device 100 includes: a processor 101, a communication interface 102 and a memory 103, wherein: The processor 101 generally controls the overall operations of the electronic device 100 .

[0310] The communication interface 102 enables the electronic device to communicate with other terminals or servers through a network.

[0311] Memory 103 is configured to store instructions and applications executable by processor 101. It can also cache data to be processed or processed by processor 101 and various modules in electronic device 100 (e.g., image data, audio data, voice communication data, and video communication data). This can be implemented using flash memory (FLASH) or random access memory (RAM). Data can be transmitted between processor 101, communication interface 102, and memory 103 via bus 104.

[0312] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present disclosure, please refer to the description of the method embodiments of the present disclosure for understanding.

[0313] It should be understood that “one embodiment” or “an embodiment” mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, “in one embodiment” or “in an embodiment” appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The serial numbers of the embodiments of the present disclosure are for description only and do not represent the advantages and disadvantages of the embodiments.

[0314] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0315] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0316] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0317] In addition, all functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0318] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0319] Alternatively, if the above-mentioned integrated units of the present disclosure are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the relevant technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for enabling an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present disclosure. The aforementioned storage medium includes: various media that can store program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0320] The above is only an embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present disclosure, and they should all be covered by the protection scope of the present disclosure.

Claims

1. A data processing method, characterized in that: The method comprises: Determine the dynamic threshold corresponding to the current initial data; Dividing the current initial data into at least one current data subset; For each current data subset, based on the dynamic threshold, a non-zero element set of the valid data of the current data subset is determined, and the valid data of the current data subset also includes an index information set, which is used to characterize the index information of each non-zero element in the non-zero element set in the current data subset.

2. The data processing method according to claim 1, wherein: Determining the dynamic threshold corresponding to the current initial data includes: Preprocessing the current initial data to obtain first data, wherein the preprocessing includes at least one of the following: standardization processing and outlier processing; Based on the data characteristics of the first data, a dynamic threshold corresponding to the current initial data is determined.

3. The data processing method according to claim 1, wherein: Determining a non-zero element set of valid data of the current data subset based on the dynamic threshold comprises: Based on the dynamic threshold, determining a sparse object set corresponding to the current data subset; Based on at least one sparse object in the sparse object set corresponding to the current data subset, a non-zero element set of valid data of the current data subset is determined.

4. The data processing method according to claim 3, wherein: The sparse object set corresponding to the current data subset includes at least one sparse object; The determining, based on the dynamic threshold, a sparse object set corresponding to the current data subset includes: Performing clustering processing on the current data subset to obtain at least one cluster; For each element in each cluster, determine the distance between the element and the cluster center of the cluster where the element is located. When the distance is greater than the dynamic threshold, treat the element as a non-zero element corresponding to a sparse object, and the sparse object includes the corresponding non-zero element and the index of the non-zero element in the current data subset.

5. The data processing method according to claim 3, wherein: The determining, based on the dynamic threshold, a sparse object set corresponding to the current data subset includes: Determining a change index set based on valid data of a previous current data subset and the current data subset, wherein the previous current data subset is a data subset obtained by dividing the previous current initial data, and a position of the previous current data subset in the previous current initial data is the same as a position of the current data subset in the current initial data; Determining a target data subset based on the change index set and the current data subset; Based on the dynamic threshold and the target data subset, a sparse object set corresponding to the current data subset is determined.

6. The data processing method according to claim 1, wherein: The data processing method further includes: Based on the change index set and the index information set in the valid data of the previous current data subset, the index information set of the valid data of the current data subset is determined.

7. The data processing method according to claim 3, characterized in that: The determining, based on at least one sparse object in the sparse object set corresponding to the current data subset, a non-zero element set of valid data of the current data subset comprises: determining the non-zero element set based on a non-zero element corresponding to each of the sparse objects and a target compression format; The data processing method further includes: determining an index information set of valid data of the current data subset based on the index of the non-zero element corresponding to each of the sparse objects in the current data subset.

8. The data processing method according to claim 7, characterized in that: The index of the non-zero element in the current data subset includes the row index of the non-zero element in the current data subset and the column index of the non-zero element in the current data subset; The determining the non-zero element set based on the non-zero elements corresponding to each of the sparse objects and the target compression format includes: When the target compression format is the first compression format or the second compression format, determining the non-zero element set based on the row index of the non-zero element corresponding to each of the sparse objects in the current data subset; When the target compression format is the third compression format, the non-zero element set is determined based on the column index of the non-zero element corresponding to each of the sparse objects in the current data subset.

9. The data processing method according to claim 7, characterized in that: The index of the non-zero element in the current data subset includes the row index of the non-zero element in the current data subset and the column index of the non-zero element in the current data subset, and the index information set includes at least one of the following: a row index information set and a column index information set; The determining, based on the index of the non-zero element corresponding to each of the sparse objects in the current data subset, an index information set of valid data of the current data subset includes: When the target compression format is the first compression format, determining the row index information set based on the row index of the non-zero element corresponding to each of the sparse objects in the current data subset; determining the column index information set based on the column index of the non-zero element corresponding to each of the sparse objects in the current data subset; When the target compression format is the second compression format, the column index information set is determined based on the column index of the non-zero element corresponding to each of the sparse objects in the current data subset; and the row index information set is determined based on the index of the non-zero element corresponding to each of the sparse objects in the non-zero element set or the column index information set. When the target compression format is the third compression format, the row index information set is determined based on the row index of the non-zero element corresponding to each of the sparse objects in the current data subset; and the column index information set is determined based on the index of the non-zero element corresponding to each of the sparse objects in the non-zero element set or the row index information set.

10. The data processing method according to claim 1, wherein: Determining a non-zero element set of valid data of the current data subset based on the dynamic threshold comprises: Determining a first thread block corresponding to the current data subset; Loading the current data subset into a first shared memory corresponding to the first thread block; A set of non-zero elements of valid data of the current data subset is determined based on at least one thread in the first thread block and the dynamic threshold.

11. The data processing method according to claim 1, wherein: The data processing method further includes: Dividing the valid data of the current data subset into at least one valid data sub-block; For each valid data sub-block, determine a second thread block corresponding to the valid data sub-block; load the valid data sub-block into a second shared memory corresponding to the second thread block; use at least one thread in the second thread block to perform a parallel operation on at least one non-zero element in the valid data sub-block to obtain a calculation result of the at least one non-zero element in the valid data sub-block, and store the calculation result of the at least one non-zero element in the valid data sub-block into the second shared memory corresponding to the second thread block.

12. The data processing method according to any one of claims 1 to 11, characterized in that: The data processing method further includes: Determining, using the trained agent, a threshold adjustment range based on a state corresponding to the current initial data, wherein the state corresponding to the current initial data is determined based on at least one of a sparsity rate corresponding to the current initial data, a dynamic threshold corresponding to the current initial data, a filtering efficiency corresponding to the current initial data, and a calculation accuracy corresponding to the current initial data; The dynamic threshold corresponding to the next initial data is determined based on the sum of the threshold adjustment amplitude and the dynamic threshold corresponding to the current initial data.

13. The data processing method according to claim 12, characterized in that: The data processing method further includes: Determining a state corresponding to the training data and an action corresponding to the training data; Determining a reward function corresponding to the training data based on a calculation accuracy corresponding to the training data; Based on the state corresponding to the training data, the action corresponding to the training data and the reward function corresponding to the training data, the intelligent agent is subjected to reinforcement training to obtain the trained intelligent agent.

14. The data processing method according to claim 13, wherein: The determining, based on the computational accuracy corresponding to the training data, a reward function corresponding to the training data, includes: determining a filtering efficiency corresponding to the training data based on valid data corresponding to the training data and the training data; Determining a calculation accuracy corresponding to the training data based on a calculation result of valid data corresponding to the training data and a target calculation result corresponding to the training data; A reward function corresponding to the training data is determined based on a filtering efficiency corresponding to the training data and a calculation accuracy corresponding to the training data.

15. The data processing method according to claim 14, characterized in that: The determining the operation accuracy corresponding to the training data based on the operation result of the valid data corresponding to the training data and the target operation result corresponding to the training data includes: Determine a first difference between a calculation result of valid data corresponding to the training data and a target calculation result corresponding to the training data; determining a first ratio based on the first difference and a target operation result corresponding to the training data; Based on a second difference between the first ratio and the target value, an operation accuracy corresponding to the training data is determined.

16. The data processing method according to claim 14, characterized in that: The determining, based on the filtering efficiency corresponding to the training data and the computational accuracy corresponding to the training data, of a reward function corresponding to the training data includes: determining a first product of a filtration efficiency corresponding to the training data and a weight of the filtration efficiency corresponding to the training data; determining a second product based on the calculation accuracy corresponding to the training data and the weight of the calculation accuracy corresponding to the training data; A reward function corresponding to the training data is determined based on a third difference between the first product and the second product.

17. The data processing method according to claim 13, characterized in that: The determining, based on the computational accuracy corresponding to the training data, a reward function corresponding to the training data, includes: Determining a computing resource utilization rate corresponding to the training data and a computing delay corresponding to the training data; A reward function corresponding to the training data is determined based on a computing resource utilization rate corresponding to the training data, a computing delay corresponding to the training data, and a computing accuracy corresponding to the training data.

18. The data processing method according to claim 17, characterized in that: The determining of the reward function corresponding to the training data based on the computing resource utilization corresponding to the training data, the computing delay corresponding to the training data, and the computing accuracy corresponding to the training data includes: determining a third product of the computing resource utilization corresponding to the training data and a weight of the computing resource utilization corresponding to the training data; determining a fourth product between the operation delay corresponding to the training data and the weight of the operation delay corresponding to the training data; determining a fifth product between the calculation accuracy corresponding to the training data and the weight of the calculation accuracy corresponding to the training data; A reward function corresponding to the training data is determined based on the third product, the fourth product, and the fifth product.

19. A data processing system, characterized in that: The device comprises a first data processing unit, wherein: The first data processing unit is used to: determine a dynamic threshold corresponding to the current initial data; divide the current initial data into at least one current data subset; and for each current data subset, determine a non-zero element set of valid data of the current data subset based on the dynamic threshold, wherein the valid data of the current data subset also includes an index information set, and the index information set is used to characterize the index information of each non-zero element in the non-zero element set in the current data subset.

20. The data processing system according to claim 19, wherein: The data processing system further comprises a second data processing unit, wherein: The first data processing unit is further configured to: adopt a pipeline transmission strategy to transmit valid data of each current data subset to the second data processing unit respectively; The second data processing unit is used to: reconstruct the valid data of each current data subset to obtain reconstructed valid data, and perform operations on the reconstructed valid data to obtain operation results of the reconstructed valid data.

21. The data processing system according to claim 20, wherein: The second data processing unit is further configured to: Dividing the reconstructed valid data into at least one data sub-block; For each data sub-block, determining a third thread block corresponding to the data sub-block; loading the data sub-block into a third shared memory corresponding to the third thread block; using at least one thread in the third thread block, performing a parallel operation on at least one non-zero element in the data sub-block to obtain a calculation result of the at least one non-zero element in the data sub-block, and storing the calculation result of the at least one non-zero element in the data sub-block into the third shared memory corresponding to the third thread block.

22. A data processing device, characterized in that: include: A first determination module is used to determine a dynamic threshold corresponding to the current initial data; a partitioning module, configured to partition the current initial data into at least one current data subset; The second determination module is used to determine, for each current data subset, a non-zero element set of valid data of the current data subset based on the dynamic threshold, wherein the valid data of the current data subset also includes an index information set, and the index information set is used to characterize the index information of each non-zero element in the non-zero element set in the current data subset.

23. An electronic device comprising a processor and a memory, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 18 is implemented.

24. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 18 is implemented.

25. A computer program product, characterized in that The computer program product comprises a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, the method according to any one of claims 1 to 18 is implemented.

Images