A data processing method, system, device and medium

Abstract

The present invention discloses a data processing method, system, device and medium, which relates to the field of data processing and is used to solve the problem of memory occupation caused by creating new tensors when converting data formats uniformly. The scheme obtains array information of an array to be processed, including the dimension size, number of dimensions and step length of each dimension of the array to be processed; determines the storage address of each element in the array to be processed according to the array information of the array to be processed; and performs preset processing on the elements in each storage address in the order of a preset data format, and the preset processing is writing or reading. It can be seen that the present invention can directly perform preset processing on the elements in the storage address by understanding the storage method and structure of the array to be processed and locating each element in the memory, without copying to a new memory location, and without creating a new tensor to store the adjusted data, thereby avoiding copying data and additional memory occupation, and being able to more efficiently handle the problem of inconsistent data formats.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data processing method, system, device and medium. Background Art

[0002] With the rapid development of deep learning and artificial intelligence technologies, the demand for high-performance computing capabilities has become increasingly urgent. Using a computing unified device architecture for parallel computing has become the key to improving computing efficiency. However, data in deep learning often exists in the form of multidimensional arrays. Different frameworks may use different data storage formats, such as different orders or dimensions, which leads to different requirements for input and output data formats for operators. In order to meet these requirements, developers often need to rearrange the data, which undoubtedly increases computing complexity and resource consumption.

[0003] The current solution is to uniformly convert the data format during the preprocessing stage. When encountering inconsistent data formats, all data is first converted according to the required data layout. However, in this process, new tensors are created to store the adjusted data, causing the original data to be copied to a new memory location, which in turn causes additional memory usage and performance overhead.

[0004] Therefore, how to efficiently and flexibly handle the needs of different data formats has become an urgent problem to be solved in this field. Summary of the invention

[0005] The purpose of the present invention is to provide a data processing method, system, device and medium. By understanding the storage method and structure of the array to be processed and locating each element in the memory, the elements in the storage address can be directly preset for processing without copying to a new memory location or creating a new tensor to store the adjusted data, thereby avoiding data duplication and additional memory usage, and being able to more efficiently handle the problem of inconsistent data formats.

[0006] In a first aspect, the present invention provides a data processing method, comprising:

[0007] Acquire array information of an array to be processed, wherein the array information includes the dimension size, the number of dimensions, and the step size on each dimension of the array to be processed;

[0008] Determining the storage address of each element in the array to be processed according to the array information of the array to be processed;

[0009] The elements in each of the storage addresses are subjected to a preset processing in the order of a preset data format, wherein the preset processing is writing or reading.

[0010] In one embodiment, determining the storage address of each element in the array to be processed according to the array information of the array to be processed includes:

[0011] Obtaining the index of each element in the array to be processed;

[0012] Determine the memory offset of each element in the array to be processed according to the index and the array information;

[0013] The storage address of each of the elements is determined according to the memory offset of each of the elements and the starting address of the array to be processed.

[0014] In one embodiment, obtaining the index of each element in the array to be processed includes:

[0015] Obtaining the one-dimensional index of each element in the array to be processed;

[0016] Determining the memory offset of each element in the array to be processed according to the index and the array information includes:

[0017] Converting the one-dimensional index of each element into a multi-dimensional index corresponding to the array to be processed according to the array information, wherein the multi-dimensional index of the element is used to indicate the position of the element in the array to be processed;

[0018] The memory offset of each of the elements is determined according to the multidimensional index of each of the elements and the array information.

[0019] In one embodiment, converting the one-dimensional index of each element into a multi-dimensional index corresponding to the array to be processed according to the array information includes:

[0020] Traverse each dimension according to the number of dimensions of the array to be processed;

[0021] In a current dimension, determining a sub-index of the element in the current dimension according to the dimension size of the current dimension and a one-dimensional index of each of the elements;

[0022] Determine a remaining offset of the element in the current dimension according to the dimension size of the current dimension, the sub-index and the one-dimensional index of each of the elements;

[0023] The one-dimensional index is updated to the sub-index, and iterates to the next dimension, and re-enters the current dimension to determine the sub-index of the element in the current dimension according to the dimension size of the current dimension and the one-dimensional index of each element, until all dimensions are traversed;

[0024] The multidimensional index is determined according to the remaining offset of each dimension.

[0025] In one embodiment, determining the memory offset of each element according to the multidimensional index of each element and the array information includes:

[0026] The memory offset of each of the elements is determined according to the multidimensional index of each of the elements and the stride of each dimension.

[0027] In one embodiment, traversing each dimension according to the number of dimensions of the array to be processed includes:

[0028] Get the maximum number of dimensions N, start traversing from dimension i, where N is a positive integer, i is an integer not greater than N-1, and the initial value of i is N-1;

[0029] In a current dimension, determining a sub-index of the element in the current dimension according to the dimension size of the current dimension and a one-dimensional index of each of the elements includes:

[0030] For dimension i, the one-dimensional index is divided integer-wise according to the dimension size of the dimension i to obtain the sub-index of the element corresponding to the one-dimensional index under the dimension i;

[0031] Determining the remaining offset of the element in the current dimension according to the dimension size of the current dimension, the sub-index, and the one-dimensional index of each of the elements includes:

[0032] A difference value obtained by subtracting a product of the sub-index and the dimension size of the dimension i from the one-dimensional index is used as a remaining offset under the dimension i;

[0033] The one-dimensional index is updated to the sub-index, and iterates to the next dimension, and re-enters the current dimension to determine the sub-index of the element in the current dimension according to the dimension size of the current dimension and the one-dimensional index of each element, until all dimensions are traversed, including:

[0034] Assign the sub-index to the one-dimensional index, set i to i-1, and determine whether i-1 is less than 0;

[0035] If i-1 is not less than 0, the step of dividing the one-dimensional index by the dimension size of the dimension i to obtain the sub-index of the element corresponding to the one-dimensional index under the dimension i is entered;

[0036] Determining the multidimensional index according to the remaining offset of each dimension includes:

[0037] If i-1 is less than 0, the step of determining the multidimensional index according to the remaining offset of each dimension is entered.

[0038] In one embodiment, before obtaining array information of the array to be processed, the method further includes:

[0039] Receiving an array to be processed, and determining whether the data type of the array to be processed is the same as the data type of a preset operator;

[0040] If they are different, the type of the array to be processed is converted to meet the data type of the preset operator.

[0041] In one embodiment, before determining the storage address of each element in the array to be processed according to the array information of the array to be processed, the method further includes:

[0042] Predefined storage address calculation function;

[0043] Determining the storage address of each element in the array to be processed according to the array information of the array to be processed includes:

[0044] The storage address of each element in the array to be processed is determined by using the storage address calculation function according to the array information of the array to be processed.

[0045] In one embodiment, before determining the storage address of each element in the array to be processed by using the storage address calculation function according to the array information of the array to be processed, the method further includes:

[0046] Determine whether the array to be processed satisfies the calling condition for calling the storage address calculation function according to the array information;

[0047] If the calling condition is met, the process proceeds to a step of determining the storage address of each element in the array to be processed using the storage address calculation function according to the array information of the array to be processed.

[0048] In one embodiment, after determining whether the array to be processed meets the calling condition for calling the storage address calculation function according to the array information, the method further includes:

[0049] If the calling condition is not met, the preset processing is directly performed on each element in the processing array according to the storage order of the storage addresses.

[0050] In one embodiment, the array to be processed includes an input array and an output array obtained by processing the input array using a preset operator;

[0051] Judging whether the array to be processed meets the calling condition for calling the storage address calculation function according to the array information includes:

[0052] Determining whether the storage addresses of the elements in the input array are continuous according to the array information;

[0053] Determining whether the storage addresses of the elements in the output array are continuous according to the array information;

[0054] If the storage addresses of the elements in the input array are continuous and the storage addresses of the elements in the output array are continuous, it is determined that the calling condition is not satisfied.

[0055] In one embodiment, after determining that the storage addresses of the elements in the input array are continuous and the storage addresses of the elements in the output array are continuous, the method further includes:

[0056] Determining whether the storage addresses of the elements of the input array and the storage addresses of the elements in the output array are continuous;

[0057] If the storage addresses of the elements of the input array and the storage addresses of the elements in the output array are also continuous, it is determined that the calling condition is not satisfied.

[0058] In a second aspect, the present invention provides a data processing system, comprising:

[0059] An acquisition unit, used for acquiring array information of an array to be processed, wherein the array information includes a dimension size, a number of dimensions, and a step size on each dimension of the array to be processed;

[0060] An address determination unit, used to determine the storage address of each element in the array to be processed according to the array information of the array to be processed;

[0061] A data processing unit is used to perform preset processing on the elements in each of the storage addresses in an order of a preset data format, wherein the preset processing is writing or reading.

[0062] In a third aspect, the present invention provides a data processing device, comprising:

[0063] Memory for storing computer programs;

[0064] The processor is used to implement the steps of the data processing method described above when executing a computer program.

[0065] In a fourth aspect, the present invention provides a non-volatile storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above-mentioned data processing method when executed by a processor.

[0066] The present invention provides a data processing method, system, device and medium, which relates to the field of data processing and is used to solve the problem of memory occupation caused by creating new tensors when converting data formats uniformly. The scheme obtains array information of an array to be processed, and the array information includes the dimension size, number of dimensions and step size of each dimension of the array to be processed; determines the storage address of each element in the array to be processed according to the array information of the array to be processed; and performs preset processing on the elements in each storage address in the order of a preset data format, and the preset processing is writing or reading. It can be seen that the present invention can directly perform preset processing on the elements in the storage address by understanding the storage method and structure of the array to be processed and locating each element in the memory, without copying to a new memory location, and without creating a new tensor to store the adjusted data, thereby avoiding copying data and additional memory occupation, and being able to more efficiently handle the problem of inconsistent data formats. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the prior art and the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0068] Figure 1 A flow chart of a data processing method provided by the present invention;

[0069] Figure 2 A flowchart for determining a multidimensional index provided by the present invention;

[0070] Figure 3 A schematic diagram of a data processing system provided by the present invention;

[0071] Figure 4 A schematic diagram of a data processing device provided by the present invention;

[0072] Figure 5 A schematic diagram of a non-volatile storage medium provided by the present invention. DETAILED DESCRIPTION

[0073] The core of the present invention is to provide a data processing method, system, device and medium. By understanding the storage method and structure of the array to be processed and locating each element in the memory, the elements in the storage address can be directly preset without copying them to a new memory location or creating a new tensor to store the adjusted data, thus avoiding data copying and additional memory usage, and being able to more efficiently handle the problem of inconsistent data formats.

[0074] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] In a first aspect, the present invention provides a data processing method, comprising:

[0076] S11: Obtain array information of the array to be processed, where the array information includes the dimension size, the number of dimensions, and the step size on each dimension of the array to be processed.

[0077] This step obtains the array information of the array to be processed. This array information includes the dimension size, number of dimensions, and stride length on each dimension of the array to be processed. The dimension size refers to the number of elements on each dimension, that is, the length of the array on each dimension. For example, for a two-dimensional array, the dimension size is the number of elements in the array in the row and column directions. The number of dimensions refers to the number of dimensions of the array, also known as the rank of the array. For example, a one-dimensional array has one dimension, while a two-dimensional array has two dimensions. The stride length on each dimension refers to the distance required to move an element on a certain dimension. In memory, the elements of an array are stored consecutively, but in a multidimensional array, different strides may be required to jump to the elements of the next dimension. The stride length usually depends on the storage order of the array and the memory model of the programming language.

[0078] By obtaining the array information of the array to be processed, we can understand the layout and organization of the array in memory, and then determine the storage address of the elements. This is crucial for subsequent data processing steps. It can help us efficiently access and process the elements in the array without copying data or performing additional memory operations, thereby improving performance and reducing memory overhead.

[0079] S12: Determine the storage address of each element in the array to be processed according to the array information of the array to be processed.

[0080] In the above steps, the array information of the array to be processed has been obtained, including the dimension size, the number of dimensions and the step size on each dimension. In this step, based on the obtained array information, the storage address of each element in the array to be processed can be calculated. This calculation process needs to consider the dimension and step size information of the array. Specifically, the storage address of each element in the memory can be determined by the index and memory layout of the array. According to the dimension size, the number of dimensions and the step size of the array, the storage address of each element in the memory can be accurately calculated. Different programming languages ​​and memory layout methods will affect the arrangement order of array elements in the memory, and then affect the calculation method of the storage address. Therefore, when determining the storage address, the programming language and memory model used need to be taken into account. By accurately calculating the storage address of each element, the original data can be directly accessed and processed without the need for additional memory copying, thereby reducing memory usage and performance overhead.

[0081] S13: performing preset processing on the elements in each storage address in the order of the preset data format, wherein the preset processing is writing or reading.

[0082] Specifically, for different operators, different preset data formats may be available. After determining the operator, the required data layout or format has been determined, such as the order of data arrangement, data type, array dimension and other information. According to the storage address of each element determined in the above steps, these storage addresses are traversed. For each storage address, according to the preset data format, we perform corresponding processing operations. This may involve reading, writing or other operations of data to ensure that the data meets the preset format requirements. In general, the input array is read, the input array is calculated by the operator, the output array is obtained, and the output array is written. When performing preset processing on the elements in each storage address, it is necessary to ensure that the processing operation is consistent with the requirements of the preset data format to ensure that the final processed data meets the expected format. By processing the elements in the storage address according to the preset data format, the original data can be directly operated without creating a new tensor or copying the data, thereby reducing additional memory usage and performance overhead, and ensuring that the data meets the preset format requirements.

[0083] For example, if you want to read the stored NHWC (height-width-channel-batch) data as NCHW (channel-height-width-batch) data, you can use the steps in the above data processing method to achieve it. Specifically, first obtain the array information of the array to be processed, including the dimension size, the number of dimensions, and the step size on each dimension. For data in NHWC format, the number of dimensions is 4, namely height, width, number of channels, and batch size. For data in NHWC format, the specific position of each element in the memory can be obtained by calculation. In this embodiment, the preset data format is NCHW, that is, the number of channels, height, and width. Therefore, it is necessary to rearrange the data in NHWC format, and adjust the order of the number of channels, height, and width to the order of NCHW. Then directly read the number of channels, read the height, width, and batch data in order from the corresponding storage address, and arrange them as NCHW, without having to read the NHWC data first and then convert it into NCHW data.

[0084] In summary, the data processing method provided in this embodiment relates to the field of data processing, and is used to solve the problem of memory usage caused by creating new tensors when converting data formats uniformly. This solution obtains array information of the array to be processed, and the array information includes the dimension size, number of dimensions and step size of each dimension of the array to be processed; determines the storage address of each element in the array to be processed according to the array information of the array to be processed; and performs preset processing on the elements in each storage address in the order of the preset data format, and the preset processing is writing or reading. It can be seen that the present invention can directly perform preset processing on the elements in the storage address by understanding the storage method and structure of the array to be processed and locating each element in the memory, without copying to a new memory location, and without creating a new tensor to store the adjusted data, thereby avoiding copying data and additional memory usage, and being able to more efficiently handle the problem of inconsistent data formats.

[0085] In one embodiment, the storage address of each element in the array to be processed is determined based on the array information of the array to be processed, including: obtaining the index of each element in the array to be processed; determining the memory offset of each element in the array to be processed based on the index and the array information; and determining the storage address of each element based on the memory offset of each element and the starting address of the array to be processed.

[0086] This embodiment describes how to determine the storage address of each element in the array according to the array information of the array to be processed. Specifically, the process involves obtaining the index of the element, calculating the memory offset, and finally determining the storage address of the element. Specifically, in a multidimensional array, each element can be uniquely identified by a set of indexes. For example, in a two-dimensional array, an element can be located by a row index and a column index. Similarly, in an array of higher dimensions, more index values ​​are needed to locate the elements. Therefore, it is first necessary to obtain the index value of each element.

[0087] In a contiguous array, the offset of each element relative to the start of the array can be calculated using the index and the stride on the dimension. The stride is the spacing between elements in each dimension of the array. For example, in a two-dimensional array, the row stride is usually the number of columns multiplied by the size of each element, while the column stride is the size of each element. For higher-dimensional arrays, this calculation process will be more complicated and involve more dimensions and stride information. But the basic idea is the same, using the index and stride to calculate the offset of each element relative to the start of the array.

[0088] Once the memory offset of each element is calculated, the storage address of the element can be obtained by adding it to the starting address of the array. The starting address of the array is usually the base address of the array in memory. In other words, the storage address of the element = the starting address of the array + the memory offset of the element. In this way, the storage address of each element in the array can be determined by the index, array information and starting address.

[0089] In summary, the process of calculating the element storage address through index and array information described in this embodiment is crucial for understanding the layout of the array in the memory and performing efficient data processing (such as read and write operations).

[0090] In one embodiment, obtaining the index of each element in the array to be processed includes: obtaining the one-dimensional index of each element in the array to be processed; determining the memory offset of each element in the array to be processed according to the index and array information, including: converting the one-dimensional index of each element into a multidimensional index corresponding to the array to be processed according to the array information, the multidimensional index of the element is used to indicate the position of the element in the array to be processed; determining the memory offset of each element according to the multidimensional index and array information of each element.

[0091] This embodiment describes a specific method for obtaining the index of each element in the array to be processed and determining the memory offset of each element based on the index and array information. Specifically, each element in the array to be processed has a unique index for identifying its position in the array. In a one-dimensional array, the index can be simply expressed as the position of the element in the array; in a multidimensional array, the index needs to be expressed as the position of the element in each dimension. For a one-dimensional array, the index of the element directly corresponds to the position of the element in the memory, so the memory offset can be calculated directly from the index. For a multidimensional array, the multidimensional index of the element needs to be converted to the position in the memory based on the array information. This involves the size and step size of each dimension. For example, for a two-dimensional array, it is necessary to know the number of elements in each row and the size of each element, as well as the span between each element (i.e., the number of bytes per row) to calculate the memory offset of the element. For a multidimensional array, the index of the element is usually given in one-dimensional form, so it is necessary to convert the one-dimensional index into a multidimensional index in order to determine the position of the element in each dimension. This can be achieved through the dimension size and step size in the array information. Once the position of the element in each dimension is obtained, the memory offset of the element can be calculated based on the size and step size of each dimension in the array information. The memory offset represents the exact location of the element in memory and is used to determine the storage address of the element. Through the above steps, the storage address of each element in the one-dimensional or multi-dimensional array to be processed can be accurately determined.

[0092] In one embodiment, the one-dimensional index of each element is converted into a multidimensional index corresponding to the array to be processed according to the array information, including: traversing each dimension according to the number of dimensions of the array to be processed; in the current dimension, determining the sub-index of the element in the current dimension according to the dimension size of the current dimension and the one-dimensional index of each element; determining the remaining offset of the element in the current dimension according to the dimension size of the current dimension, the sub-index and the one-dimensional index of each element; updating the one-dimensional index to the sub-index, iterating to the next dimension, re-entering the step of determining the sub-index of the element in the current dimension according to the dimension size of the current dimension and the one-dimensional index of each element in the current dimension, until all dimensions are traversed; determining the multidimensional index according to the remaining offset of each dimension.

[0093] The present embodiment describes the process of how to convert the one-dimensional index of each element into a multidimensional index corresponding to the array to be processed. Specifically, in a multidimensional array, each element has a unique multidimensional index to identify the position of the element in the array. However, in actual programming and data processing, one-dimensional index may be more commonly used to represent the position of the element because it is more intuitive and easy to calculate. The steps in the present embodiment can convert the one-dimensional index back to a multidimensional index to more accurately understand and operate the multidimensional array.

[0094] Specifically, first, you need to know the number of dimensions of the array to be processed, and then traverse each dimension in turn. This step is to ensure that the index can be converted in the order of the array dimensions. For the current dimension, according to its dimension size (that is, the number of elements under this dimension) and the one-dimensional index of the element, calculate the sub-index of the element under this dimension, and this sub-index represents the specific position of the element under the current dimension. In addition to the sub-index, the remaining offset also needs to be calculated. The remaining offset represents the remaining part of the one-dimensional index after removing the current dimension information. After completing the processing of the current dimension, the one-dimensional index is updated to the sub-index under the next dimension, and the processing of the next dimension is entered. This process will be repeated until all dimensions are traversed. After all dimensions have been traversed, the multidimensional index of the element is determined according to the remaining offset of each dimension. This multidimensional index is the actual position of the element in the multidimensional array.

[0095] When converting from a one-dimensional index to a multidimensional index, you need to ensure that the range of the one-dimensional index is legal, that is, it should be within the valid index range of the multidimensional array. The sub-index and remaining offset are correctly calculated based on the dimension size and stride information of the array.

[0096] It can be seen that through the above method, the one-dimensional index of each element can be effectively converted into a multi-dimensional index corresponding to the array to be processed, thereby achieving efficient processing of the array.

[0097] In one embodiment, determining the memory offset of each element according to the multidimensional index and array information of each element includes: determining the memory offset of each element according to the multidimensional index of each element and the step size of each dimension.

[0098] This embodiment determines the memory offset of each element based on the multidimensional index and array information of each element. Specifically, each dimension is processed in turn according to the number of dimensions of the array to be processed. The offset of the element on each dimension is calculated based on the multidimensional index of each element and the step size of each dimension. This offset represents the offset of the storage position of the element on this dimension relative to the starting position under the current dimension. The offsets on each dimension are accumulated to obtain the memory offset of each element relative to the starting address of the array to be processed. Through the above steps, the memory offset of each element can be determined, so that it can be accurately located and processed. This method can ensure that each element of the array to be processed is processed in an orderly manner according to a preset format, and ensure the accuracy and efficiency of data processing.

[0099] In one embodiment, each dimension is traversed according to the number of dimensions of the array to be processed, including: obtaining the maximum number of dimensions N, starting the traversal from dimension i, where N is a positive integer, i is an integer not greater than N-1, and the initial value of i is N-1; in the current dimension, determining the sub-index of the element in the current dimension according to the dimension size of the current dimension and the one-dimensional index of each element, including: for dimension i, dividing the one-dimensional index according to the dimension size of dimension i to obtain the sub-index of the element corresponding to the one-dimensional index in dimension i; determining the remaining offset of the element in the current dimension according to the dimension size of the current dimension, the sub-index and the one-dimensional index of each element, including: subtracting the product of the sub-index and the dimension size of dimension i from the one-dimensional index as the difference The remaining offset under dimension i; updating the one-dimensional index to the sub-index, and iterating to the next dimension, re-entering the current dimension, and determining the sub-index of the element under the current dimension according to the dimension size of the current dimension and the one-dimensional index of each element, until all dimensions are traversed, including: assigning the sub-index to the one-dimensional index, and letting i be i-1, and judging whether i-1 is less than 0; if i-1 is not less than 0, then entering the step of dividing the one-dimensional index according to the dimension size of dimension i for dimension i, and obtaining the sub-index of the element corresponding to the one-dimensional index under dimension i; determining the multi-dimensional index according to the remaining offset of each dimension, including: if i-1 is less than 0, then entering the step of determining the multi-dimensional index according to the remaining offset of each dimension.

[0100] like Figure 2As shown, this embodiment describes how to traverse each dimension according to the number of dimensions of the array to be processed so as to convert a one-dimensional index into a multidimensional index. Specifically, it is necessary to know the maximum number of dimensions N of the array to be processed. This information can usually be obtained from the metadata or header information of the array. Start traversing from the last dimension of the array (i.e., dimension N-1, i.e., in this process, i is N-1). This is done because in a row-first or column-first linearization strategy, the last dimension usually corresponds to the least significant bit of the one-dimensional index. For the current dimension i, the one-dimensional index cur is divided by the dimension size sahpe[i] of the dimension, and the quotient obtained is the sub-index div of the element under the current dimension. The integer division operation actually extracts the high-order part of the one-dimensional index, which corresponds to the position under the current dimension. In addition to the sub-index, the remaining offset w needs to be calculated. This offset represents the remaining part after removing the current dimension information from the one-dimensional index cur, which will be used to determine the sub-index in subsequent dimensions. The remaining offset w is obtained by subtracting the difference between the one-dimensional index and the product of the dimension size shape[i] of dimension i. After completing the processing of the current dimension, assign the sub-index div to the one-dimensional index cur, and reduce the dimension i by 1 to enter the processing of the next dimension. This process will be repeated until all dimensions are traversed. When all dimensions are traversed, the sub-index under each dimension is obtained. These sub-indexes are combined to form the multi-dimensional index of the element. At the same time, additional verification or processing can be performed based on the remaining offset to ensure the correctness of the index. idx is the memory offset, and stride[i] is the stride on dimension i.

[0101] After traversing all dimensions, necessary validation or processing should be performed based on the remaining offsets to handle possible exceptions.

[0102] In one embodiment, before obtaining array information of the array to be processed, the method further includes:

[0103] Receive the array to be processed, and determine whether the data type of the array to be processed is the same as the data type of the preset operator;

[0104] If they are different, the type of the array to be processed is converted to meet the data type of the preset operator.

[0105] Specifically, in the present embodiment, the array to be processed is first received as input. This means that the algorithm needs to be able to receive data from the outside and use it as the basis for subsequent processing. Then, the data type of the array to be processed is determined. This is to ensure that the data type of the array to be processed is the same as the data type of the preset operator (i.e., the specific processing operation to be applied to the array to be processed). If the data type of the array to be processed is the same as the data type of the preset operator, subsequent processing can be performed directly. If the data type of the array to be processed is different from the data type of the preset operator, type conversion will be performed. This means that the data type of the array to be processed is converted to a type that matches the data type of the preset operator. The purpose of type conversion is to ensure that subsequent processing can proceed smoothly, because usually, the processing operation requires that the type of input data matches the data type it operates on. It can be seen that the present embodiment ensures that before obtaining the array information of the array to be processed, the data type of the array to be processed is checked and necessary conversion is performed to ensure that subsequent data processing can be performed correctly.

[0106] In one embodiment, before determining the storage address of each element in the array to be processed according to the array information of the array to be processed, it also includes: predefining a storage address calculation function; determining the storage address of each element in the array to be processed according to the array information of the array to be processed, including: determining the storage address of each element in the array to be processed using the storage address calculation function according to the array information of the array to be processed.

[0107] This embodiment provides that before determining the storage address of each element according to the array information of the array to be processed, a storage address calculation function needs to be predefined, and this function is used in subsequent steps to calculate the storage address of each element. Specifically, in data processing, especially when processing multidimensional arrays, determining the storage address of each element is a key step. This usually involves calculating the specific location of each element in the memory based on information such as the size of the array's dimensions, the number of dimensions, and the step length on each dimension. In order to improve computing efficiency and code reusability, a storage address calculation function can be predefined, which can calculate the corresponding storage address based on the input array information.

[0108] Specifically, the purpose of predefining a storage address calculation function is to create a reusable calculation tool that can output the corresponding element storage address based on different array information. This function usually needs to accept array information (such as dimension size, number of dimensions, and step length, etc.) and index as input parameters and return an address value. By predefining this function, you can avoid repeatedly writing the same logic code every time you need to calculate the storage address, thereby improving the reusability and maintainability of the code.

[0109] In one embodiment, before determining the storage address of each element in the array to be processed using a storage address calculation function according to the array information of the array to be processed, it also includes: judging whether the array to be processed meets the calling conditions of calling the storage address calculation function according to the array information; if the calling conditions are met, entering the step of determining the storage address of each element in the array to be processed using a storage address calculation function according to the array information of the array to be processed.

[0110] In the present embodiment, before determining the storage address of each element in the array to be processed, a series of conditional judgments need to be performed first to ensure that it is known that the storage address calculation function needs to be called appropriately. Specifically, the principle is based on considerations for data processing efficiency. In some cases, directly calling the storage address calculation function may not be the most efficient way to determine the storage address of an element. For example, if the array to be processed has certain specific properties or structures, different strategies or methods may need to be adopted to calculate the storage address. Therefore, by introducing the judgment of the calling condition, it can be ensured that the storage address calculation function is called under appropriate circumstances, thereby improving the overall efficiency and security of data processing.

[0111] Specifically, judging whether the array to be processed meets the calling conditions for calling the storage address calculation function based on the array information: the algorithm will judge based on the array information of the array to be processed to determine whether the conditions for calling the storage address calculation function are met. This may involve comparing and calculating information such as the dimension size, number of dimensions, and step length on each dimension of the array to be processed to verify whether the calling conditions are met. The calling conditions can be flexibly set according to the actual application scenarios and requirements. For example, you can check whether the number of dimensions of the array to be processed is within a certain range, or check whether the step length of the array on each dimension meets a certain pattern. These conditions can be customized according to the actual storage method and access mode of the array.

[0112] If the calling condition is met, the step of determining the storage address using the storage address calculation function according to the array information of the array to be processed is entered: If the array to be processed meets the conditions for calling the storage address calculation function, the algorithm will enter the next step, that is, using the predefined storage address calculation function according to the array information of the array to be processed to determine the storage address of each element. This step is very critical in determining the storage address because it determines the basis for subsequent data processing operations.

[0113] In one embodiment, after determining whether the array to be processed meets the calling conditions for calling the storage address calculation function according to the array information, it also includes: if the calling conditions are not met, directly performing preset processing on each element in the array to be processed according to the storage order of the storage addresses.

[0114] When the array to be processed does not meet the calling conditions, it means that the array has some special properties or structures, so that using the predefined storage address calculation function is not the most efficient or direct method to locate the elements. In this case, it may be more efficient and simple to process the elements directly according to the storage address order. For example, if the array to be processed is a one-dimensional array stored continuously, then the storage addresses of its elements are likely to be arranged in linear order. In this case, there is no need to perform complex address calculations, and you only need to traverse the array in ascending order of storage addresses. In addition, even if the array to be processed is a multidimensional array, if its elements are stored continuously in memory and arranged in a fixed order (such as row priority or column priority), then the elements can be processed directly in this order without calculating the exact storage address of each element.

[0115] In one embodiment, the array to be processed includes an input array and an output array obtained by processing the input array using a preset operator; judging whether the array to be processed meets the calling conditions for calling the storage address calculation function according to the array information, including: judging whether the storage addresses of each element in the input array are continuous according to the array information; judging whether the storage addresses of each element in the output array are continuous according to the array information; if the storage addresses of each element in the input array are continuous and the storage addresses of each element in the output array are continuous, it is judged that the non-calling condition is met.

[0116] In this embodiment, the array to be processed not only includes the input array, but also covers the output array obtained after the input array is processed by the preset operator. When the storage addresses of the elements in the input array and the output array in the memory are continuous, that is, they are all arrays stored continuously, it is usually the most direct and efficient way to preset the elements according to the storage order of the storage address. This is because the layout of the array stored continuously in the memory is compact, and the address difference between the elements is fixed, so the entire array can be traversed by simple address increment without complicated address calculation. In this case, it is unnecessary to call the storage address calculation function to determine the storage address of each element, and it may even introduce additional calculation overhead. Therefore, when it is detected that the input array and the output array are both stored continuously, the data processing method will be determined as not meeting the conditions for calling the storage address calculation function, so that the elements are directly preset according to the storage order of the storage address.

[0117] In practical applications, determining whether the storage addresses of elements in the input array and the output array are continuous can be achieved by checking the storage layout and access mode of the array. For example, the memory allocation of the array can be checked to determine whether the elements are stored in a continuous address space. In addition, the storage order and continuity of the elements can be inferred based on information such as the dimension and step size of the array. If both the input array and the output array are stored continuously, the data processing method will directly perform preset processing on the elements according to the storage order of the storage addresses, such as traversing the array, reading or writing data, etc. This can avoid unnecessary address calculations and improve processing efficiency.

[0118] In one embodiment, after determining that the storage addresses of the elements in the input array are continuous and the storage addresses of the elements in the output array are continuous, it also includes: determining whether the storage addresses of the elements in the input array and the storage addresses of the elements in the output array are continuous; if the storage addresses of the elements in the input array and the storage addresses of the elements in the output array are also continuous, it is determined that the calling condition is met.

[0119] It should be noted that even if the input array is stored continuously, the output array may no longer maintain continuity after being processed by certain preset operators. Therefore, when judging whether the calling condition is met, it is necessary to consider the conditions of the input array and the output array at the same time. Therefore, in this embodiment, in addition to judging whether the storage addresses of the elements in the input array and the output array are continuous, a step is added, namely, judging whether the storage addresses of the elements in the input array and the storage addresses of the elements in the output array are continuous. This additional judgment condition is based on an assumption: in certain specific data processing scenarios, the input array and the output array are not only stored continuously in their respective internal parts, but also their layout in the memory is adjacent, that is, the storage address of the last element of the input array is immediately followed by the storage address of the first element of the output array.

[0120] Specifically, when the storage addresses of the input array and the output array are continuous in memory, it means that the entire data processing flow (from input to output) can be completed in a continuous memory area without additional memory access or data copying. This continuity not only improves the speed of data access, but also reduces the complexity of memory management. Therefore, when judging the conditions for calling the storage address calculation function, in addition to considering the continuity within the input array and the output array, the continuity between them also needs to be considered. If the storage addresses of the input array and the output array in memory are continuous, then the entire data processing process can be completed through a continuous memory access, which is usually the optimal situation.

[0121] In practical applications, the continuity of storage addresses between the input array and the output array can be determined by comparing their starting addresses and sizes. Specifically, it can be checked whether the storage address of the last element of the input array is immediately followed by the storage address of the first element of the output array, and whether the total size of the two arrays is equal to the size of the continuous memory area they occupy.

[0122] In summary, this embodiment determines whether the conditions for calling the storage address calculation function are met by judging the continuity of the storage addresses within the input array and the output array and between them, which can further improve the data processing efficiency while meeting the continuous storage characteristics.

[0123] In one embodiment, judging whether the storage addresses of the elements in the input array are continuous according to the array information includes: obtaining the dimension size and the step size of the input array, and determining the step size of each element in the input array according to the dimension size and the step size; judging whether the step size of each element in the input array is equal to the expected step size corresponding to itself, and if they are equal, then judging that the storage addresses of the elements in the input array are continuous;

[0124] In one embodiment, judging whether the storage addresses of the elements in the output array are continuous according to the array information includes: obtaining the dimension size and the step size of the output array, and determining the step size of each element in the output array according to the dimension size and the step size; judging whether the step size of each element in the output array is equal to the expected step size corresponding to itself, and if they are equal, then judging that the storage addresses of the elements in the output array are continuous;

[0125] In one embodiment, determining whether the storage addresses of elements of an input array and the storage addresses of elements in an output array are continuous includes: obtaining the dimension size and stride of the input array, and the dimension size and stride of the output array; determining whether the dimension size of the input array and the dimension size of the output array are equal, and determining whether the stride of the input array and the stride of the output array are equal; if the dimension size and the stride of the input array and the output array are equal, determining whether the stride of any element in the input array and / or the stride of any element in the output array are equal to the expected step size, and if they are equal to the expected step size, determining that the storage addresses of the elements of the input array and the storage addresses of the elements in the output array are continuous.

[0126] In this embodiment, how to determine whether the storage addresses of each element in the input array and the output array are continuous based on the array information, and how to further determine whether the element storage addresses of the input array and the output array are continuous is described in detail. This method is based on the dimension size and step size of the array to perform analysis to determine whether a specific storage address continuity condition is met.

[0127] First, the dimension size and stride information of the input array are obtained. The dimension size indicates the length of the array in each dimension, while the stride describes how the array is stored across multiple dimensions in the memory. Based on the dimension size and stride, the stride of each element in the input array can be calculated. The stride is actually the offset between the storage locations of the array elements in the memory. Then, the calculated stride of each element is compared with the expected stride. The expected stride is usually predetermined based on the dimension and storage layout of the array. If the stride of each element is equal to the expected stride, then it can be considered that the storage addresses of the elements of the input array are continuous. For the output array, the judgment method is the same as that of the input array, and the present invention will not be repeated here.

[0128] In order to determine whether the storage addresses of the input array and the output array are continuous, it is necessary to obtain the dimension sizes and strides of the input array and the output array at the same time. First, compare whether the dimension sizes of the two arrays are equal. Then, compare whether their strides are equal. If both the dimension sizes and strides are equal, then the two arrays are similar in structure. Finally, check whether the stride of any element in the input array or the output array is equal to the expected stride corresponding to the element. If they are equal, then it can be considered that the storage addresses of the elements of the input array and the output array are continuous.

[0129] Through this method, before calling the storage address calculation function, it is possible to effectively determine whether the storage addresses of the input array and the output array are continuous, thereby optimizing the data processing flow and improving processing efficiency.

[0130] Second, as Figure 3 As shown, the present invention provides a data processing system, comprising:

[0131] An acquisition unit 31 is used to acquire array information of the array to be processed, where the array information includes the dimension size, the number of dimensions and the step size of each dimension of the array to be processed;

[0132] An address determination unit 32, used to determine the storage address of each element in the array to be processed according to the array information of the array to be processed;

[0133] The data processing unit 33 is used to perform preset processing on the elements in each storage address according to the sequence of the preset data format, and the preset processing is writing or reading.

[0134] In one embodiment, the address determination unit includes:

[0135] A first acquisition unit is used to obtain the index of each element in the array to be processed;

[0136] A first determining unit, used to determine the memory offset of each element in the array to be processed according to the index and array information;

[0137] The second determining unit is used to determine the storage address of each element according to the memory offset of each element and the starting address of the array to be processed.

[0138] In one embodiment, the first acquisition unit is specifically used to obtain a one-dimensional index of each element in the array to be processed;

[0139] The first determining unit includes:

[0140] A conversion unit, used for converting the one-dimensional index of each element into a multi-dimensional index corresponding to the array to be processed according to the array information, wherein the multi-dimensional index of the element is used to indicate the position of the element in the array to be processed;

[0141] The third determining unit is used to determine the memory offset of each element according to the multidimensional index and array information of each element.

[0142] In one embodiment, the conversion unit includes:

[0143] A traversal unit, used to traverse each dimension according to the number of dimensions of the array to be processed;

[0144] A fourth determining unit, configured to determine, in a current dimension, a sub-index of an element in the current dimension according to a dimension size of the current dimension and a one-dimensional index of each element;

[0145] a fifth determining unit, configured to determine a remaining offset of an element in the current dimension according to the dimension size of the current dimension, the sub-index, and the one-dimensional index of each element;

[0146] a sixth determining unit, configured to update the one-dimensional index to a sub-index, iterate to the next dimension, re-enter the current dimension, and determine the sub-index of the element in the current dimension according to the dimension size of the current dimension and the one-dimensional index of each element, until all dimensions are traversed;

[0147] The seventh determining unit is used to determine the multidimensional index according to the remaining offset of each dimension.

[0148] In one embodiment, the third determining unit is specifically configured to determine the memory offset of each element according to the multidimensional index of each element and the step length of each dimension.

[0149] In one embodiment, the traversal unit is specifically used to obtain the maximum number of dimensions N, and the traversal starts from dimension i, where N is a positive integer, i is an integer not greater than N-1, and the initial value of i is N-1;

[0150] A fourth determining unit is specifically configured to divide the one-dimensional index by an integer according to the dimension size of the dimension i, so as to obtain a sub-index of the element corresponding to the one-dimensional index under the dimension i;

[0151] a fifth determining unit, specifically configured to use a difference value obtained by subtracting a product of a sub-index and a dimension size of dimension i from a one-dimensional index as a remaining offset under dimension i;

[0152] The sixth determination unit is specifically used to assign the sub-index to the one-dimensional index, and let i be i-1, and determine whether i-1 is less than 0; if i-1 is not less than 0, then enter the step of dividing the one-dimensional index by the dimension size of dimension i for dimension i, and obtaining the sub-index of the element corresponding to the one-dimensional index under dimension i;

[0153] The seventh determining unit is specifically configured to, if i-1 is less than 0, enter the step of determining a multidimensional index according to the remaining offset of each dimension.

[0154] In one embodiment, it also includes: a type conversion unit, which is used to receive the array to be processed before obtaining the array information of the array to be processed, and determine whether the data type of the array to be processed is the same as the data type of the preset operator; if different, the type of the array to be processed is converted to meet the data type of the preset operator.

[0155] In one embodiment, it also includes:

[0156] A definition unit, used for predefining a storage address calculation function before determining the storage address of each element in the array to be processed according to the array information of the array to be processed;

[0157] The address determination unit is specifically used to determine the storage address of each element in the array to be processed by using a storage address calculation function according to the array information of the array to be processed.

[0158] In one embodiment, it also includes:

[0159] A judging unit, used for judging whether the array to be processed satisfies a calling condition for calling the storage address calculation function according to the array information of the array to be processed before determining the storage address of each element in the array to be processed by using the storage address calculation function according to the array information of the array to be processed;

[0160] The first execution unit is used to enter the step of determining the storage address of each element in the array to be processed by using a storage address calculation function according to the array information of the array to be processed if the calling condition is met.

[0161] In one embodiment, it also includes:

[0162] The second execution unit is used to directly perform preset processing on each element in the processing array according to the storage order of the storage addresses if the calling condition is not met.

[0163] In one embodiment, the array to be processed includes an input array and an output array obtained by processing the input array using a preset operator;

[0164] The judgment unit comprises:

[0165] The first judgment unit is used to judge whether the storage addresses of each element in the input array are continuous according to the array information; judge whether the storage addresses of each element in the output array are continuous according to the array information; if the storage addresses of each element in the input array are continuous and the storage addresses of each element in the output array are continuous, it is determined that the calling condition is met.

[0166] In one embodiment, it also includes:

[0167] The second judgment unit is used to judge whether the storage addresses of the elements of the input array and the storage addresses of the elements in the output array are continuous after judging that the storage addresses of the elements in the input array are continuous and the storage addresses of the elements in the output array are continuous; if the storage addresses of the elements of the input array and the storage addresses of the elements in the output array are also continuous, it is judged that the calling condition is met.

[0168] For the introduction of the data processing system, please refer to the above embodiments, and the present invention will not be described in detail here.

[0169] Thirdly, Figure 4 As shown, the present invention provides a data processing device, comprising:

[0170] A memory 41, used for storing computer programs;

[0171] The processor 42 is used to implement the steps of the above-mentioned data processing method when executing the computer program.

[0172] For the introduction of the data processing device, please refer to the above embodiments, and the present invention will not be described in detail here.

[0173] Fourthly, Figure 5 As shown, the present invention provides a non-volatile storage medium 51, on which a computer program 52 is stored. When the computer program 52 is executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0174] For the introduction of the non-volatile storage medium 51 , please refer to the above embodiment, and the present invention will not go into details here.

[0175] In a fifth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which implements the steps of the above-mentioned data processing method when executed by a processor.

[0176] For the introduction of the computer program product, please refer to the above embodiments, and the present invention will not be described in detail here.

[0177] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0178] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized in that: include: Acquire array information of an array to be processed, wherein the array information includes the dimension size, the number of dimensions, and the step size on each dimension of the array to be processed; Determining the storage address of each element in the array to be processed according to the array information of the array to be processed; Performing a preset process on the elements in each of the storage addresses in the order of a preset data format, wherein the preset process is writing or reading; Before determining the storage address of each element in the array to be processed according to the array information of the array to be processed, the method further includes: Predefined storage address calculation function; Determine whether the storage addresses of the elements in the input array are continuous according to the array information; determine whether the storage addresses of the elements in the output array are continuous according to the array information; determine whether the storage addresses of the elements in the input array and the storage addresses of the elements in the output array are continuous; the array to be processed includes the input array and the output array obtained by processing the input array using a preset operator; Determining the storage address of each element in the array to be processed according to the array information of the array to be processed includes: Determine whether the array to be processed satisfies the calling condition for calling the storage address calculation function according to the array information; If the calling condition is met, the storage address of each element in the array to be processed is determined by using the storage address calculation function according to the array information of the array to be processed; If the storage addresses of the elements in the input array are continuous and the storage addresses of the elements in the output array are continuous, and the storage addresses of the elements in the input array and the storage addresses of the elements in the output array are also continuous, it is determined that the calling condition is not met, and the preset processing is directly performed on each element in the processing array according to the storage order of the storage addresses; Determining whether the storage addresses of the elements in the input array are continuous according to the array information includes: obtaining the dimension size and the step size of the input array, and determining the step size of each element in the input array according to the dimension size and the step size of the input array; determining whether the step size of each element in the input array is equal to the expected step size corresponding to itself, and if they are equal, determining that the storage addresses of the elements in the input array are continuous; Determining whether the storage addresses of the elements in the output array are continuous according to the array information includes: obtaining the dimension size and the step size of the output array, and determining the step size of each element in the output array according to the dimension size and the step size of the output array; determining whether the step size of each element in the output array is equal to the expected step size corresponding to itself, and if they are equal, determining that the storage addresses of the elements in the output array are continuous; Determining whether the storage addresses of the elements of the input array and the storage addresses of the elements in the output array are continuous includes: determining whether the storage address of the last element of the input array is immediately followed by the storage address of the first element of the output array, and whether the total size of the two arrays is equal to the size of the continuous memory area they occupy.

2. The data processing method according to claim 1, characterized in that: Determining the storage address of each element in the array to be processed according to the array information of the array to be processed includes: Obtaining the index of each element in the array to be processed; Determine the memory offset of each element in the array to be processed according to the index and the array information; The storage address of each of the elements is determined according to the memory offset of each of the elements and the starting address of the array to be processed.

3. The data processing method according to claim 2, characterized in that: Get the index of each element in the array to be processed, including: Obtaining the one-dimensional index of each element in the array to be processed; Determining the memory offset of each element in the array to be processed according to the index and the array information includes: Converting the one-dimensional index of each element into a multi-dimensional index corresponding to the array to be processed according to the array information, wherein the multi-dimensional index of the element is used to indicate the position of the element in the array to be processed; The memory offset of each of the elements is determined according to the multidimensional index of each of the elements and the array information.

4. The data processing method according to claim 3, characterized in that: Converting the one-dimensional index of each element into a multi-dimensional index corresponding to the array to be processed according to the array information, including: Traverse each dimension according to the number of dimensions of the array to be processed; In a current dimension, determining a sub-index of the element in the current dimension according to the dimension size of the current dimension and a one-dimensional index of each of the elements; Determine a remaining offset of the element in the current dimension according to the dimension size of the current dimension, the sub-index and the one-dimensional index of each of the elements; The one-dimensional index is updated to the sub-index, and iterates to the next dimension, and re-enters the current dimension to determine the sub-index of the element in the current dimension according to the dimension size of the current dimension and the one-dimensional index of each element, until all dimensions are traversed; The multidimensional index is determined according to the remaining offset of each dimension.

5. The data processing method according to claim 4, characterized in that: Determining the memory offset of each of the elements according to the multidimensional index of each of the elements and the array information includes: The memory offset of each of the elements is determined according to the multidimensional index of each of the elements and the stride of each dimension.

6. The data processing method according to claim 4, characterized in that: Traversing each dimension according to the number of dimensions of the array to be processed, including: Get the maximum number of dimensions N, start traversing from dimension i, where N is a positive integer, i is an integer not greater than N-1, and the initial value of i is N-1; In a current dimension, determining a sub-index of the element in the current dimension according to the dimension size of the current dimension and a one-dimensional index of each of the elements includes: For dimension i, the one-dimensional index is divided integer-wise according to the dimension size of the dimension i to obtain the sub-index of the element corresponding to the one-dimensional index under the dimension i; Determining the remaining offset of the element in the current dimension according to the dimension size of the current dimension, the sub-index, and the one-dimensional index of each of the elements includes: A difference value obtained by subtracting a product of the sub-index and the dimension size of the dimension i from the one-dimensional index is used as a remaining offset under the dimension i; The one-dimensional index is updated to the sub-index, and iterates to the next dimension, and re-enters the current dimension to determine the sub-index of the element in the current dimension according to the dimension size of the current dimension and the one-dimensional index of each element, until all dimensions are traversed, including: Assign the sub-index to the one-dimensional index, set i to i-1, and determine whether i-1 is less than 0; If i-1 is not less than 0, the step of dividing the one-dimensional index by the dimension size of the dimension i to obtain the sub-index of the element corresponding to the one-dimensional index under the dimension i is entered; Determining the multidimensional index according to the remaining offset of each dimension includes: If i-1 is less than 0, the step of determining the multidimensional index according to the remaining offset of each dimension is entered.

7. The data processing method according to claim 1, characterized in that: Before obtaining the array information of the array to be processed, it also includes: Receiving an array to be processed, and determining whether the data type of the array to be processed is the same as the data type of a preset operator; If they are different, the type of the array to be processed is converted to meet the data type of the preset operator.

8. A data processing system, characterized in that: include: An acquisition unit, used for acquiring array information of an array to be processed, wherein the array information includes a dimension size, a number of dimensions, and a step size on each dimension of the array to be processed; An address determination unit, used to determine the storage address of each element in the array to be processed according to the array information of the array to be processed; A data processing unit, configured to perform a preset process on the elements in each of the storage addresses in a sequence of a preset data format, wherein the preset process is writing or reading; Before determining the storage address of each element in the array to be processed according to the array information of the array to be processed, the method further includes: A definition unit, used to predefine a storage address calculation function; A judging unit, configured to judge whether the storage addresses of the elements in the input array are continuous according to the array information; judge whether the storage addresses of the elements in the output array are continuous according to the array information; judge whether the storage addresses of the elements in the input array and the storage addresses of the elements in the output array are continuous; the array to be processed includes the input array and the output array obtained by processing the input array using a preset operator; The address determination unit is specifically used for: Determine whether the array to be processed satisfies the calling condition for calling the storage address calculation function according to the array information; If the calling condition is met, the storage address of each element in the array to be processed is determined by using the storage address calculation function according to the array information of the array to be processed; If the storage addresses of the elements in the input array are continuous and the storage addresses of the elements in the output array are continuous, and the storage addresses of the elements in the input array and the storage addresses of the elements in the output array are also continuous, it is determined that the calling condition is not met, and the preset processing is directly performed on each element in the processing array according to the storage order of the storage addresses; The judgment unit is specifically used for: Obtain the dimension size and step size of the input array, and determine the step size of each element in the input array according to the dimension size and step size of the input array; determine whether the step size of each element in the input array is equal to the expected step size corresponding to itself, and if they are equal, determine that the storage addresses of the elements of the input array are continuous; Obtain the dimension size and step size of the output array, and determine the step size of each element in the output array according to the dimension size and step size of the output array; determine whether the step size of each element in the output array is equal to the expected step size corresponding to itself, and if they are equal, determine that the storage addresses of the elements of the output array are continuous; Determine whether the storage address of the last element of the input array is immediately followed by the storage address of the first element of the output array, and whether the total size of the two arrays is equal to the size of the continuous memory area they occupy.

9. A data processing device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the data processing method according to any one of claims 1 to 7 when executing a computer program.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images