Image data processing method and device and electronic equipment

Abstract

The invention discloses an image data processing method and device and electronic equipment. The method comprises the following steps: acquiring original image data before processing from a global memory of an image processor, and acquiring an original matrix corresponding to the original image data; transposing the original matrix to obtain a target matrix; reading data from a second data block in the original matrix through a first thread block in the target matrix and storing the data in a shared memory, wherein the ordinate of the second data block is determined according to the first abscissa and the first ordinate of the first thread block and the number of the thread blocks of the target matrix in the first preset direction; and reading the data from the shared memory, and transposing the read data to the target position of the target matrix in the global memory. According to the method and the device, the technical problem that the data processing speed in the field of image processing is relatively low due to serious data conflicts when the image data is read from the global memory and transposed in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to a method, apparatus, and electronic device for image data processing. Background Technology

[0002] Transpose is a fundamental operation for many higher-level applications and is widely used in linear algebra, machine learning, image processing, signal processing, numerical analysis, and other fields. For example, in the field of embodied intelligent logistics, embodied intelligent robots rely heavily on matrix transpose for core processes such as multimodal perception image data processing and network model inference, demonstrating strong scenario-specificity and practical application value. However, due to the complexity and unique nature of the transpose operation, severe data conflicts occur when reading data from global memory and transposing it in related technologies, resulting in slow data processing speeds in the field of image processing.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for image data processing, which at least solves the technical problem of slow data processing speed in the field of image processing due to severe data conflicts when reading image data from global memory and transposing it in related technologies.

[0005] According to one aspect of the embodiments of this application, an image data processing method is provided, comprising: obtaining raw image data before processing from the global memory of an image processor GPU, and obtaining a raw matrix corresponding to the raw image data, wherein each position of the raw matrix represents a data block; transposing the raw matrix to obtain a target matrix, wherein each position of the target matrix represents a thread block; reading data from a second data block in the raw matrix through a first thread block in the target matrix and storing it in shared memory, wherein the ordinate of the second data block is determined based on the first abscissa, the first ordinate of the first thread block and the number of thread blocks in the target matrix in a first preset direction; reading data from the shared memory and transposing the read data to a target position in the target matrix in the global memory, wherein the coordinates of the target position are the transpose of the coordinates of the second data block.

[0006] In some embodiments of this application, each thread block of the target matrix corresponds to two shared memory blocks, and the memory spaces of the two shared memory blocks are the same.

[0007] In some embodiments of this application, the coordinates of the second data block are determined as follows: the abscissa of the second data block is the ordinate of the first thread block; the ordinate of the second data block is determined as follows: the first abscissa and the first ordinate of the first thread block are obtained, and the number of thread blocks in the target matrix in the first preset direction is obtained; the first abscissa and the first ordinate are added together to obtain a first value; the first value and the number of thread blocks in the target matrix in the first preset direction are moduloed to obtain a second value; the second value is determined as the ordinate of the second data block.

[0008] In some embodiments of this application, reading data from a second data block in the original matrix and storing it in shared memory via a first thread block in the target matrix includes: reading data of continuous dimensions in a second preset direction from the second data block via the first thread block; storing the read data of continuous dimensions in the second preset direction in a first shared memory in the shared memory according to the same data arrangement; after all data is stored in the first shared memory, reading data of continuous dimensions in the second preset direction from the first shared memory and storing it in a data block in a third preset direction of the second shared memory, wherein the third preset direction is the transpose of the second preset direction.

[0009] In some embodiments of this application, reading data from shared memory and transposing the read data to the target position of the target matrix includes: after all the data of the continuous dimensions in the second preset direction in the first shared memory are stored in the corresponding data blocks in the third preset direction of the second shared memory, reading the data of the continuous dimensions in the second preset direction from the second shared memory; and copying the read data to the target position of the target matrix in the global memory.

[0010] In some embodiments of this application, reading continuous dimension data in a second preset direction from a second data block via a first thread block includes: determining whether the data in the original matrix is ​​continuous dimension data in the row direction or column direction; if the data in the original matrix is ​​continuous dimension data in the row direction, determining the row direction as the second preset direction; if the data in the original matrix is ​​continuous dimension data in the column direction, determining the column direction as the second preset direction; and reading continuous dimension data in the second preset direction from the second data block via the first thread block.

[0011] According to another aspect of the embodiments of this application, an image data processing apparatus is also provided, comprising: an acquisition module, configured to acquire raw image data before processing from the global memory of an image processor GPU, and acquire a raw matrix corresponding to the raw image data, wherein each position of the raw matrix represents a data block; a first transpose module, configured to transpose the raw matrix to obtain a target matrix, wherein each position of the target matrix represents a thread block; a reading module, configured to read data from a second data block in the raw matrix through a first thread block in the target matrix and store it in shared memory, wherein the ordinate of the second data block is determined based on the first abscissa, the first ordinate of the first thread block and the number of thread blocks in the target matrix in a first preset direction; and a second transpose module, configured to read data from the shared memory and transpose the read data to a target position in the target matrix in the global memory, wherein the coordinates of the target position are the transpose of the coordinates of the second data block.

[0012] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory for storing program instructions; and a processor connected to the memory for executing program instructions to perform the following functions: obtaining raw image data before processing from the global memory of an image processor GPU, and obtaining a raw matrix corresponding to the raw image data, wherein each position of the raw matrix represents a data block; transposing the raw matrix to obtain a target matrix, wherein each position of the target matrix represents a thread block; reading data from a second data block in the raw matrix through a first thread block in the target matrix and storing it in shared memory, wherein the ordinate of the second data block is determined based on the first abscissa, the first ordinate of the first thread block and the number of thread blocks in the target matrix in a first preset direction; reading data from the shared memory and transposing the read data to a target position in the target matrix in the global memory, wherein the coordinates of the target position are the transpose of the coordinates of the second data block.

[0013] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device on which the non-volatile storage medium is located executes the above-described image data processing method by running the computer program.

[0014] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the above-described image data processing method.

[0015] In this embodiment, by determining the original matrix and the target matrix, and then using the data block of the target matrix to locate the data block of the original matrix that needs to read the image data, the read image data is stored in shared memory. Finally, the image data is read from the shared memory and transposed to the corresponding position in the target matrix. This achieves the goal of using shared memory buffering and thread block task reordering technology to realize efficient matrix transposition operation, thereby improving GPU hardware utilization and processing speed. It also solves the technical problem of slow data processing speed in the field of image processing due to serious data conflicts when reading image data from global memory and transposing it in related technologies. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 This is a schematic diagram based on the principle of matrix transposition according to related technologies;

[0018] Figure 2 This is a hardware structure block diagram of a computer terminal for an image data processing method according to an embodiment of this application;

[0019] Figure 3 This is a flowchart of an image data processing method according to an embodiment of this application;

[0020] Figure 4 This is a schematic diagram illustrating the principle of data access between global memory data blocks based on related technologies;

[0021] Figure 5 This is a schematic diagram comparing the principles of global memory bank conflict optimization based on embodiments of this application and related technologies;

[0022] Figure 6 This is a schematic diagram illustrating the principle of the dual shared memory buffer technology according to an embodiment of this application;

[0023] Figure 7 This is a schematic diagram of an image data processing apparatus according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:

[0027] Graphics Processing Unit (GPU): A processor capable of large-scale parallel computing, particularly suitable for processing image and video data, as well as high-performance computing tasks.

[0028] Matrix transpose: A mathematical operation that interchanges the rows and columns of a matrix. In two-dimensional arrays, the transpose operation can change how data is stored to suit different algorithmic requirements.

[0029] Shared memory: A type of storage resource within the GPU that can be shared by all threads within the same thread block, used to accelerate inter-thread communication and data access.

[0030] Thread block: In GPU programming, a group of threads that can execute in parallel. Threads within a thread block can share data and execution resources.

[0031] A bank refers to a shared memory space divided into modules of equal size, allowing simultaneous reading and writing of data in different modules.

[0032] Bank conflicts: In GPU shared memory, when multiple threads within the same thread block access different addresses of the same bank simultaneously, it can cause data access to be serialized, reducing efficiency.

[0033] The matrix transpose operation interchanges the rows and columns of a matrix and is commonly used in matrix operations. Specifically, as shown below... Figure 1 As shown, assume that the dimension of matrix A is m. n (m rows, n columns), in Figure 1 The dimension of matrix A is 2 before transpose. 4, which is 2 rows and 4 columns, then the transpose of A The dimension is n m (n rows, m columns), in Figure 1 middle The dimension is 4 after transpose. 2, meaning 4 rows and 2 columns. For example, in the field of embodied intelligent logistics, embodied intelligent robots rely heavily on matrix transposition for core processes such as multimodal perception image data processing and network model inference, demonstrating strong scenario specificity and practical application value. As embodied intelligence accelerates its industrialization in scenarios such as express delivery sorting and cargo unloading, edge GPUs have become mainstream intelligent brain hardware and have experienced rapid development. Consequently, transposition operations have become core operations on edge GPUs, widely used in scenarios such as robot brain image processing and model inference. Major GPU hardware manufacturers have also focused on the development and optimization of transposition operations. However, due to the complexity and special nature of transposition operations, their performance is severely limited by hardware bandwidth and scheduling strategies. This results in existing transposition optimization methods being unable to fully adapt to the hardware characteristics of embodied intelligent edge GPUs, the characteristics of multimodal perception data, and scenario requirements, failing to fully utilize the hardware performance of embodied intelligent edge GPUs, leading to low transposition efficiency. Consequently, this affects core indicators such as environmental perception accuracy and motion control response speed of embodied intelligent robots in the logistics field.

[0034] In related technologies, major edge GPU hardware manufacturers and computing library providers typically consider memory access patterns, load balancing, and memory consumption when developing and optimizing transpose operations. However, in actual optimization, the performance of various GPU hardware components, such as bandwidth, shared memory size, and register size, is affected. Furthermore, severe data conflicts can occur when reading data from global memory and transposing it, leading to serialization of the execution process. Under these constraints, transpose operations cannot be effectively optimized, and their performance often falls short of expectations.

[0035] To address the aforementioned technical problems, this application provides corresponding solutions, which are detailed below.

[0036] The image data processing method embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 2 A hardware block diagram of a computer terminal for implementing a method for image data processing is shown. Figure 2 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 2 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 2 The more or fewer components shown, or having the same Figure 2 The different configurations shown.

[0037] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0038] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image data processing method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the above-mentioned image data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0039] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a radio frequency (RF) module, used for wireless communication with the Internet.

[0040] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0041] It should be noted here that, in some optional embodiments, the above... Figure 2 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 2 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0042] In the above operating environment, this application provides an embodiment of an image data processing method. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0043] Figure 3 This is a flowchart of an image data processing method according to an embodiment of this application, such as... Figure 3 As shown, this method can be applied to GPUs or to intelligent robots in the logistics field, and specifically includes the following steps:

[0044] Step S302: Obtain the raw image data before processing from the global memory of the image processor GPU, and obtain the raw matrix corresponding to the raw image data, wherein each position of the raw matrix represents a data block.

[0045] In step S302 above, the original image data is the original data to be transposed. This original image data can be original logistics image data, including various types of image data from sensors, databases, files, network transmissions, etc., such as the pixel values ​​of the image. The original matrix is ​​a mathematical representation of the original image data. For example, the original matrix can be a two-dimensional array whose number of rows and columns matches the structure of the original image data.

[0046] In some embodiments, when processing image recognition tasks such as those in the logistics field, the original image data can be a JPEG image. Assuming the original image resolution is 640x480, the corresponding original matrix is ​​a two-dimensional array with 640 rows and 480 columns. Each data block contains 16x16 data points, and the original matrix is ​​divided into multiple 16x16 smaller matrix blocks. Each smaller matrix block is processed by a thread block of the target matrix. For example, the first thread block of the target matrix at coordinates (0, 0) is responsible for processing the data in the first data block of the original matrix, which may contain, for example, 16x16 data points.

[0047] Step S304: Transpose the original matrix to obtain the target matrix, where each position in the target matrix represents a thread block.

[0048] In step S304 above, matrix transpose is the operation of interchanging the rows and columns of a matrix. For example, for an m x n matrix, its transpose is an n x m matrix. The target matrix is ​​the result matrix obtained by transposing the original matrix. In the target matrix, each position corresponds to a thread block, and each thread block is responsible for processing the data in a data block (e.g., the second data block) of the original matrix.

[0049] In one alternative embodiment, Figure 4 This diagram illustrates the principle of data access between global memory data blocks. It shows the relationship between the original matrix before transpose and the target matrix after transpose. Figure 4 As shown, the original matrix before transpose is a 3-row (x-dimensional) and 4-column (y-dimensional) matrix, and the target matrix after transpose is a 4-row (y-dimensional) and 3-column (x-dimensional) matrix. In the original matrix, each coordinate (or position) represents a data block, and in the target matrix, each coordinate or position marks a thread block. For example, in the target matrix, (0,0) is a thread block, (0,1) is a thread block, (0,2) is a thread block, that is, the region corresponding to each coordinate represents a thread block.

[0050] Step S306: Read data from the second data block in the original matrix through the first thread block in the target matrix and store it in the shared memory. The ordinate of the second data block is determined based on the first abscissa, the first ordinate of the first thread block and the number of thread blocks in the target matrix in the first preset direction.

[0051] In step S306 above, the element blocks in the original matrix and the target matrix can be represented using coordinates including rows and columns. Since the target matrix is ​​the transpose of the original matrix, the coordinates of the thread blocks in the target matrix have swapped row and column positions compared to the data block coordinates in the original matrix. For example, the data block located in row i and column j in the original matrix is ​​located in row j and column i after transposition. The first thread block is a thread block determined by any coordinate in the target matrix, used to locate the second data block in the original matrix and store the data in the second data block in shared memory. Shared memory is a memory area of ​​a preset size that can be shared among multiple threads. The first preset direction can be the target matrix along the y-axis (or the y-dimensional dimension, such as...). Figure 4 The number of thread blocks on the transposed y-dimensional plane.

[0052] In some embodiments, the coordinates of the second data block are determined as follows: the x-coordinate of the second data block is the first y-coordinate of the first thread block; the y-coordinate of the second data block is determined as follows: the first x-coordinate and the first y-coordinate of the first thread block are obtained, and the number of thread blocks in the target matrix in the first preset direction is obtained; the first x-coordinate and the first y-coordinate are added together to obtain a first value; the first value and the number of thread blocks in the target matrix in the first preset direction are moduloed to obtain a second value; and the second value is determined as the y-coordinate of the second data block.

[0053] For example, Figure 5 This is a schematic diagram comparing the principles of global memory bank conflict optimization based on embodiments of this application and related technologies, such as... Figure 5 As shown, the number of data blocks configured for the transpose operation is based on the transposed target matrix. The target matrix has continuous dimensions in the row direction, and the first row of data after transpose is the first column of data before transpose. Before optimization, the transposed target matrix is ​​divided into blocks. Figure 5 The target matrix is ​​divided into 4-row (y-dimensional) and 3-column (x-dimensional) matrix blocks. The thread block numbers in the first row are (0, 0), (0, 1), and (0, 2) from left to right. The coordinates of the data block accessed by each thread block (y, x) of the target matrix are (y, x), and the coordinates of the data block accessed by the thread block (y, x) of the original matrix before transpose are (x, y). Because this method involves cross-dimensional access when reading data from global memory, the access frequency of different memory channels is severely unbalanced, resulting in low memory access efficiency and low bandwidth utilization.

[0054] After optimization, the transposed target matrix is ​​divided into thread blocks and numbered. Figure 5 The first row of the optimized target matrix has thread block numbers distributed as (0, 0), (0, 1), and (0, 2). The first dimension of the coordinates represents the y-axis (or y-dimensional), and the second dimension represents the x-axis (or x-dimensional). Taking the first thread block (y, x) in the target matrix as an example, the first thread block (y, x) of the target matrix is ​​responsible for reading image data from the data block with coordinates (x, (x+y)%group_num_y) in the original matrix before transpose and storing it in shared memory. Here, y in the first thread block (y, x) represents the first horizontal coordinate, x in (y, x) represents the first vertical coordinate, the parameter group_num_y represents the number of thread blocks in the y-axis (or y-dimensional, i.e., the first preset direction) direction of the target matrix, x+y represents the first value, and (x+y)%group_num_y represents the second value, i.e., the vertical coordinate of the second data block corresponding to coordinates (x, (x+y)%group_num_y) in the original matrix before transpose. Because the data blocks of the original matrix before transpose that are accessed by the thread blocks of the target matrix that are executed concurrently in the optimized method are not all concentrated in one column, but are distributed, it can ensure that the frequency of access to each memory channel is relatively balanced, thereby improving the global memory bandwidth utilization.

[0055] Step S308: Read data from shared memory and transpose the read data to the target position of the target matrix in global memory, where the coordinates of the target position are the transpose of the coordinates of the second data block.

[0056] In step S308 above, the number of shared memory can be set according to actual needs. The data read from the shared memory can be the data after processing the original image data stored in the shared memory. The target position is the transpose of the coordinates of the data block (i.e. the second data block) of the original image data corresponding to the read data in the original matrix. For example, when the coordinates of the second data block of the original matrix are (x, (x+y)%group_num_y) (the parameter group_num_y represents the number of thread blocks in the y-axis direction of the target matrix), the coordinates of the target position of the target matrix are ((x+y)%group_num_y, x). This target position can be called the position of the first data block in the target matrix.

[0057] For ease of understanding, the first data block, the first thread block, and the second data block are explained below:

[0058] The first data block and the first thread block are the data block and thread block corresponding to any position in the transposed target matrix. The second data block is the data block at any position in the original matrix. A thread block in the target matrix (such as the first thread block) is responsible for transferring the data block in the original matrix (such as the second data block) to the first data block in the target matrix. The position of the first data block is the target position mentioned above, and the transpose of the coordinates of the first data block is the coordinates of the second data block.

[0059] According to steps S302 to S308 above, by determining the original matrix and the target matrix, and then using the data block of the target matrix to locate the data block of the original matrix that needs to read the image data, the read image data is stored in shared memory. Finally, the image data is read from the shared memory and transposed to the corresponding position in the target matrix. This achieves the goal of using shared memory buffering and thread block task reordering technology to realize efficient matrix transposition operation, thereby improving the GPU hardware utilization and processing speed. It also solves the technical problem of slow data processing speed in the field of image processing due to serious data conflicts when reading image data from global memory and transposing it in related technologies.

[0060] In order to take advantage of the parallel processing capabilities and high-bandwidth data access characteristics of the GPU to achieve faster and more efficient matrix transpose operations, in some embodiments of this application, the original image data in the original matrix and the transposed data in the target matrix (that is, the data obtained by transposing the image data read from shared memory to the target matrix) are located in the global memory of the image processor GPU.

[0061] Because directly reading (image) data from consecutive dimensions of the global memory data block and storing it in local memory would cause an entire row of data from the global memory data block to be stored in a single column of shared memory, resulting in numerous shared memory bank conflicts. This severely impacts bandwidth utilization and significantly reduces data transfer speed. Therefore, to alleviate shared memory bank conflicts and improve memory bandwidth utilization and data transfer speed, the shared memory can be configured as follows: each thread block of the target matrix corresponds to two shared memory blocks, and the memory space of the two shared memory blocks is identical. In some embodiments, the memory space of the shared memory can be the same as the memory space of the global memory data block.

[0062] The image data is read from the second data block of the original matrix through the first thread block of the target matrix and stored in shared memory. For example, if two shared memory blocks are allocated for each thread block of the target matrix, the data of the original matrix can be stored in the first shared memory first, and then the data can be stored from the first shared memory to the second shared memory. Specifically: the data of the continuous dimension in the second preset direction is read from the second data block through the first thread block; the data of the continuous dimension in the second preset direction is stored in the first shared memory in the same data arrangement; after all the data is stored in the first shared memory, the data of the continuous dimension in the second preset direction is read from the first shared memory and stored in the data block in the third preset direction of the second shared memory, where the third preset direction is the transpose of the second preset direction.

[0063] In some embodiments, the second preset direction can be a row direction or a column direction. When the second preset direction is a row direction, the third preset direction is a column direction; when the second preset direction is a column direction, the third preset direction is a row direction. Specifically, this is determined through the following steps: determining whether the data in the original matrix is ​​continuous in either the row or column direction; if the data in the original matrix is ​​continuous in the row direction, the row direction is determined as the second preset direction; if the data in the original matrix is ​​continuous in the column direction, the column direction is determined as the second preset direction.

[0064] The same data arrangement is the same as the second preset direction. For example, after reading the data of the continuous dimension in the row direction (second preset direction), the data is stored in the first shared memory in the row direction. Multiple rows of data can be stored in the first shared memory sequentially or in parallel.

[0065] Since the bandwidth inside shared memory is much greater than the transmission bandwidth from global memory to shared memory, after all (image) data is stored in the first shared memory, data in a continuous dimension in a second preset direction can be read from the first shared memory and stored in a data block in a third preset direction of the second shared memory. For example, a row of data can be read from the first shared memory and stored in a data block in a column of the second shared memory. By transferring the bank conflict from global memory to shared memory to the bank conflict inside shared memory, the speed of copying data from global memory to shared memory can be greatly improved, and the bandwidth utilization and efficiency of transpose operation can be improved.

[0066] In some embodiments of this application, reading data from shared memory and transposing the read data to the target position of the target matrix can be performed by the following steps: after all the data of the continuous dimensions in the second preset direction in the first shared memory are stored in the corresponding data block in the third preset direction of the second shared memory, the data of the continuous dimensions in the second preset direction are read from the second shared memory; the read data is copied to the target position of the target matrix in the global memory.

[0067] During matrix transposition, taking the second preset direction as the row direction as an example, the data in each row becomes each column of the transposed data. If data is directly read from the first shared memory and transposed to global memory, the data in the same column (the transposed row) may need to wait for the previous row to be completely processed before processing can begin. This introduces data dependency and limits the efficiency of parallel processing. By using the second shared memory as a buffer, it can be ensured that the processing of each row of data does not depend on the data in the previous row, thereby reducing the waiting time in parallel operations. Furthermore, it can be ensured that each data block can operate independently when processing data, without being blocked due to unprepared data. After the data rows in the first shared memory are completely stored in the corresponding columns of the second shared memory, all data blocks can simultaneously read data from the second shared memory and transpose it, thereby maximizing the utilization of the GPU's parallel computing resources.

[0068] It should be noted that the data in the embodiments of this application can be represented as image data, for example, it can be logistics image data or image data in other fields, and is not limited here.

[0069] Figure 6 This is a schematic diagram illustrating the principle of the dual shared memory buffer technology according to an embodiment of this application, such as... Figure 6 As shown, data in global memory is read in data blocks. These data blocks have a certain number of rows and columns; for simplicity, we assume they are contiguous rows-based storage. The dual shared memory buffer technique can be divided into the following steps:

[0070] (1) First shared memory buffer (shared memory 1): Data read from global memory is first stored in shared memory 1. However, since the data is read continuously row by row, if it is written directly to shared memory, a row of data in global memory will be written to a column in shared memory. This will cause bank conflicts in shared memory. Because in shared memory, data is stored in the form of columns, and each column corresponds to a bank. If a row of data is written to the same column, it will cause conflicts between banks and reduce the data transmission speed.

[0071] (2) Data rearrangement: In order to avoid the bank conflict problem mentioned above, a row of data can be read from shared memory 1 (i.e., the first shared memory), but instead of writing it directly back to global memory, it is written to a column of another shared memory buffer (shared memory 2, i.e. the second shared memory). Although bank conflicts may still occur in shared memory 2, since the internal bandwidth of shared memory is much higher than the transmission bandwidth from global memory to shared memory, the impact of such conflicts on the overall data transmission speed is small.

[0072] (3) Second shared memory buffer (shared memory 2): The data stored in shared memory 2 is stored in a row-based contiguous manner, which matches the data layout after the GPU transpose operation. This method first performs data layout transformation (i.e. row to column or column to row) within the shared memory, thereby avoiding bank conflicts that occur during data copying directly from global memory to shared memory.

[0073] (4) Data write-back to global memory: Data is written back to global memory from shared memory 2 in a continuous dimension manner. Since the data layout is now matched with the storage method of global memory, the data write-back speed is faster, reducing data transmission delay and improving overall computational efficiency.

[0074] By effectively reducing data conflicts between global memory and shared memory through dual shared memory buffering technology, the read and write speed and bandwidth utilization of data in the field of image processing are improved, thereby significantly enhancing the performance and efficiency of matrix transpose operations on edge GPU platforms.

[0075] According to embodiments of this application, an apparatus for image data processing is provided. It should be noted that the image data processing apparatus of this application embodiment can be used to execute the image data processing method provided in this application embodiment. The image data processing apparatus provided in embodiments of this application will be described below.

[0076] Figure 7 This is a schematic diagram of an image data processing apparatus according to an embodiment of this application. Figure 7 As shown, the device includes:

[0077] The acquisition module 72 is used to acquire the raw image data before processing from the global memory of the image processor GPU, and to acquire the raw matrix corresponding to the raw image data, wherein each position of the raw matrix represents a data block;

[0078] The first transpose module 74 is used to transpose the original matrix to obtain the target matrix, wherein each position of the target matrix represents a thread block;

[0079] The reading module 76 is used to read data from the second data block in the original matrix through the first thread block in the target matrix and store it in the shared memory, wherein the ordinate of the second data block is determined based on the first abscissa, the first ordinate of the first thread block and the number of thread blocks in the target matrix in the first preset direction;

[0080] The second transpose module 78 is used to read data from shared memory and transpose the read data to the target position of the target matrix in global memory, wherein the coordinates of the target position are the transpose of the coordinates of the second data block.

[0081] In the image data processing apparatus provided in the embodiments of this application, each thread block of the target matrix corresponds to two shared memory blocks, and the memory space of the two shared memory blocks is the same.

[0082] In the image data processing apparatus provided in this application embodiment, the coordinates of the second data block are determined in the following way: the horizontal coordinate of the second data block is the first vertical coordinate of the first thread block; the vertical coordinate of the second data block is determined in the following way: the first horizontal coordinate and the first vertical coordinate of the first thread block are obtained, and the number of thread blocks in the target matrix in the first preset direction is obtained; the first horizontal coordinate and the first vertical coordinate are added together to obtain a first value; the first value and the number of thread blocks in the target matrix in the first preset direction are moduloed to obtain a second value; and the second value is determined as the vertical coordinate of the second data block.

[0083] In the image data processing apparatus provided in this application embodiment, the reading module is further configured to read data of continuous dimensions in a second preset direction from the second data block through a first thread block; store the read data of continuous dimensions in the second preset direction in a first shared memory in the shared memory according to the same data arrangement; after all the data is stored in the first shared memory, read the data of continuous dimensions in the second preset direction from the first shared memory and store it in a data block in a third preset direction of the second shared memory of the shared memory, wherein the third preset direction is the transpose of the second preset direction.

[0084] In the image data processing apparatus provided in this application embodiment, the second transpose module is further configured to, after all the data of the continuous dimensions in the second preset direction in the first shared memory are stored in the corresponding data block in the third preset direction of the second shared memory, read the data of the continuous dimensions in the second preset direction from the second shared memory; and copy the read data to the target position of the target matrix in the global memory.

[0085] In the image data processing apparatus provided in this application embodiment, the reading module is further configured to determine whether the data in the original matrix is ​​continuous dimension data in the row direction or column direction; if the data in the original matrix is ​​continuous dimension data in the row direction, the row direction is determined as a second preset direction; if the data in the original matrix is ​​continuous dimension data in the column direction, the column direction is determined as a second preset direction; and the continuous dimension data in the second preset direction is read from the second data block through the first thread block.

[0086] This application embodiment also provides an electronic device, which includes a memory and a processor. The memory is used to store program instructions. The processor is connected to the memory and is used to execute program instructions that perform the following functions: obtaining raw image data before processing from the global memory of the image processor GPU, and obtaining the raw matrix corresponding to the raw image data, wherein each position of the raw matrix represents a data block; transposing the raw matrix to obtain a target matrix, wherein each position of the target matrix represents a thread block; reading data from a second data block in the raw matrix through a first thread block in the target matrix and storing it in shared memory, wherein the first thread block is a data block corresponding to any element block in the target matrix, and the ordinate of the second data block is determined based on the first abscissa, the first ordinate of the first thread block and the number of data blocks in the target matrix in a first preset direction; reading data from the shared memory and transposing the read data to a target position in the target matrix in the global memory, wherein the coordinates of the target position are the transpose of the coordinates of the second data block.

[0087] This application embodiment also provides a non-volatile storage medium, which includes a stored computer program. The device containing the non-volatile storage medium executes the following image data processing method by running the computer program: obtaining unprocessed image data from the global memory of the image processor GPU and obtaining an original matrix corresponding to the original image data, wherein each position of the original matrix represents a data block; transposing the original matrix to obtain a target matrix, wherein each position of the target matrix represents a thread block; reading data from a second data block in the original matrix through a first thread block in the target matrix and storing it in shared memory, wherein the ordinate of the second data block is determined based on the first abscissa, the first ordinate of the first thread block, and the number of thread blocks in the target matrix in a first preset direction; reading data from the shared memory and transposing the read data to a target position in the target matrix in global memory, wherein the coordinates of the target position are the transpose of the coordinates of the second data block.

[0088] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the image data processing method in various embodiments of this application.

[0089] This application also provides a computer program that, when executed by a processor, implements the steps of the image data processing method in various embodiments of this application.

[0090] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0091] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0092] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0093] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0095] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0096] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for image data processing, characterized in that, include: The raw image data before processing is obtained from the global memory of the image processor GPU, and the original matrix corresponding to the raw image data is obtained, wherein each position of the original matrix represents a data block; The original matrix is ​​transposed to obtain the target matrix, wherein each position of the target matrix represents a thread block; Data is read from the second data block in the original matrix through the first thread block in the target matrix and stored in the shared memory, wherein the ordinate of the second data block is determined based on the first abscissa, the first ordinate of the first thread block and the number of thread blocks in the target matrix in the first preset direction; Data is read from the shared memory and transposed to the target position of the target matrix in the global memory, wherein the coordinates of the target position are the transpose of the coordinates of the second data block.

2. The method according to claim 1, characterized in that, Each thread block of the target matrix corresponds to two shared memory blocks, and the two shared memory blocks have the same memory space.

3. The method according to claim 1, characterized in that, The coordinates of the second data block are determined in the following way: The x-coordinate of the second data block is the y-coordinate of the first thread block; The ordinate of the second data block is determined in the following way: Obtain the first x-coordinate and the first y-coordinate of the first thread block, and obtain the number of thread blocks of the target matrix in the first preset direction; Add the first horizontal coordinate and the first vertical coordinate to obtain the first value; The first value and the number of thread blocks in the target matrix in the first preset direction are moduloed to obtain the second value; The second value is determined as the ordinate of the second data block.

4. The method according to claim 1, characterized in that, Data is read from the second data block in the original matrix and stored in shared memory through the first thread block in the target matrix, including: The first thread block reads data in a continuous dimension along a second preset direction from the second data block. The data read in the continuous dimensions of the second preset direction are stored in the first shared memory of the shared memory in the same data arrangement; After all the data is stored in the first shared memory, the data of the continuous dimension in the second preset direction is read from the first shared memory and stored in the data block in the third preset direction of the second shared memory of the shared memory, wherein the third preset direction is the transpose of the second preset direction.

5. The method according to claim 4, characterized in that, Reading data from the shared memory and transposing the read data to the target position in the target matrix includes: After the data of the continuous dimensions in the second preset direction of the first shared memory are all stored in the data block in the third preset direction of the corresponding second shared memory, the data of the continuous dimensions in the second preset direction are read from the second shared memory. The read data is copied to the target location of the target matrix in global memory.

6. The method according to claim 4, characterized in that, Reading data in a continuous dimension along a second preset direction from the second data block through the first thread block includes: Determine whether the data in the original matrix is ​​continuous in either the row or column direction; When the data in the original matrix is ​​continuous in the row direction, the row direction is determined as the second preset direction; When the data in the original matrix is ​​continuous in the column direction, the column direction is determined as the second preset direction; The first thread block reads data of the continuous dimension in the second preset direction from the second data block.

7. An image data processing apparatus, characterized in that, include: The acquisition module is used to acquire the raw image data before processing from the global memory of the image processor GPU, and to acquire the raw matrix corresponding to the raw image data, wherein each position of the raw matrix represents a data block; The first transpose module is used to transpose the original matrix to obtain the target matrix, wherein each position of the target matrix represents a thread block; The reading module is used to read data from the second data block in the original matrix through the first thread block in the target matrix and store it in shared memory, wherein the ordinate of the second data block is determined based on the first abscissa, the first ordinate of the first thread block and the number of thread blocks in the target matrix in a first preset direction; The second transpose module is used to read data from the shared memory and transpose the read data to the target position of the target matrix in the global memory, wherein the coordinates of the target position are the transpose of the coordinates of the second data block.

8. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor, connected to the memory, is configured to execute program instructions to perform the following functions: retrieve raw image data from the global memory of the image processor (GPU) before processing, and retrieve the raw matrix corresponding to the raw image data, wherein each position of the raw matrix represents a data block; transpose the raw matrix to obtain a target matrix, wherein each position of the target matrix represents a thread block; read data from a second data block in the raw matrix through a first thread block in the target matrix and store it in shared memory, wherein the ordinate of the second data block is determined based on the first abscissa, the first ordinate of the first thread block, and the number of thread blocks in the target matrix in a first preset direction; read data from the shared memory and transpose the read data to a target position in the target matrix in the global memory, wherein the coordinates of the target position are the transpose of the coordinates of the second data block.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the image data processing method according to any one of claims 1 to 6 by running the computer program.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the image data processing method according to any one of claims 1 to 6.

Images