An image processing method, apparatus and electronic device

Abstract

The application provides an image processing method, device and electronic equipment, the method comprising: dividing a to-be-processed image into a plurality of sub-blocks; processing the plurality of sub-blocks using a plurality of CUDA (Compute Unified Device Architecture) streams, each CUDA stream in the plurality of CUDA streams comprising: a calculation operation and a memory operation, and a sub-block in the plurality of sub-blocks being processed asynchronously between the calculation operation in a first CUDA stream and the memory operation in a second CUDA stream; and splicing and fusing the processing results of the plurality of sub-blocks to obtain a processing result of the to-be-processed image. In the implementation process of the above scheme, the multi-core resources of the GPU are fully utilized through the asynchronous processing between the calculation operation and the memory operation in different CUDA streams, the idle time of the processing unit is reduced, the resource utilization rate of the CUDA stream is maximized, and therefore the overall processing efficiency of the high-resolution image is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image processing method, apparatus, and electronic device. Background Technology

[0002] Currently, processing high-resolution images typically uses traditional image scaling algorithms implemented on a CPU. However, the serial computation mode is insufficient to meet the real-time processing requirements of high-resolution images. For example, when processing a 38400×21600 pixel image, the CPU needs to perform calculations pixel by pixel, resulting in processing times of several minutes or even longer.

[0003] To meet the demands of massive data processing, the industry has gradually ported scaling algorithms to the Graphics Processing Unit (GPU) platform, leveraging its massively parallel architecture to accelerate computation. However, this GPU implementation is typically based on a single unified computing device architecture, such as CUDA streams or OpenCL command queues, resulting in low overall processing efficiency for large-resolution images. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method, apparatus, and electronic device to improve the problem of low overall processing efficiency for high-resolution images.

[0005] This application provides an image processing method, comprising: dividing an image to be processed into multiple sub-tiles; processing the multiple sub-tiles using multiple Unified Computing Device Architecture (CUDA) streams, each CUDA stream including computational operations and memory operations, wherein the computational operation of one sub-tile in a first CUDA stream and the memory operation of another sub-tile in a second CUDA stream are processed asynchronously; and stitching and fusing the processing results of the multiple sub-tiles to obtain the processing result of the image to be processed. In the implementation of the above scheme, by dividing the image to be processed into multiple sub-tiles and utilizing the asynchronous processing characteristic of the computational and memory operations of multiple CUDA streams to process the multiple sub-tiles, and then stitching and fusing the processing results of the multiple sub-tiles, the asynchronous processing between computational and memory operations in different CUDA streams fully utilizes the multi-core resources of the GPU, reduces the idle time of the processing unit, maximizes the resource utilization of the CUDA streams, and thus improves the overall processing efficiency of high-resolution images.

[0006] Optionally, in this embodiment, the memory operation includes: an H2D operation copying CPU memory to GPU memory, and a D2H operation copying GPU memory to CPU memory; processing multiple sub-tiles using multiple Unified Computing Device Architecture (CUDA) streams includes: processing multiple sub-tiles using an H2D operation in one CUDA stream and a D2H operation in another CUDA stream; and / or, processing multiple sub-tiles using an H2D operation in one CUDA stream and a computation operation in another CUDA stream; and / or, processing multiple sub-tiles using a D2H operation in one CUDA stream and a computation operation in another CUDA stream. In the implementation of the above scheme, by distributing the H2D, D2H and computation operations of multiple sub-tiles to different CUDA streams and utilizing the asynchronous characteristics of streams, deep pipelined computation and transmission are achieved. This cross-stream computation-transmission-computation-transmission overlapping mechanism ensures that the GPU's computing cores and PCIe bus are almost always busy, greatly reducing the latency of memory copying. This extreme exploitation of hardware resources is a technical effect of breaking through the single-stream bottleneck through stream-level parallelism.

[0007] Optionally, in this embodiment, the processing results of multiple sub-tiles are stitched and fused, including: stream synchronization of the processing results of H2D operations, D2H operations, and computation operations in each CUDA stream; if the stream synchronization of multiple sub-tiles is completed, the processing results of the multiple sub-tiles are stitched and fused. In the implementation of the above scheme, by introducing a stream synchronization mechanism in the stitching and fusion stage, the process forces the waiting for the completion of H2D, D2H, and computation operations in all CUDA streams before result stitching, thereby solving the potential data competition and state inconsistency problems in asynchronous processing. Furthermore, by concentrating the stream synchronization points in the stitching stage rather than dispersing them in the processing of each sub-tile, the uncertainty of intermediate states is reduced, thereby significantly improving stitching quality (such as natural edge transitions and geometric alignment accuracy) and system stability (such as avoiding crashes or invalid memory accesses).

[0008] Optionally, in this embodiment, the processing results of multiple sub-plots are stitched and fused, including: obtaining the position information of multiple sub-plots; and stitching and fusing the processing results of multiple sub-plots based on the position information of the multiple sub-plots. In the implementation of the above scheme, by introducing an explicit perception and utilization mechanism for the position information of sub-plots during the stitching and fusion process, the processing result of each sub-plot can be accurately located and aligned according to its spatial coordinates in the original image, thereby effectively avoiding problems such as stitching misalignment, overlap distortion, or edge breakage caused by ignoring positional relationships in traditional methods. This position information-driven stitching strategy not only improves the accuracy of image processing results but also enhances the system's robustness to data deviations under non-uniform transformations, heterogeneous processing flows, and asynchronous execution paths, achieving stable output of high-quality image reconstruction in complex parallel environments.

[0009] Optionally, in this embodiment, the computation operation is an image scaling operation, and the memory operation is a non-paged memory copy operation. Before stitching and fusing the processing results of multiple sub-tiles, the method further includes: stream synchronization of the processing results of the image scaling and memory copy operations in each CUDA stream to obtain the processing results of multiple sub-tiles. In the implementation of the above scheme, by introducing a stream synchronization mechanism for each CUDA stream in the image scaling scenario, and combining it with the use of non-paged memory for memory copy operations, the processing results of each sub-tile have high determinism and consistency in time. Since non-paged memory improves the transmission efficiency of H2D or D2H operations, and stream synchronization ensures that all sub-tiles have completed their respective image scaling and data transmission before entering the stitching and fusing stage, the timing disorder or data race problem caused by asynchronous execution is avoided. This design not only ensures that multiple sub-tiles have a consistent spatial transformation benchmark (such as scaling ratio, boundary alignment, etc.) after scaling, but also makes the entire image processing flow more predictable and stable.

[0010] Optionally, in this embodiment, the image scaling operation includes: performing a weighted average interpolation on the four pixel values ​​surrounding the target pixel position in the sub-block to obtain the scaled pixel value of the target pixel position in the sub-block. In the above implementation, by performing a weighted average interpolation on the four pixel values ​​surrounding the target pixel position in the sub-block, this local weighting method (i.e., increasing computation by only a small amount) significantly improves the image processing quality while also enhancing the performance efficiency of image processing, effectively balancing image quality and performance efficiency.

[0011] Optionally, in this embodiment, the number of CUDA streams is the same as the number of sub-tiles, and different sub-tiles are assigned to different CUDA streams. In the implementation of the above scheme, by having each sub-tile occupy its own CUDA stream instead of sharing a stream, operations such as H2D / D2H memory copying and image scaling calculations for each sub-tile are executed in parallel within completely independent streams, thereby completely eliminating resource contention and implicit dependencies between stream tasks. On high-end GPUs that support a sufficient number of streams (such as the NVIDIA A100), this design can increase image processing throughput from 80% utilization in the traditional multi-stream mode to a theoretical peak close to 100%. Especially when processing high-resolution images, it can compress the overall processing time from a linear superposition of "serial computation + transmission" modes to approximately the processing time of a single sub-tile, thus achieving a super-linear speedup.

[0012] This application also provides an image processing apparatus, including: an image segmentation module for dividing an image to be processed into multiple sub-tiles; a sub-tile stream processing module for processing the multiple sub-tiles using multiple Unified Computing Device Architecture (CUDA) streams, each of the multiple CUDA streams including computational operations and memory operations, wherein the computational operation of one sub-tile in the first CUDA stream and the memory operation of another sub-tile in the second CUDA stream are processed asynchronously; and a tile stitching and fusion module for stitching and fusion of the processing results of the multiple sub-tiles to obtain the processing result of the image to be processed.

[0013] Optionally, in this embodiment, the memory operation includes: an H2D operation copying CPU memory to GPU memory, and a D2H operation copying GPU memory to CPU memory; the sub-tile stream processing module includes: a first processing submodule, used to process multiple sub-tiles using an H2D operation in one CUDA stream and a D2H operation in another CUDA stream; and / or, a second processing submodule, used to process multiple sub-tiles using an H2D operation in one CUDA stream and a computation operation in another CUDA stream; and / or, a third processing submodule, used to process multiple sub-tiles using a D2H operation in one CUDA stream and a computation operation in another CUDA stream.

[0014] Optionally, in this embodiment of the application, the tile stitching and fusion module includes: an operation multi-stream synchronization submodule, used to perform stream synchronization on the processing results of H2D operations, D2H operations and computation operations in each of the multiple CUDA streams; and a result stitching and fusion submodule, used to stitch and fuse the processing results of the multiple sub-tiles if the stream synchronization of the multiple sub-tiles is completed.

[0015] Optionally, in this embodiment of the application, the tile stitching and fusion module includes: a location information acquisition submodule, used to acquire the location information of multiple sub-tiles; and a tile result stitching submodule, used to stitch and fuse the processing results of multiple sub-tiles based on the location information of the multiple sub-tiles.

[0016] Optionally, in this embodiment, the computation operation is an image scaling operation, and the memory operation is a non-pageable memory copy operation; the image processing apparatus further includes: a result stream synchronization submodule, used to stream synchronize the processing results of the image scaling operation and memory copy operation in each CUDA stream to obtain the processing results of multiple sub-tiles.

[0017] Optionally, in this embodiment of the application, the image scaling operation includes: performing a weighted average interpolation on the four pixel values ​​surrounding the target pixel position in the sub-block to obtain the scaled pixel value of the target pixel position in the sub-block.

[0018] Optionally, in this embodiment, the number of multiple CUDA streams is the same as the number of multiple sub-tiles, and different sub-tiles of the multiple sub-tiles are assigned to different CUDA streams.

[0019] This application also provides an electronic device, including a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions are executed by the processor to perform the methods described above.

[0020] This application also provides a computer-readable storage medium storing a computer program, which is executed by a processor to perform the methods described above.

[0021] This application also provides a computer program product, including: a computer program or computer instructions, which are executed by a processor to perform the method described above. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 The illustration shown is a schematic flowchart of the image processing method provided in an embodiment of this application;

[0024] Figure 2The illustration shows a comparison diagram of the default stream and the custom stream provided in the embodiments of this application;

[0025] Figure 3 The illustration shows a schematic diagram of the execution flow of multiple CUDA streams provided in an embodiment of this application;

[0026] Figure 4 The diagram shown is a structural schematic of the image processing apparatus provided in an embodiment of this application;

[0027] Figure 5 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in the embodiments of this application are for illustrative and descriptive purposes only and are not intended to limit the protection scope of the embodiments of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in the embodiments of this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of the embodiments of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0029] Furthermore, the described embodiments are merely a part of the embodiments of this application, and not all of them. The components of the embodiments of this application described and illustrated herein can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments of this application, but merely to illustrate selected embodiments of this application.

[0030] It is understood that the terms "first" and "second" in the embodiments of this application are used to distinguish similar objects. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different. In the description of the embodiments of this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. The term "multiple" refers to two or more (including two), and similarly, "multiple groups" refers to two or more groups (including two groups).

[0031] It should be noted that the image processing method provided in this application embodiment can be executed by an electronic device. Here, an electronic device refers to a device terminal or server with the function of executing computer programs. Device terminals include, for example, smartphones, personal computers, tablets, personal digital assistants, or mobile internet devices. A server refers to a device that provides computing services through a network. Servers include, for example, x86 servers and non-x86 servers. Non-x86 servers include, for example, mainframes, minicomputers, and UNIX servers.

[0032] The following section introduces application scenarios for this image processing method. Scaling high-resolution images is a common task in fields such as satellite imagery, medical imaging, and computer vision. This task primarily aims to change the image size without significant loss of detail or distortion. Currently, processing high-resolution images typically uses traditional image scaling algorithms implemented using a Central Processing Unit (CPU). However, their serial computation mode is insufficient to meet the real-time processing requirements of high-resolution images. For example, when processing a 38400×21600 pixel image, the CPU needs to perform pixel-by-pixel calculations, resulting in processing times of several minutes or even longer.

[0033] To meet the massive data processing demands of high-resolution images, the industry has gradually ported scaling algorithms to the graphics processing unit (GPU) platform, leveraging its massively parallel architecture to accelerate computation. However, this GPU implementation is typically based on a single Compute Unified Device Architecture (CUDA) stream or OpenCL command queue, resulting in low overall processing efficiency for high-resolution images.

[0034] For the above issues, please refer to Figure 1 The illustrated diagram shows a flowchart of the image processing method provided in this application embodiment. The main idea of ​​this image processing method is to fully utilize the multi-core resources of the GPU by asynchronously processing computational operations and memory operations in different CUDA streams, reducing the idle time of processing units, thereby maximizing the resource utilization of CUDA streams and improving the overall processing efficiency of high-resolution images. Implementation methods of the above image processing method may include:

[0035] Step S110: Divide the image to be processed into multiple sub-blocks.

[0036] The image to be processed refers to the target image that needs to be processed, such as an 8K resolution image (38400*21600 pixels).

[0037] A sub-image block is a local region of an image to be processed. By dividing a large image into multiple smaller, independent regions (such as horizontal blocks, vertical blocks, or grid blocks), each sub-image block can be processed independently (such as computational operations and memory operations).

[0038] Step S120: Process multiple sub-tiles using multiple unified computing device architecture CUDA streams. Each CUDA stream includes computation operations and memory operations. The computation operations of one sub-tile in the first CUDA stream and the memory operations of another sub-tile in the second CUDA stream are processed asynchronously.

[0039] Compute Unified Device Architecture (CUDA) is a computing platform and programming model developed by NVIDIA. In the field of image processing, CUDA can be used to distribute image scaling tasks to thousands of GPU cores for execution, rather than relying on serial processing by the CPU.

[0040] A CUDA stream is a series of operations (such as memory operations and / or computation operations) executed sequentially on a GPU. It can be understood as a queue of operation instructions. Operations within the same CUDA stream are executed synchronously; that is, while one operation is executing within a CUDA stream, other operations must wait for the previous operation to complete. Operations between different streams are executed asynchronously, thus achieving overlap between computation and transmission. A CUDA stream includes both computation operations and memory operations; that is, both computation and memory operations are contained within a specific CUDA stream.

[0041] A computation operation refers to the process of performing image processing algorithms (such as image scaling or filtering algorithms) on a sub-tile. For example, in the scaling of a sub-tile, the nearest neighbor interpolation algorithm or bilinear interpolation algorithm can be used to calculate the value of each target pixel.

[0042] Memory operations refer to operations related to GPU memory or CPU memory, such as copying between GPU memory and CPU memory, or copying between CPU memory and hard disk.

[0043] Asynchronous processing refers to the concurrent execution of operations in different streams without waiting for each other to complete. For example, in multiple CUDA streams, Stream A can execute computation operations on subplot A while Stream B can simultaneously execute memory operations on subplot B.

[0044] Step S130: The processing results of multiple sub-plots are stitched together and fused to obtain the processing result of the image to be processed.

[0045] Merging fusion is the process of recombining the processed results of multiple sub-tiles according to their positions in the original image to form a complete processed image as the result.

[0046] In the implementation of the above scheme, the image to be processed is divided into multiple sub-plots. The asynchronous processing of computation and memory operations in the CUDA streams of multiple unified computing device architectures is utilized to process the multiple sub-plots. Then, the processing results of the multiple sub-plots are stitched and fused together. By using the asynchronous processing between computation and memory operations in different CUDA streams, the multi-core resources of the GPU are fully utilized, and deep pipelined computation and memory operations are achieved. This ensures that the GPU's computing cores and memory bandwidth are almost always active, greatly reducing the idle time of the processing units and maximizing the resource utilization of the CUDA streams. Therefore, the overall processing efficiency of high-resolution images is improved.

[0047] As an optional implementation of step S110 above, for example, assuming the image to be processed is an 8K resolution image (38400*21600 pixels), the image can be divided into 100 sub-tiles. The specific division method can be horizontal division, vertical division, or grid-like division, etc., to obtain horizontal blocks, vertical blocks, or grid-like blocks. Each sub-tile is 1920×1080 pixels in size, and each sub-tile independently completes calculation operations (such as scaling or filtering) in subsequent processing. The number of sub-tiles can be determined or configured based on factors such as GPU stream count, video memory size, and resolution. Furthermore, while dividing the image into multiple sub-tiles, the position information of multiple sub-tiles can be recorded. For example, assuming the original image is divided into n*m sub-tiles, the position encoding information for each sub-tile can be based on its horizontal and vertical coordinates in the original image, using [0,0] to [n-1,m-1].

[0048] Please see Figure 2The diagram illustrates a comparison between the default stream and the custom stream provided in this embodiment of the application. Assume a CUDA stream includes: a copy operation (A, i.e., copy A or A in the diagram), a copy operation (B, i.e., copy B or B in the diagram), an image scaling operation (R, i.e., resize or R in the diagram), and a copy operation (C, i.e., copy C or C in the diagram). The aforementioned CUDA stream is divided into a globally synchronized default stream and a custom stream. A custom stream requires explicitly specifying which CUDA stream an operation will be executed in. If not specified, the operation will be executed on the default stream (see the upper part of the diagram). Referring to the lower part of the diagram, assuming the image to be processed is divided into tile 0, tile 1, tile 2, and tile 3, the copy operation A (A, i.e., A in the diagram), copy operation B (B, i.e. B in the diagram), image scaling operation R (R in the diagram), and copy operation C (C, i.e. C in the diagram) of tile 0 can be assigned to custom stream 0 for execution. Similarly, the relevant operations of tiles 1-3 can be assigned to custom streams 1-3 for execution. It is understandable that the computational and memory operations in the different streams of the custom streams 0-3 described above are implemented using different hardware. Therefore, the computational operations do not consume the bus resources of memory operations. Consequently, the CUDA pipeline formed by the custom streams 0-3 can improve the GPU's performance in image processing. The image scaling operation described above can employ a custom-implemented interpolation algorithm or the Resize interface in the standard library (such as the CV-CUDA library).

[0049] Please see Figure 3The diagram illustrates the execution flow of multiple CUDA streams provided in this embodiment of the application. It is understood that creating and initializing more CUDA streams in the GPU is not necessarily better, as the GPU consumes memory and computing resources during the creation and management of CUDA streams. Therefore, when the number of CUDA streams exceeds the optimal number, the overall efficiency of the GPU in processing high-resolution images will actually decrease. Thus, the process of determining the number of CUDA streams in the diagram typically involves statistically analyzing CPU and GPU performance metrics (e.g., latency) during experiments, and then determining the optimal number of CUDA streams for use in the final production environment based on these metrics. For ease of understanding and explanation, four streams are used as examples in the diagram: Stream0, Stream1, Stream2, and Stream3. Some GPUs include a Compute Engine (CPE), an H2D copy engine, and a D2H copy engine. Image processing operations are performed on the Compute Engine (CPE), and these operations can be image resizing calculations. In the diagram, CPE can represent either the Compute Engine itself or the computation being performed on it. Correspondingly, image copy operations can be performed on the H2D copy engine and / or the D2H copy engine. In the diagram, H2D and / or D2H can represent either the copy engine itself or the copy operation being performed on it. Because the H2D and D2H copy engines are separate engines, they allow H2D and D2H memory operations from different CUDA streams to be asynchronously and concurrently executed on the same GPU. In some implementations, scaling calculations, H2D operations, and D2H operations in different CUDA streams can be executed asynchronously and concurrently. For example, while the first CUDA stream is performing a scaling calculation operation on a sub-tile, the second CUDA stream can simultaneously asynchronously upload data of another sub-tile from the host's CPU memory to the GPU's video memory (H2D operation). Even the third CUDA stream can simultaneously asynchronously download the processed sub-tile results from the GPU's video memory back to the host's CPU memory (D2H operation). This complete overlap of computation and transmission not only avoids the waste of the GPU idling while waiting for data copying in traditional single-stream mode, but also maximizes the utilization of hardware resources (such as compute units, DMA channels, and PCIe buses) through multi-stream concurrency.

[0050] As an optional implementation of step S120 above, the memory operation may include: an H2D operation where the CPU memory of the host is copied to the GPU memory of the device, and a D2H operation where the GPU memory of the device is copied to the CPU memory of the host; the implementation of processing multiple sub-tiles using multiple unified computing device architecture CUDA streams may include:

[0051] Step S121: Process multiple sub-tiles using an H2D operation in one of the multiple CUDA streams and a D2H operation in another CUDA stream.

[0052] For example, the above-mentioned multiple CUDA streams are executed asynchronously and in an overlapping manner. For instance, the H2D operation in the Stream3 stream and the D2H operation in the Stream0 stream are executed asynchronously and in an overlapping manner. Similarly, the H2D operation in the Stream3 stream and the D2H operation in the Stream1 stream are executed asynchronously and in an overlapping manner.

[0053] And / or, the above-described implementation of processing multiple sub-tiles using multiple unified computing device architecture CUDA streams may further include:

[0054] Step S122: Process multiple sub-tiles using H2D operations in one of the multiple CUDA streams and computation operations in another CUDA stream.

[0055] The implementation of step S122 above is as follows: the multiple CUDA streams are executed asynchronously and overlappingly. For example, the H2D operation in Stream1 and the CPE calculation operation in Stream0 are executed asynchronously and overlappingly. Similarly, the H2D operation in Stream2 and the CPE calculation operation in Stream1 are executed asynchronously and overlappingly. Similarly, the H2D operation in Stream3 and the CPE calculation operation in Stream2 are executed asynchronously and overlappingly. The H2D operation can be performed by calling the cudaMemcpyAsync function through the H2D copy engine, thereby asynchronously copying sub-tiles from CPU memory to GPU memory.

[0056] And / or, the above-described implementation of processing multiple sub-tiles using multiple unified computing device architecture CUDA streams may further include:

[0057] Step S123: Process multiple sub-tiles using a D2H operation in one of the multiple CUDA streams and a computation operation in another CUDA stream.

[0058] For example, the above-described steps S123 can be implemented as follows: the multiple CUDA streams are executed asynchronously and in an overlapping manner. For instance, the D2H operation in Stream0 and the CPE calculation operation in Stream1 are executed asynchronously and in an overlapping manner. Similarly, the D2H operation in Stream1 and the CPE calculation operation in Stream2 are executed asynchronously and in an overlapping manner. Likewise, the D2H operation in Stream2 and the CPE calculation operation in Stream3 are executed asynchronously and in an overlapping manner. The CPE calculation operation can be performed by submitting operation instructions to the computation engine through the kernel function after the kernel function is started, so that the computation engine can execute the operation instructions.

[0059] As an optional implementation of step S130 above, one implementation of stitching and fusing the processing results of multiple sub-plots may include:

[0060] Step S131: Perform stream synchronization on the processing results of H2D operations, D2H operations, and computation operations in each of the multiple CUDA streams.

[0061] Step S132: If multiple sub-plots are processed synchronously, the processing results of the multiple sub-plots are spliced ​​and merged.

[0062] The implementation of steps S131 to S132 above is as follows: For each CUDA stream, the processing results of H2D operations, D2H operations, and computation operations are synchronized. That is, after all operations in all CUDA streams are completed, the processing results of multiple sub-tiles can be obtained. It is determined whether the processing results of multiple sub-tiles have been obtained through synchronized flow. If the processing results of multiple sub-tiles have been obtained through synchronized flow, the processing results of the multiple sub-tiles are then stitched together. The specific stitching and fusion method can be found in the description below.

[0063] As an optional implementation of the above method, one implementation of stitching and fusing the processing results of multiple sub-plots may include:

[0064] Step S133: Obtain the location information of multiple sub-plots.

[0065] Step S134: The processing results of multiple sub-plots are spliced ​​and merged based on the position information of multiple sub-plots.

[0066] For example, after obtaining the processing results (e.g., scaling results) of multiple sub-blocks, the processing results of each sub-block can be stitched together. At this time, the position information of multiple sub-blocks can be obtained first, and then the processing results of multiple sub-blocks can be stitched together according to the position information of multiple sub-blocks.

[0067] As an optional implementation of the above image processing method, the above calculation operation can be an image scaling operation, and the above memory operation can be a non-pageable memory copy operation; before stitching and fusing the processing results of multiple sub-tiles, the method further includes:

[0068] Step S135: Stream synchronization is performed on the processing results of image scaling and memory copy operations in each CUDA stream to obtain the processing results of multiple sub-tiles.

[0069] For example, in implementing step S135 above, before stitching and merging the processing results of multiple sub-tiles, since the completion order of the multiple CUDA streams of the GPU is not specific, the processing results of image scaling operations and memory copy operations in each CUDA stream can be synchronized first. This ensures that all CUDA streams of the GPU are completed before obtaining the processing results of multiple sub-tiles. It is understood that the memory corresponding to the aforementioned non-pageable memory copy operation can be requested from host memory through the cudaHostAlloc function. This non-pageable memory is also called pinned memory. The physical address of this memory is contiguous and will not be replaced by the operating system's paging mechanism (i.e., the physical address of the memory remains unchanged during program execution).

[0070] Optionally, the image scaling operation described above can also be performed using nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, Lanczos interpolation, or super-resolution algorithms. Among these, the nearest neighbor interpolation algorithm, when estimating the value of a point, selects the nearest known data point and returns the value of that data point. The nearest neighbor interpolation algorithm described above can be represented as follows: in, and These represent the x and y coordinates of the sub-tile's position at the scaled target pixel location, respectively. and These represent the length and width of the sub-tile after scaling. and These represent the x and y coordinates of the sub-tile's pixel position before scaling, respectively. and These represent the length and width of the sub-tile before scaling, respectively. In other words, nearest neighbor interpolation does not perform a weighted average calculation; instead, it directly uses the value of the data point closest to the target point. This simple method of selecting the nearest point can easily lead to significant errors in the interpolation results when encountering data with large variations or irregular objects, and may even result in a "blocky" or mosaic effect. Other bilinear interpolation algorithms, bicubic interpolation algorithms, Lanczos interpolation algorithms, or super-resolution algorithms have higher algorithmic complexity and require a considerable amount of time to complete the scaling effect when using the aforementioned GPUs. Therefore, image scaling algorithms struggle to balance image quality and performance efficiency after scaling.

[0071] As an optional implementation of step S135 above, in order to balance the image quality and performance efficiency after scaling, the implementation of the image scaling operation may include:

[0072] Step S135a: Perform a weighted average interpolation on the four pixel values ​​surrounding the target pixel position in the sub-block to obtain the scaled pixel value of the target pixel position in the sub-block.

[0073] For example, the above step S135a can be implemented by using the following formula to perform a weighted average interpolation on the four pixel values ​​surrounding the target pixel position in the sub-block, thereby obtaining the scaled pixel value of the target pixel position in the sub-block.

[0074]

[0075]

[0076] in, and These represent the x and y coordinates of the sub-tile's position at the scaled target pixel location, respectively. , (), , (), , (), , ) represent the x and y coordinates of the four pixels surrounding the target pixel in the atomic tile before scaling, respectively. D represents the sub-tile in the scaled target image, and S represents the sub-tile in the original image before scaling.

[0077] In the above implementation process, by performing weighted average interpolation on the four pixel values ​​surrounding the target pixel position in the sub-block, this local weighting method (i.e., only increasing the amount of computation by a small amount) significantly improves the image processing quality while also improving the performance efficiency of image processing, effectively balancing the image quality and performance efficiency of image processing.

[0078] As an optional implementation of the above image processing method, the number of CUDA streams and the number of sub-tiles can be the same. That is, after determining the number of CUDA streams, the number of sub-tiles to be divided can be set to be the same as the number of CUDA streams. Specifically, CPU performance metrics (e.g., time consumption) are calculated during the experiment, and then the optimal number of CUDA streams for use in the final production environment is determined based on the GPU performance metrics. Finally, the number of CUDA streams is simultaneously set to the number of sub-tiles in the configuration file, so that when the image to be processed is divided into multiple sub-tiles, it is divided into that number of sub-tiles. Different sub-tiles of the above-mentioned multiple sub-tiles can be assigned to different CUDA streams.

[0079] In implementing the above scheme, by assigning each sub-tile its own CUDA stream instead of sharing a stream, operations such as H2D / D2H memory copying and image scaling calculations for each sub-tile are executed in parallel within completely independent streams, thus completely eliminating resource contention and implicit dependencies between stream tasks. For example, in traditional multi-stream models, multiple sub-tiles may share a small number of streams (e.g., 4 streams), leading to serialization of tasks within a stream or insufficient utilization of hardware resources due to stream scheduling policy limitations. In contrast, this scheme ensures that the number of sub-tiles is strictly consistent with the number of streams, guaranteeing that each sub-tile's operations obtain dedicated DMA channels, computing units, and PCIe bandwidth in an independent stream, maximizing the concurrency of hardware resources. On high-end GPUs that support a sufficient number of streams (such as the NVIDIA A100), this design can increase image processing throughput from 80% utilization in traditional multi-stream modes to a theoretical peak close to 100%. Especially when processing high-resolution images, it can compress the overall processing time from a linear superposition of "serial computation + transmission" modes to approximately the processing time of a single sub-tile, thereby achieving a super-linear speedup.

[0080] Please see Figure 4 The diagram shown is a structural schematic of an image processing apparatus provided in an embodiment of this application; this application provides an image processing apparatus 200, including:

[0081] The image segmentation module 210 is used to divide the image to be processed into multiple sub-blocks.

[0082] The sub-tile stream processing module 220 is used to process multiple sub-tiles using multiple unified computing device architecture CUDA streams. Each CUDA stream includes computation operations and memory operations. The computation operations of one sub-tile in the first CUDA stream and the memory operations of another sub-tile in the second CUDA stream are processed asynchronously.

[0083] The tile stitching and fusion module 230 is used to stitch and fuse the processing results of multiple sub-tiles to obtain the processing result of the image to be processed.

[0084] As an optional implementation of the above-mentioned device, the memory operations include: an H2D operation of copying data from the CPU memory to the GPU memory, and a D2H operation of copying data from the GPU memory to the CPU memory; the sub-tile stream processing module includes:

[0085] The first processing submodule is used to process multiple sub-tiles using H2D operations in one CUDA stream and D2H operations in another CUDA stream.

[0086] And / or, a second processing submodule, for processing multiple sub-tiles using H2D operations in one of the multiple CUDA streams and computation operations in another CUDA stream.

[0087] And / or, a third processing submodule, for processing multiple sub-tiles using a D2H operation in one of the multiple CUDA streams and a computation operation in another CUDA stream.

[0088] As an optional implementation of the above-mentioned device, the tile stitching and fusion module includes:

[0089] The multi-stream synchronization submodule is used to synchronize the processing results of H2D operations, D2H operations, and computation operations in each of the multiple CUDA streams.

[0090] The result stitching and fusion submodule is used to stitch and fuse the processing results of multiple sub-plots if they are all processed synchronously.

[0091] As an optional implementation of the above-mentioned device, the tile stitching and fusion module includes:

[0092] The location information acquisition submodule is used to acquire the location information of multiple sub-tiles.

[0093] The tile result stitching submodule is used to stitch and merge the processing results of multiple sub-tiles based on their location information.

[0094] As an optional embodiment of the above-mentioned apparatus, the computational operation is an image scaling operation, and the memory operation is a non-pageable memory copy operation; the image processing apparatus further includes:

[0095] The result stream synchronization submodule is used to synchronize the processing results of image scaling and memory copy operations in each CUDA stream to obtain the processing results of multiple sub-tiles.

[0096] As an optional implementation of the above-described device, the image scaling operation includes:

[0097] The target pixel position in the sub-plot is obtained by performing a weighted average interpolation on the four pixel values ​​surrounding it. The target pixel position in the sub-plot is then scaled.

[0098] As an optional implementation of the above-described apparatus, the number of multiple CUDA streams is the same as the number of multiple sub-tiles, and different sub-tiles of the multiple sub-tiles are assigned to different CUDA streams.

[0099] It should be understood that this device corresponds to the image processing method embodiments described above and is capable of performing the various steps involved in the above method embodiments. The specific functions of this device can be found in the description above, and detailed descriptions are appropriately omitted here. The device includes at least one software functional module that can be stored in memory or embedded in the device's operating system (OS) in the form of software or firmware.

[0100] Please see Figure 5 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 300 provided in this application includes a processor 310 and a memory 320. The memory 320 stores machine-readable instructions executable by the processor 310. When the machine-readable instructions are executed by the processor 310, the method described above is performed.

[0101] This application embodiment also provides a computer-readable storage medium 330, on which a computer program is stored. This computer program is executed by a processor 310 to perform the methods described above. The computer-readable storage medium 330 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0102] This application also provides a computer program product, including: a computer program or computer instructions, which are executed by a processor to perform the method described above.

[0103] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0104] It should be understood that the disclosed apparatus and methods can also be implemented in other ways, as provided in the embodiments of this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending primarily on the functions involved.

[0105] Furthermore, the functional modules of each embodiment in this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the reference to terms such as "one embodiment," "some embodiments," "example," "specific example," "some examples," etc., means that the specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0106] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. An image processing method, characterized in that, include: Divide the image to be processed into multiple sub-plots; The multiple sub-tiles are processed using multiple unified computing device architecture CUDA streams. Each CUDA stream includes computation operations and memory operations. The computation operations of one sub-tile in the first CUDA stream and the memory operations of another sub-tile in the second CUDA stream are processed asynchronously. The processing results of the multiple sub-image patches are stitched and fused together to obtain the processing result of the image to be processed; The memory operations include: an H2D operation copying memory from the CPU to the GPU memory, and a D2H operation copying memory from the GPU to the CPU memory; the processing of the multiple sub-tiles using multiple Unified Computing Device Architecture (CUDA) streams includes: processing the multiple sub-tiles using an H2D operation from one CUDA stream and a D2H operation from another CUDA stream; and / or, processing the multiple sub-tiles using an H2D operation from one CUDA stream and a computation operation from another CUDA stream; and / or, processing the multiple sub-tiles using a D2H operation from one CUDA stream and a computation operation from another CUDA stream. The number of the multiple CUDA streams is the same as the number of the multiple sub-tiles, and the different sub-tiles of the multiple sub-tiles are assigned to different CUDA streams.

2. The method according to claim 1, characterized in that, The process of stitching and fusing the processing results of the multiple sub-plots includes: Stream synchronization is performed on the processing results of the H2D operation, the D2H operation, and the computation operation in each of the multiple CUDA streams; If the processing of the multiple sub-plots is completed synchronously, the processing results of the multiple sub-plots are then spliced ​​and merged.

3. The method according to claim 1, characterized in that, The process of stitching and fusing the processing results of the multiple sub-plots includes: Obtain the position information of the multiple sub-plots; The processing results of the multiple sub-plots are spliced ​​and merged based on their position information.

4. The method according to claim 1, characterized in that, The calculation operation is an image scaling operation, and the memory operation is a non-pageable memory copy operation; Before stitching and merging the processing results of the multiple sub-plots, the process also includes: The processing results of the image scaling operation and the memory copy operation in each CUDA stream are stream synchronized to obtain the processing results of the multiple sub-tiles.

5. The method according to claim 4, characterized in that, The image scaling operation includes: The target pixel position in the sub-block is obtained by performing a weighted average interpolation on the four pixel values ​​surrounding it. The target pixel position in the sub-block is then scaled.

6. An image processing apparatus, characterized in that, include: The image segmentation module is used to divide the image to be processed into multiple sub-blocks; The sub-tile stream processing module is used to process the multiple sub-tiles using multiple unified computing device architecture CUDA streams. Each of the multiple CUDA streams includes: computation operations and memory operations. The computation operations of one sub-tile in the first CUDA stream and the memory operations of another sub-tile in the second CUDA stream are processed asynchronously. The image patch stitching and fusion module is used to stitch and fuse the processing results of the multiple sub-image patches to obtain the processing result of the image to be processed; The memory operations include: H2D operation of copying CPU memory to GPU memory, and D2H operation of copying GPU memory to CPU memory. The sub-plot stream processing module includes: The first processing submodule is used to process the multiple sub-tiles using an H2D operation in one of the multiple CUDA streams and a D2H operation in another CUDA stream; And / or, a second processing submodule, for processing the plurality of sub-tiles using an H2D operation in one of the plurality of CUDA streams and a computation operation in another CUDA stream; And / or, a third processing submodule, for processing the plurality of sub-tiles using a D2H operation in one of the plurality of CUDA streams and a computation operation in another CUDA stream; The number of the multiple CUDA streams is the same as the number of the multiple sub-tiles, and the different sub-tiles of the multiple sub-tiles are assigned to different CUDA streams.

7. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, the machine-readable instructions being executed by the processor to perform the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the method according to any one of claims 1 to 5.

9. A computer program product, characterized in that, include: A computer program or computer instructions that, when executed by a processor, perform the method described in any one of claims 1 to 5.

Images