Industrial image processing system and method based on multi-thread and GPU memory resource pool

Abstract

The application belongs to the technical field of industrial image processing, and particularly relates to an industrial image processing system and method based on multi-thread and GPU memory resource pool, which solves the technical defects of low throughput, high delay and insufficient utilization of computing resources of the existing image processing system, and the system comprises an abnormality processing and monitoring module, a request processing module, a thread pool module, a preprocessing module, a post-processing module, a GPU inference module and a GPU memory resource pool module; the system adopts a multi-thread receiving and processing mechanism to realize task-level parallelism, realizes data-level parallelism through multi-thread optimization of image preprocessing and result post-processing processes, designs a dynamically managed GPU memory resource to realize decoupling, reuse, concurrency and parallelism of memory resources and inference resources, taps the concurrency and parallel computing potential of modern computers and servers, effectively meets the real-time processing requirements of massive industrial image data, significantly improves the system throughput, and greatly reduces the processing delay.

Description

Technical Field

[0001] This invention relates to the field of industrial image processing technology, and in particular to an industrial image processing system and method based on multithreading and GPU memory resource pools. Background Technology

[0002] Industrial image processing technology has been widely applied in fields such as intelligent manufacturing, product quality inspection, automated production monitoring, and medical image analysis. With the advancement of Industry 4.0 and intelligent manufacturing, the scale of image data generated by industrial cameras, sensors, and other equipment is growing exponentially, placing higher demands on real-time performance, accuracy, and processing efficiency. In typical application scenarios such as industrial quality inspection and assembly line monitoring, image acquisition equipment can generate thousands or even tens of thousands of high-resolution images per second, resulting in massive data volumes and stringent timeliness requirements for processing.

[0003] Existing image processing methods often struggle to meet the real-time and high-throughput requirements of modern industrial scenarios when dealing with high-resolution, multi-channel, and massive industrial images. Current image processing systems typically employ fixed batch processing, single-task pipeline modes, or simple multi-threaded modes, resulting in low utilization of computing resources, severe task backlog, and high overall processing latency. For example, in semiconductor wafer defect detection, even millisecond-level latency increases can cause quality inspection delays for entire batches of products, severely impacting production efficiency.

[0004] In recent years, computer hardware performance has advanced rapidly, especially with advancements in GPUs, multi-core CPUs, and high-speed storage technologies providing the foundation for high-performance computing. However, many existing image processing systems still employ coarse-grained task allocation strategies, failing to fully utilize heterogeneous computing resources and resulting in idle hardware computing power. For example, traditional systems may leave the CPU in a waiting state while the GPU is performing computations, or bottlenecks may occur in the CPU-GPU data transfer process, preventing an overall increase in throughput.

[0005] In summary, industrial image processing systems urgently need a highly efficient parallel architecture that can maximize hardware resource utilization and achieve overlap between computation and transmission in order to meet the high throughput and low latency requirements of industrial image processing. Summary of the Invention

[0006] To overcome the technical shortcomings of existing image processing systems, such as low throughput, high latency, and insufficient utilization of computing resources, this invention provides an industrial image processing system and method based on multi-threading and GPU memory resource pools.

[0007] This invention provides an industrial image processing system based on multithreading and GPU memory resource pools, including an exception handling and monitoring module, a request processing module, a thread pool module, a preprocessing module, a postprocessing module, a GPU inference module, and a GPU memory resource pool module. The request processing module supports multiple communication protocols such as HTTP / RESTful API, gRPC, and WebSocket. It receives client requests and image data with high concurrency, parses request parameters to determine the task type, and returns the final processing result or output data from the postprocessing module to the client. The thread pool module manages a group of worker threads through pooling technology to asynchronously and concurrently execute image processing tasks issued by the request processing module, achieving task scheduling and resource reuse. The preprocessing module executes standardized preprocessing operations on the input image in parallel. During system initialization, the GPU memory resource pool module pre-allocates and manages a set of structured memory resource blocks in the GPU device memory and host memory. The GPU memory resource pool module provides an atomic, lock-free allocation interface for the GPU. The inference module requests and releases memory resources on demand; the GPU inference module calls memory resources allocated by the GPU memory resource pool module, loads preprocessed image data into the GPU, executes inference calculations of deep learning models or algorithms, and writes the resulting results into the corresponding output memory area; the post-processing module processes and integrates the output data of the GPU inference module in parallel on the CPU, including result confidence filtering, overlapping box deduplication, cross-batch result merging, and data structure formatting, to generate the final deliverable results; the exception handling and monitoring module is used for the observability and fault tolerance of the entire system link, records operation logs, monitors the health status of each module, and triggers recovery mechanisms and alarm notifications when task execution times out, hardware failures, or system anomalies occur.

[0008] The system described in this invention employs a multi-threaded receiving and processing mechanism to achieve task-level parallelism. It optimizes image preprocessing and post-processing workflows through multi-threading to achieve data-level parallelism. Furthermore, it innovatively designs dynamically managed GPU memory resources to decouple, reuse, and concurrently parallelize memory and inference resources. This architecture fully leverages the concurrent and parallel computing potential of modern computers and servers, effectively addressing the real-time processing needs of massive industrial image data. While significantly improving system throughput, it drastically reduces processing latency, providing a high-performance solution for industrial image processing.

[0009] Preferably, the request processing module defines a unified service protocol in Protocol Buffers format, which is used to receive image processing requests submitted by clients. The request message includes the task ID, raw image data, image size, task type, and processing parameters. After receiving the request, the request processing module first performs protocol parsing and data deserialization, then distributes it to the internal processing pipeline according to the task type, and encapsulates the structured result into a response message and returns it to the client after processing. The request processing module has a built-in connection pool and request queue, supporting the processing of thousands of concurrent connections per second.

[0010] Preferably, the thread pool module is implemented using the BSThreadPool thread pool based on C++17. During system initialization, a fixed number of worker threads are dynamically created based on the number of CPU physical cores. The thread pool maintains a lock-free task queue, supporting asynchronous task submission and load-balanced scheduling. Each worker thread is bound to an independent GPU context and memory resource handle during its lifecycle. The thread pool module provides a task priority scheduling mechanism, allowing high-priority tasks to be executed in the queue, while monitoring the health status of each thread and automatically restarting abnormal threads.

[0011] Preferably, the preprocessing module integrates OpenCV 4.5 and FFmpeg libraries, and uses Intel TBB and OpenMP to achieve data-level parallel acceleration. The preprocessing operations it performs include video stream decoding and frame extraction using FFmpeg, image color space conversion using OpenCV, size scaling, histogram equalization, Gaussian filtering for noise reduction, and local contrast enhancement based on CLAHE.

[0012] Preferably, the GPU memory resource pool module pre-allocates a fixed number of memory resource blocks on the CPU and GPU sides when the system starts. Each memory resource block includes a GPU-side input buffer, a GPU-side output buffer, and a corresponding CPU-side fixed memory buffer. The memory resource blocks are managed through a specific data structure, and the GPU memory resource pool module uses atomic operations to achieve lock-free allocation and reclamation.

[0013] Preferably, the GPU inference module is built on TensorRT 8.5 and is responsible for executing the optimized deep learning model inference. During the initialization phase, the GPU inference module loads pre-compiled TensorRT engine files from the disk. Each TensorRT engine file corresponds to a specific model and batch size configuration.

[0014] Preferably, the post-processing module is implemented based on OpenCV and custom C++ operators. It performs data-level parallel post-processing on the raw results returned by the GPU inference module. The post-processing module uses TBB for parallel loop optimization. Its processing steps include: 1. Confidence filtering: filtering out results with low confidence according to a preset threshold; 2. Result fusion: performing coordinate mapping and deduplication merging on the results of multi-scale or block inference; 3. Formatted output.

[0015] This invention also discloses an industrial image processing method based on multi-threading and GPU memory resource pools, which is implemented based on the industrial image processing system based on multi-threading and GPU memory resource pools described in this invention, and includes the following steps:

[0016] S1. System Deployment:

[0017] The request processing module, thread pool module, preprocessing module, postprocessing module, GPU inference module, GPU memory resource pool module, and exception handling and monitoring module are integrated and deployed on the server in a loosely coupled and standardized manner to complete the construction of the system software environment.

[0018] S2, System Initialization:

[0019] Each module performs startup and memory initialization operations. The thread pool module creates and maintains a thread pool containing n worker threads based on a preset concurrency level. The GPU memory resource pool module pre-allocates several fixed-size contiguous storage areas in the GPU memory and allocates a fixed-size buffer of equal size for each contiguous storage area in the CPU memory, establishing a one-to-one mapping "GPU-CPU" memory pair for inference data input and result feedback. The exception handling and monitoring module starts synchronously and begins to record the system's running status and resource usage in real time.

[0020] S3, High-concurrency request reception and parsing:

[0021] The thread pool module uses a lock-free scheduling strategy to allocate an idle worker thread. The idle worker thread will be bound to and responsible for handling a single client request until the final result is returned, achieving request-level isolation and resource tracking. In the current worker thread, the request processing module receives the image processing request and the attached image data stream initiated by the client, parses the request message, extracts the parameters of task type, image format or processing requirements, and obtains the raw image data to be processed.

[0022] S4. Parallel preprocessing of image data:

[0023] Following step S3, the thread selects the appropriate processing flow from the configurable preprocessing strategy library based on the parsed task type. Subsequently, the preprocessing module is called to perform data-level parallel standardized operations on the image data, including deserialization, size normalization, color space conversion, noise filtering, and image enhancement, ultimately obtaining target tensor data that meets the model input requirements.

[0024] S5. Pipeline inference based on GPU memory pool:

[0025] S5.1 CPU to GPU data transfer: The preprocessed target tensor data is asynchronously copied from the CPU memory to the contiguous storage area allocated by the GPU memory resource pool module, and the CPU memory is released for subsequent reuse.

[0026] S5.2, GPU Parallel Inference: The GPU inference module locks the corresponding contiguous storage area, and calls the corresponding deep learning model or algorithm on the GPU inference module to perform forward inference calculation;

[0027] S5.3, GPU to CPU data transfer: After inference is completed, the output result is asynchronously copied from the contiguous storage area allocated by the GPU memory resource pool module back to the corresponding fixed buffer on the CPU side, and the contiguous storage area allocated by the GPU memory resource pool module is released. The scheduling is carried out through the synchronization event mechanism in the GPU memory resource pool module. The three stages between different inference tasks can achieve time overlap, that is, when the GPU is executing the current task calculation, the CPU is simultaneously performing data transfer for the next task and result return for the previous task.

[0028] S6. Parallel post-processing of inference results:

[0029] The post-processing module is invoked to perform data-level parallel post-processing operations on the inference results that have been sent back to the fixed buffer on the CPU. The post-processing operations include confidence threshold filtering, non-maximum suppression deduplication, multi-scale result fusion, and data format standardization, and finally generate the output results.

[0030] S7. Result Return and Resource Recovery:

[0031] The current worker thread returns the formatted final result to the corresponding CPU through the request processing module. After the request is processed, the current worker thread releases all the temporary resources it occupies and actively returns to the thread pool, marking it as available, waiting for the next task allocation, thus realizing the recycling and reuse of resources.

[0032] S8. Dynamic Resource Adjustment and System Monitoring:

[0033] During system operation, the exception handling and monitoring module analyzes system load, queue length, and hardware status in real time; the GPU memory resource pool module dynamically adjusts the number and size of memory blocks based on the monitored data to adapt to the memory requirements of different batches and models; the thread pool size automatically expands and shrinks according to the current request queue length to achieve elastic scaling and load balancing; when a task timeout, hardware anomaly, or data error is detected, the system automatically triggers a recovery strategy and issues an alarm.

[0034] The technical solution provided by this invention has the following technical effects compared with the prior art:

[0035] I. Synergistic Optimization of High Throughput and Low Latency: By decoupling the mutual transmission between CPU memory and GPU memory through a pipelined structure, the overlap of computation and communication is achieved, reducing the GPU idle waiting time in the traditional serial process to near zero. Combined with the pre-allocation and reuse mechanism of the GPU memory resource pool module, the overhead and fragmentation caused by dynamic memory allocation are avoided. As a result, when the system is processing high-throughput scenarios such as industrial images, the single-path inference latency is significantly reduced, and the overall throughput can be increased to 2-3 times that of traditional solutions.

[0036] II. Multi-level Parallelism and Efficient Resource Utilization: The system achieves triple parallelism at the data level (image preprocessing, post-processing), task level (multi-threaded concurrent processing), and device level (GPU pipeline). The thread pool module ensures efficient scheduling of CPU computing resources; the GPU memory resource pool module realizes centralized management and on-demand allocation of device memory, avoiding performance jitter caused by repeated allocation and release; the multi-level pipeline further squeezes the computing potential of GPU hardware, so that the CPU and GPU always maintain parallel load, and the overall system resource utilization rate can reach over 85%.

[0037] III. High Reliability and Maintainability: An anomaly handling and monitoring module is introduced to achieve end-to-end status tracking and fault self-healing. When a task times out, hardware malfunctions, or data anomalies occur, the system can automatically retry the task, reclaim resources, and report alarms, ensuring stable operation 24 / 7 in a continuous industrial environment. Furthermore, the modular design and configurable pre-processing / post-processing chain enable the system to quickly adapt to different algorithm models and business scenarios, significantly reducing deployment and iteration costs.

[0038] IV. Elastic Scaling and Load Balancing: The GPU memory resource pool module can dynamically adjust the number and size of memory blocks according to the actual load, and the thread pool size can also automatically expand or shrink according to the request queue length. This elastic mechanism enables the system to maintain a stable response during request surges and automatically releases excess resources during low loads, achieving load-adaptive resource scheduling, which is especially suitable for uneven task arrival scenarios in industrial production. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of the structure of an industrial image processing system based on multi-threading and GPU memory resource pool according to a certain embodiment of the present invention;

[0042] Figure 2 This is a flowchart of an industrial image processing method based on multi-threading and GPU memory resource pool according to a certain embodiment of the present invention. Detailed Implementation

[0043] To better understand the above-mentioned objectives, features, and advantages of the present invention, the solutions of the present invention will be further described below. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0044] Many specific details are set forth in the following description in order to provide a full understanding of the invention, but the invention may also be practiced in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of the invention, and not all embodiments.

[0045] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0046] In one embodiment, such as Figure 1As shown, an industrial image processing system based on multi-threading and GPU memory resource pools is disclosed. This industrial image processing system is deployed on a high-performance server, and its hardware configuration includes: dual Intel Xeon Platinum 8490H processors, totaling 120 physical cores and 240 logical threads; 1TB DDR5-5600 RDIMMECC memory; eight NVIDIA H100 80GB PCIe graphics cards, each with 80GB HBM3 video memory, connected via two Samsung PM1743 15.36TB PCIe 5.0 NVMe hard drives; dual-port NVIDIA ConnectX-7 400GbE network cards and 32GbFC HBA cards; and dual 3000W power supplies. The software environment is based on the Ubuntu 20.04 LTS operating system, with CUDA 11.8 toolkit, cuDNN 8.6 acceleration library, and TensorRT 8.5 inference optimizer installed. This environment provides the necessary computing, storage, and communication capabilities for large-scale parallel image processing and deep learning inference.

[0047] The industrial image processing system includes an exception handling and monitoring module, a request processing module, a thread pool module, a preprocessing module, a postprocessing module, a GPU inference module, and a GPU memory resource pool module.

[0048] The request processing module supports multiple communication protocols including HTTP / RESTful API, gRPC, and WebSocket. It receives client requests and image data concurrently, parses request parameters to determine the task type, and returns the final processing result or output data from the post-processing module to the client. The thread pool module manages a group of worker threads using pooling technology to asynchronously and concurrently execute image processing tasks issued by the request processing module, achieving task scheduling and resource reuse. The preprocessing module performs standardized preprocessing operations on the input image in parallel, including but not limited to data deserialization, pixel depth conversion, image resizing, color space transformation, and image enhancement. During system initialization, the GPU memory resource pool module pre-allocates and manages a set of structured memory resource blocks in the GPU device memory and host memory. The GPU memory resource pool module provides atomic, lock-free allocation. The interface allows the GPU inference module to request and release memory as needed, ensuring efficient memory access and thread safety. The GPU inference module calls memory resources allocated by the GPU memory resource pool module, loads preprocessed image data into the GPU, executes inference calculations of deep learning models or algorithms, and writes the resulting results to the corresponding output memory area. The post-processing module processes and integrates the output data of the GPU inference module in parallel on the CPU, including result confidence filtering, overlapping box deduplication, cross-batch result merging, and data structure formatting, to generate the final deliverable results. The exception handling and monitoring module is used for the observability and fault tolerance of the entire system, records operation logs, monitors the health status of each module, and triggers recovery mechanisms and alarm notifications when task execution times out, hardware failures, or system anomalies occur, ensuring system reliability.

[0049] Based on the above embodiments, in a preferred embodiment, the request processing module implements a high-concurrency, low-latency remote service interface based on the gRPC framework, defining a unified service protocol in Protocol Buffers format, which is used to receive image processing requests submitted by clients. The request message includes task ID, raw image data (supporting formats such as JPEG, PNG, and BMP), image size, task type (such as defect detection, size measurement, and OCR recognition), and processing parameters. After receiving the request, the request processing module first performs protocol parsing and data deserialization, then distributes it to the internal processing pipeline according to the task type, and encapsulates the structured result into a response message and returns it to the client after processing. The request processing module has a built-in connection pool and request queue, supporting the processing of thousands of concurrent connections per second.

[0050] Based on the above embodiments, in a preferred embodiment, the thread pool module is implemented using the BSThreadPool thread pool based on C++17. During system initialization, a fixed number of worker threads are dynamically created based on the number of CPU physical cores, such as 128. The thread pool maintains a lock-free task queue, supporting asynchronous task submission and load-balanced scheduling. Each worker thread is bound to an independent GPU context and memory resource handle during its lifecycle. The thread pool module provides a task priority scheduling mechanism, allowing high-priority tasks (such as real-time detection) to be executed in the queue, while monitoring the health status of each thread and automatically restarting abnormal threads.

[0051] Based on the above embodiments, in a preferred embodiment, the preprocessing module integrates OpenCV 4.5 and FFmpeg libraries, utilizing Intel TBB and OpenMP for data-level parallel acceleration. The preprocessing operations it performs include video stream decoding and frame extraction using FFmpeg, image color space conversion using OpenCV (e.g., RGB to grayscale or YUV), size scaling (supporting bilinear, bicubic, and Lanczos interpolation), histogram equalization, Gaussian filtering for noise reduction, and CLAHE-based local contrast enhancement. All operations are executed in parallel in host memory, and the output is a standardized floating-point tensor that meets the input requirements of deep learning models, typically in NCHW (batch size, channels, height, width) format.

[0052] Based on the above embodiments, in a preferred embodiment, the GPU memory resource pool module pre-allocates a fixed number of memory resource blocks on both the CPU and GPU sides during system startup. Each memory resource block includes a GPU-side input buffer, a GPU-side output buffer, and a corresponding CPU-side host-pinned buffer. The memory resource blocks are managed through specific data structures, including pointers to the GPU input buffer, GPU output buffer, and host-pinned buffer, usage status flags, and bound CUDA streams. The GPU memory resource pool module uses atomic operations to achieve lock-free allocation and deallocation. When a task requests memory, the pool manager returns a free MemoryBlock and marks `in_use` as true; after the task completes, the mark is reset to false, and the memory block returns to the pool for later use, avoiding frequent `cudaMalloc` / `cudaFree` calls. The pool size can be dynamically adjusted according to the GPU memory capacity, typically set to an integer multiple of the maximum number of concurrent tasks supported multiplied by the memory required by the model.

[0053] Based on the above embodiments, in a preferred embodiment, the GPU inference module is built on TensorRT 8.5 and is responsible for executing the optimized deep learning model inference. During the initialization phase, the GPU inference module loads pre-compiled TensorRT engine files from disk. Each TensorRT engine file corresponds to a specific model and batch size configuration. During inference, the GPU inference module obtains a MemoryBlock from the GPU memory resource pool, binds the device buffer pointer within it to the TensorRT execution context, and calls the enqueueV2() method to start asynchronous inference.

[0054] Based on the above embodiments, in a preferred embodiment, the post-processing module is implemented using OpenCV and custom C++ operators to perform data-level parallel post-processing on the raw results returned by the GPU inference module. The post-processing module utilizes TBB for parallel loop optimization to ensure high throughput even when processing large batches of detection results. Its processing steps include: 1. Confidence filtering: filtering out results with low confidence based on a preset threshold (e.g., 0.5); 2. Result fusion: performing coordinate mapping and deduplication merging on results from multi-scale or block inference; 3. Formatted output: converting the final results into structured formats such as JSON, XML, or Protobuf, including result information such as category labels, confidence scores, bounding box coordinates, and segmentation masks (if applicable).

[0055] Based on the above embodiments, in a preferred embodiment, the anomaly handling and monitoring module includes: ① Monitoring metric collection: Real-time collection of GPU utilization, memory usage, temperature, and power consumption via NVML; CPU / memory usage, network I / O, and disk I / O via system calls; and statistics on request queue length, average latency, throughput, and error rate via internal counters. ② Alarming and visualization: Integration with Prometheus + Grafana for metric storage and dashboard display; configuration of AlertManager rules to trigger email / DingTalk alarms when GPU utilization remains below 10% or request latency exceeds 100ms while requests are in use. ③ Fault recovery strategy: The system has a multi-layered fault tolerance mechanism, including task-level timeout retry, automatic re-addition of individual tasks after timeout, resource isolation (automatic migration of tasks on abnormal GPU cards to other cards), health checks (periodic heartbeat monitoring of module status, automatic restart during idle periods and alarms in case of anomalies). ④ Log auditing: All operations and anomalies are structured and recorded in the ELK (Elasticsearch, Logstash, Kibana) stack, supporting distributed tracing and root cause analysis.

[0056] This invention also discloses an industrial image processing method based on multi-threading and GPU memory resource pools, which is implemented based on the industrial image processing system based on multi-threading and GPU memory resource pools described in this invention, and includes the following steps:

[0057] S1. System Deployment:

[0058] The request processing module, thread pool module, preprocessing module, postprocessing module, GPU inference module, GPU memory resource pool module, and exception handling and monitoring module are integrated and deployed on the server in a loosely coupled and standardized manner to complete the construction of the system software environment.

[0059] S2, System Initialization:

[0060] Each module performs startup and memory initialization operations. The thread pool module creates and maintains a thread pool containing n worker threads based on a preset concurrency level. The GPU memory resource pool module pre-allocates several fixed-size contiguous storage areas in the GPU memory and allocates a fixed-size buffer of equal size for each contiguous storage area in the CPU memory, establishing a one-to-one mapping "GPU-CPU" memory pair for inference data input and result feedback. The exception handling and monitoring module starts synchronously and begins to record the system's running status and resource usage in real time.

[0061] S3, High-concurrency request reception and parsing:

[0062] The thread pool module uses a lock-free scheduling strategy to allocate an idle worker thread. The idle worker thread will be bound to and responsible for handling a single client request until the final result is returned, achieving request-level isolation and resource tracking. In the current worker thread, the request processing module receives the image processing request and the attached image data stream initiated by the client, parses the request message, extracts the parameters of task type, image format or processing requirements, and obtains the raw image data to be processed.

[0063] S4. Parallel preprocessing of image data:

[0064] Following step S3, the thread selects the appropriate processing flow from the configurable preprocessing strategy library based on the parsed task type. Subsequently, the preprocessing module is called to perform data-level parallel standardized operations on the image data, including deserialization, size normalization, color space conversion, noise filtering, and image enhancement, ultimately obtaining target tensor data that meets the model input requirements.

[0065] S5. Pipeline inference based on GPU memory pool:

[0066] S5.1 CPU to GPU data transfer: The preprocessed target tensor data is asynchronously copied from the CPU memory to the contiguous storage area allocated by the GPU memory resource pool module, and the CPU memory is released for subsequent reuse.

[0067] S5.2, GPU Parallel Inference: The GPU inference module locks the corresponding contiguous storage area, and calls the corresponding deep learning model or algorithm on the GPU inference module to perform forward inference calculation;

[0068] S5.3, GPU to CPU data transfer: After inference is completed, the output result is asynchronously copied from the contiguous storage area allocated by the GPU memory resource pool module back to the corresponding fixed buffer on the CPU side, and the contiguous storage area allocated by the GPU memory resource pool module is released. The scheduling is carried out through the synchronization event mechanism in the GPU memory resource pool module. The three stages between different inference tasks can achieve time overlap, that is, when the GPU is executing the current task calculation, the CPU is simultaneously performing data transfer for the next task and result return for the previous task.

[0069] S6. Parallel post-processing of inference results:

[0070] The post-processing module is invoked to perform data-level parallel post-processing operations on the inference results that have been sent back to the fixed buffer on the CPU. The post-processing operations include confidence threshold filtering, non-maximum suppression deduplication, multi-scale result fusion, and data format standardization, and finally generate the output results.

[0071] S7. Result Return and Resource Recovery:

[0072] The current worker thread returns the formatted final result to the corresponding CPU through the request processing module. After the request is processed, the current worker thread releases all the temporary resources it occupies and actively returns to the thread pool, marking it as available, waiting for the next task allocation, thus realizing the recycling and reuse of resources.

[0073] S8. Dynamic Resource Adjustment and System Monitoring:

[0074] During system operation, the exception handling and monitoring module analyzes system load, queue length, and hardware status in real time; the GPU memory resource pool module dynamically adjusts the number and size of memory blocks based on the monitored data to adapt to the memory requirements of different batches and models; the thread pool size automatically expands and shrinks according to the current request queue length to achieve elastic scaling and load balancing; when a task timeout, hardware anomaly, or data error is detected, the system automatically triggers a recovery strategy and issues an alarm.

[0075] Combination Figure 2The flowchart shown describes the execution process of the method of the present invention in detail, taking a complete image processing request as an example. This embodiment aims to process a single request (10 images of industrial product surface defect detection with a resolution of 2048×2048), demonstrating the entire process from request arrival to result return:

[0076] 1. System Startup and Initialization (corresponding to "Start" → "System Deployment" → "System Initialization" in the flowchart): The system powers on and starts up, with each software module loading sequentially. First, the request processing module starts the gRPC server, listening on a specified port (e.g., 50051). Next, the thread pool module creates and initializes a thread pool containing 128 worker threads based on the number of CPU cores. Simultaneously, the GPU memory resource pool module pre-allocates a total of 256 MemoryBlocks on four NVIDIA H100 graphics cards using cudaMalloc. Each Block contains input / output device buffers and a corresponding fixed CPU buffer, and host-side memory is allocated through cudaMallocHost, establishing a one-to-one mapping. The GPU inference module loads the pre-compiled TensorRT engine file from storage and creates multiple execution contexts. Finally, the exception handling and monitoring module starts a background monitoring thread to collect system metrics. At this point, the system enters a ready state, awaiting client requests.

[0077] 2. Request Reception and Task Dispatch (corresponding to "Client Request" → "Y" → "High-Concurrency Request Reception and Parsing" in the flowchart): A client sends an image processing request to the system via the gRPC protocol. The message contains image binary data, the task type "Defect Detection," and necessary parameters. The request processing module receives the request, parses and deserializes it, and extracts key information. Subsequently, the thread pool module allocates an idle worker thread (denoted as WorkerThread#45) from the pool to handle this request entirely. WorkerThread#45 obtains the raw JPEG / PNG image byte stream from the request.

[0078] 3. Parallel Image Data Preprocessing (corresponding to "Parallel Image Data Preprocessing" in the flowchart): WorkerThread#45 calls the preprocessing module to decode and preprocess the image data using OpenCV. Specifically, this includes: ① Decoding the JPEG byte stream into a cv::Mat object; ② Using TBB parallelization, converting the image set from BGR color space to RGB space; ③ Normalizing the image size to the model requirement of 1024×1024 (using bilinear interpolation); ④ Normalizing the pixel values ​​(dividing by 255.0) and applying mean-standard-deviation normalization; ⑤ Converting the final cv::Mat object into a contiguous memory floating-point tensor with a shape of [10,3,1024,1024] (NCHW format). The preprocessing process fully utilizes multi-core CPU parallel computing, taking approximately 8ms.

[0079] 4. Pipeline inference based on GPU memory pool (core step, corresponding to the sub-process on the right side of the flowchart):

[0080] ① Host-to-GPU Data Transfer: WorkerThread#45 requests a free MemoryBlock (let's say Block_A) from the GPU memory resource pool module. The memory pool marks its status as "in use" and returns after an atomic operation. The thread then calls the asynchronous copy function cudaMemcpyAsync to transfer the preprocessed tensor data from the host memory to the GPU's input buffer (d_input) of Block_A. This operation is performed on the dedicated CUDA stream (stream_h2d) bound to Block_A and does not block the CPU. After the copy is complete, a CUDA event evt_h2d_done is recorded on stream_h2d; ② GPU Parallel Inference: The GPU inference module acquires Block_A and ensures that its computation stream (stream_compute) waits for the event evt_h2d_done (implemented through cudaStreamWaitEvent). Once the data is ready, the module binds the device buffer of Block_A to the TensorRT execution context and calls enqueueV2 to start asynchronous inference. The Mask R-CNN model performs forward propagation on the GPU, performing feature extraction, region proposal, object classification, bounding box regression, and instance segmentation. This computation process takes approximately 15ms. After computation, the event `evt_compute_done` is recorded on `stream_compute`; ③ GPU-to-host data transfer: Another dedicated result return stream (`stream_d2h`) awaits the `evt_compute_done` event. Once the GPU computation is complete, the system automatically initiates asynchronous copying, transferring the inference results (including bounding box coordinates, class confidence, segmentation mask, etc.) from the GPU output buffer (`d_output`) of Block_A back to the corresponding host fixed memory buffer (`h_buffer`). This transfer can be performed in parallel with subsequent requested H2D transfers and GPU computations.

[0081] 5. Parallel Post-Processing of Inference Results (corresponding to "Parallel Post-Processing of Results" in the flowchart): Once the `evt_compute_done` event completes, post-processing can begin on the CPU. WorkerThread#45 calls the post-processing module to process the raw results returned to host memory. Confidence Filtering: Filters out detection boxes with a confidence level below 0.5. Non-Maximum Suppression (NMS): Applies an IoU threshold of 0.45 to remove duplicate detection boxes. Mask Post-Processing: Binarizes and extracts contours from the instance segmentation masks. Result Formatting: Converts the final detected defect category, location, confidence level, and mask contour point set into a predefined JSON format. Post-processing also uses TBB parallelization, taking approximately 3ms.

[0082] 6. Result Return and Resource Reclamation (corresponding to "Result Return and Resource Reclamation" in the flowchart): After post-processing, WorkerThread#45 encapsulates the formatted JSON result into a gRPC response message through the request processing module and sends it back to the client. Subsequently, this thread performs resource reclamation: atomically marking the state of MemoryBlock A as "idle" and returning it to the GPU memory resource pool. Finally, WorkerThread#45 itself returns to the thread pool, marked as available. At this point, a single request is processed, with a total end-to-end latency of approximately 8ms for preprocessing + 15ms for inference + 3ms for post-processing + transport overlap ≈ 28ms.

[0083] 7. Continuous System Operation and Dynamic Adjustment (corresponding to the loop returning "Client Request" and the "Shut Down System" judgment in the flowchart) The system continues to run, repeating steps 2 to 6 to handle massive concurrent requests. The exception handling and monitoring module monitors the entire process in real time. For example, if the GPU card utilization of WorkerThread#45 is found to be consistently higher than 95%, the thread pool strategy is dynamically adjusted to prioritize scheduling new requests to other GPU cards. If a MemoryBlock request times out, an alarm is triggered and an attempt is made to allocate memory from the backup memory pool. When the "Shut Down System" instruction (Y) is detected, the system executes in an orderly manner: stops receiving new requests, waits for all ongoing tasks to complete, releases all resources such as the thread pool, GPU memory pool, and models in sequence, and finally exits safely ("End").

[0084] This embodiment clearly demonstrates how the present invention optimizes the traditional serial processing flow into a highly parallel pipeline through multi-threaded scheduling and GPU memory resource pooling, thereby achieving high throughput and low latency industrial-grade image processing while ensuring the accuracy of the results.

[0085] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the present invention. Although detailed descriptions have been provided with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments, and they should all be covered within the protection scope of the claims.

Claims

1. An industrial image processing system based on multi-threading and GPU memory resource pool, characterized in that, It includes an exception handling and monitoring module, a request handling module, a thread pool module, a preprocessing module, a postprocessing module, a GPU inference module, and a GPU memory resource pool module; The request processing module supports multiple communication protocols, including HTTP / RESTful API, gRPC, and WebSocket. It receives client requests and image data with high concurrency, parses request parameters to determine the task type, and returns the final processing result or output data from the post-processing module to the client. The thread pool module manages a group of worker threads through pooling technology to asynchronously and concurrently execute image processing tasks issued by the request processing module, thereby achieving task scheduling and resource reuse. The preprocessing module performs standardized preprocessing operations on the input image in parallel; During the system initialization phase, the GPU memory resource pool module pre-allocates and manages a set of structured memory resource blocks in the GPU device memory and host memory; the GPU memory resource pool module provides an atomic, lock-free allocation interface for the GPU inference module to request and release memory as needed. The GPU inference module calls the memory resources allocated by the GPU memory resource pool module, loads the preprocessed image data into the GPU, executes the inference calculation of the deep learning model or algorithm, and writes the resulting results into the corresponding output memory area. The post-processing module processes and integrates the output data of the GPU inference module in parallel on the CPU side, including result confidence filtering, overlapping box deduplication, cross-batch result merging, and data structure formatting, to generate the final deliverable result. The exception handling and monitoring module is used for the observability and fault tolerance of the entire system chain. It records the operation log, monitors the health status of each module, and triggers recovery mechanisms and alarm notifications when task execution times out, hardware failures or system anomalies occur.

2. The industrial image processing system based on multi-threading and GPU memory resource pool according to claim 1, characterized in that, The request processing module defines a unified service protocol in Protocol Buffers format, which is used to receive image processing requests submitted by clients. The request message contains the task ID, raw image data, image size, task type, and processing parameters. After receiving the request, the request processing module first performs protocol parsing and data deserialization, then distributes it to the internal processing pipeline according to the task type, and encapsulates the structured result into a response message and returns it to the client after processing. The request processing module has a built-in connection pool and request queue, supporting the processing of thousands of concurrent connections per second.

3. The industrial image processing system based on multi-threading and GPU memory resource pool according to claim 2, characterized in that, The thread pool module is implemented using the BSThreadPool thread pool based on C++17. During system initialization, a fixed number of worker threads are dynamically created based on the number of CPU physical cores. The thread pool maintains a lock-free task queue, supporting asynchronous task submission and load-balanced scheduling. Each worker thread is bound to an independent GPU context and memory resource handle during its lifecycle. The thread pool module provides a task priority scheduling mechanism, allowing high-priority tasks to be executed in the queue, while monitoring the health status of each thread and automatically restarting abnormal threads.

4. The industrial image processing system based on multi-threading and GPU memory resource pool according to claim 3, characterized in that, The preprocessing module integrates OpenCV 4.5 and FFmpeg libraries, and uses Intel TBB and OpenMP to achieve data-level parallel acceleration. The preprocessing operations it performs include video stream decoding and frame extraction using FFmpeg, image color space conversion using OpenCV, size scaling, histogram equalization, Gaussian filtering for noise reduction, and local contrast enhancement based on CLAHE.

5. The industrial image processing system based on multi-threading and GPU memory resource pool according to claim 4, characterized in that, The GPU memory resource pool module pre-allocates a fixed number of memory resource blocks on the CPU and GPU sides when the system starts. Each memory resource block includes a GPU-side input buffer, a GPU-side output buffer, and a corresponding CPU-side fixed memory buffer. The memory resource blocks are managed through a specific data structure, and the GPU memory resource pool module uses atomic operations to achieve lock-free allocation and reclamation.

6. The industrial image processing system based on multi-threading and GPU memory resource pool according to claim 5, characterized in that, The GPU inference module is built on TensorRT 8.5 and is responsible for executing optimized deep learning model inference. During the initialization phase, the GPU inference module loads pre-compiled TensorRT engine files from the disk. Each TensorRT engine file corresponds to a specific model and batch size configuration.

7. The industrial image processing system based on multi-threading and GPU memory resource pool according to claim 6, characterized in that, The post-processing module is implemented based on OpenCV and custom C++ operators. It performs data-level parallel post-processing on the raw results returned by the GPU inference module. The post-processing module uses TBB for parallel loop optimization. Its processing steps include:

1. Confidence filtering: filtering out results with low confidence according to a preset threshold; 2. Result fusion: performing coordinate mapping and deduplication merging on the results of multi-scale or block inference; 3. Formatted output.

8. An industrial image processing method based on multi-threading and GPU memory resource pool, characterized in that, It is implemented based on the industrial image processing system based on multi-threading and GPU memory resource pool as described in any one of claims 1-6, and includes the following steps: S1. System Deployment: The request processing module, thread pool module, preprocessing module, postprocessing module, GPU inference module, GPU memory resource pool module, and exception handling and monitoring module are integrated and deployed on the server in a loosely coupled and standardized manner to complete the construction of the system software environment. S2, System Initialization: Each module performs startup and memory initialization operations. The thread pool module creates and maintains a thread pool containing n worker threads based on a preset concurrency level. The GPU memory resource pool module pre-allocates several fixed-size contiguous storage areas in the GPU memory and allocates a fixed-size buffer of equal size for each contiguous storage area in the CPU memory, establishing a one-to-one mapping "GPU-CPU" memory pair for inference data input and result return. The exception handling and monitoring module starts synchronously and begins to record the system running status and resource usage in real time. S3, High-concurrency request reception and parsing: The thread pool module uses a lock-free scheduling strategy to allocate an idle worker thread. The idle worker thread will be bound to and responsible for handling a single client request until the final result is returned, achieving request-level isolation and resource tracking. In the current worker thread, the request processing module receives the image processing request and the attached image data stream initiated by the client, parses the request message, extracts the parameters of task type, image format or processing requirements, and obtains the raw image data to be processed. S4. Parallel preprocessing of image data: Following step S3, the thread selects the appropriate processing flow from the configurable preprocessing strategy library based on the parsed task type. Subsequently, the preprocessing module is called to perform data-level parallel standardized operations on the image data, including deserialization, size normalization, color space conversion, noise filtering, and image enhancement, ultimately obtaining target tensor data that meets the model input requirements. S5. Pipeline inference based on GPU memory pool: S5.1 CPU to GPU data transfer: The preprocessed target tensor data is asynchronously copied from the CPU memory to the contiguous storage area allocated by the GPU memory resource pool module, and the CPU memory is released for subsequent reuse. S5.2, GPU Parallel Inference: The GPU inference module locks the corresponding contiguous storage area, and calls the corresponding deep learning model or algorithm on the GPU inference module to perform forward inference calculation; S5.3, GPU to CPU data transfer: After inference is completed, the output result is asynchronously copied from the contiguous storage area allocated by the GPU memory resource pool module back to the corresponding fixed buffer on the CPU side, and the contiguous storage area allocated by the GPU memory resource pool module is released. The scheduling is carried out through the synchronization event mechanism in the GPU memory resource pool module. The three stages between different inference tasks can achieve time overlap, that is, when the GPU is executing the current task calculation, the CPU is simultaneously performing data transfer for the next task and result return for the previous task. S6. Parallel post-processing of inference results: The post-processing module is invoked to perform data-level parallel post-processing operations on the inference results that have been sent back to the fixed buffer on the CPU. The post-processing operations include confidence threshold filtering, non-maximum suppression deduplication, multi-scale result fusion, and data format standardization, and finally generate the output results. S7. Result Return and Resource Recovery: The current worker thread returns the formatted final result to the corresponding CPU through the request processing module. After the request is processed, the current worker thread releases all the temporary resources it occupies and actively returns to the thread pool, marking it as available, waiting for the next task allocation, thus realizing the recycling and reuse of resources. S8. Dynamic Resource Adjustment and System Monitoring: During system operation, the exception handling and monitoring module analyzes system load, queue length, and hardware status in real time; the GPU memory resource pool module dynamically adjusts the number and size of memory blocks based on the monitored data to adapt to the memory requirements of different batches and models; the thread pool size automatically expands and shrinks according to the current request queue length to achieve elastic scaling and load balancing; when a task timeout, hardware anomaly, or data error is detected, the system automatically triggers a recovery strategy and issues an alarm.

Images