Medical image segmentation method and system based on accelerated convolutional neural network

Abstract

The invention relates to the technical field of image processing, and provides a medical image segmentation method and system based on an accelerated convolutional neural network, and the method comprises the steps: analyzing the network structure of the convolutional neural network for a medical image, and dynamically scheduling a medical image segmentation task to a corresponding calculation engine according to the convolution kernel type of each network layer; wherein the calculation engine comprises a frequency domain processing engine which is used for processing calculation of a large-size convolution kernel and accelerating by adopting frequency domain complex multiplication; the Winograd engine is used for accelerating calculation of a medium-sized convolution kernel and reducing multiplication times through linear transformation; and the GEMM engine is used for executing small-size convolution and general matrix multiplication. And high fidelity required by precise segmentation of the medical image is kept while huge hardware benefits are realized.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a medical image segmentation method and system based on an accelerated convolutional neural network. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the deep integration of precision medicine and artificial intelligence, Convolutional Neural Networks (CNNs) have become a core technology for medical image segmentation (such as lesion segmentation and tissue classification), achieving results comparable to or even surpassing those of human experts. However, the success of these advanced CNN models comes at the cost of enormous computational complexity. For example, classic networks such as VGG16 and ResNet-18 contain tens to hundreds of millions of parameters, requiring billions of GigaOperations Per Second (GOPs) for a single inference operation. This computational requirement makes real-time processing on general-purpose CPUs impractical, while although GPUs are powerful, their high power consumption, cost, and physical size severely limit their application in portable, bedside, or implantable embedded medical devices.

[0004] Field-Programmable Gate Arrays (FPGAs) have become an ideal platform for accelerating CNN inference due to their inherent parallelism, reconfigurable hardware architecture, and excellent energy efficiency. In recent years, although academia and industry have proposed various FPGA-based CNN accelerator solutions, which have made significant progress in certain aspects, the following key bottlenecks still exist when designing high-performance, energy-efficient, and highly flexible FPGA accelerators for practical medical image segmentation applications: (1) Mismatch between computation and architecture: Especially in frequency domain acceleration, there is a lack of hardware architecture that can efficiently map low-precision (e.g., 8-bit) complex number operations to physical digital signal processing (DSP) modules, resulting in low resource utilization and huge redundant operation overhead. (2) Rigid computing mode: Most accelerators adopt a "one-size-fits-all" computing architecture (pure spatial domain or pure frequency domain), which cannot dynamically switch to the optimal computing mode according to the different computing characteristics of different layers in CNN (such as 1×1, 3×3, large kernel convolution), thus limiting the overall end-to-end performance. (3) The memory wall problem remains severe: existing frequency domain methods either face memory bottlenecks due to high-precision calculations or lead to unnecessary data transfer and redundant calculations due to a lack of optimized data flow design, failing to fundamentally alleviate the pressure on the bandwidth of Dynamic Random Access Memory (DRAM); (4) Insufficient co-design of model and hardware: Model compression technology and hardware architecture design are often separate, failing to achieve "hardware-aware" model design, which makes it impossible to fully release the potential of algorithm optimization on hardware. Summary of the Invention

[0005] To address the technical problems mentioned above, this invention provides a medical image segmentation method and system based on accelerated convolutional neural networks. The designed dynamically reconfigurable hybrid computing engine includes a specially optimized frequency domain processing engine, a Winograd engine, and a GEMM engine. It intelligently distributes tasks to the most efficient hardware units based on the specific computational characteristics of the network layers. This "divide and conquer" structural feature ensures that each type of convolutional operation is executed on its optimal hardware path, avoiding resource waste and performance bottlenecks. This significantly improves the end-to-end inference speed of the entire convolutional neural network, achieving substantial hardware benefits while maintaining the high fidelity required for accurate medical image segmentation.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of this invention provides a medical image segmentation method based on an accelerated convolutional neural network, comprising: Receive medical images to be processed; For medical images, the network structure of the convolutional neural network is analyzed, and the medical image segmentation task is dynamically scheduled to the corresponding computing engine according to the type of convolution kernel in each network layer. The computing engines include: a frequency domain processing engine for processing the calculation of large-sized convolution kernels, which is accelerated by frequency domain complex multiplication; a Winograd engine for accelerating the calculation of medium-sized convolution kernels, which reduces the number of multiplications through linear transformation; and a GEMM engine for performing small-sized convolution and general matrix multiplication operations.

[0007] Furthermore, the frequency domain complex multiplication includes: transforming the feature map from the spatial domain where the pixel is located to the frequency domain through discrete Fourier transform; performing element-wise multiplication in the frequency domain, and decomposing one complex multiplication into three real multiplications and five real additions / subtractions to obtain the real and imaginary parts of the complex result, thereby realizing convolution calculation in the spatial domain; coupling the real and imaginary parts of the two weighted multiplications to registers respectively, and adding a bias to the registers to realize the bit conversion from unsigned to signed.

[0008] Furthermore, the bias is: ; in, and These are the least significant 8 bits of A and B, respectively; and These are the corresponding unsigned inputs; The input is a signed multiplicand, and inmode indicates the operation mode.

[0009] Furthermore, the results of addition / subtraction calculations can be cached and reused directly in subsequent clock cycles.

[0010] Furthermore, a set of packaged complex weights can be reused by multiple feature map activation values ​​from different input blocks, or by the real and imaginary parts of the same activation value respectively; after the calculation of a set of weights is completed, a set of packaged complex activation values ​​is reused by the next set of weights from different output channels.

[0011] Furthermore, the network structure, quantization parameters, and weight data of the convolutional neural network are stored in a non-volatile storage medium, and the convolutional neural network is pruned before storage based on the L1 norm to evaluate the importance of the filters.

[0012] Furthermore, before storage, the data bit width is adaptively allocated to the pruned convolutional neural network based on the sensitivity of the network layers.

[0013] A second aspect of the present invention provides a medical image segmentation system based on an accelerated convolutional neural network, comprising: An image receiving module is configured to receive medical images to be processed. The image segmentation module is configured to: for medical images, analyze the network structure of the convolutional neural network, and dynamically schedule the medical image segmentation task to the corresponding computing engine according to the type of convolution kernel in each network layer; wherein the computing engine includes: a frequency domain processing engine, used to process the calculation of large-size convolution kernels, using frequency domain complex multiplication for acceleration; a Winograd engine, used to accelerate the calculation of medium-size convolution kernels, reducing the number of multiplications through linear transformation; and a GEMM engine, used to perform small-size convolution and general matrix multiplication operations.

[0014] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method for medical image segmentation based on an accelerated convolutional neural network.

[0015] A fourth aspect of the present invention provides a computer device including a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method for medical image segmentation based on an accelerated convolutional neural network.

[0016] Compared with the prior art, the beneficial effects of the present invention are: The dynamically reconfigurable hybrid computing engine designed in this invention includes a specially optimized frequency domain processing engine, a Winograd engine, and a GEMM engine. It can intelligently distribute tasks to the most efficient hardware units according to the specific computational characteristics of the network layers. This "divide and conquer" structural feature ensures that each type of convolution operation is executed on its optimal hardware path, avoiding resource waste and performance bottlenecks. This significantly improves the end-to-end inference speed of the entire convolutional neural network, achieving huge hardware benefits while maintaining the high fidelity required for accurate medical image segmentation.

[0017] This invention employs DSP packaging technology and designs a corresponding dynamic bias correction mechanism. This structure can package two independent 8-bit complex correlation operations and perform them in parallel in a single DSP module in a streamlined manner. This mechanism directly increases the throughput of DSP hardware resources used for frequency domain calculation by nearly 100%. With minimal additional logic overhead, it achieves in-depth exploitation of the DSP computing potential and enables real-time analysis of video streams from endoscopes or ultrasound probes.

[0018] This invention designs a mapping data stream tightly coupled with FPE computation. Its core mechanism is to maximize the three-level reuse of on-chip data (reuse of packed weights, reuse of packed activation values, and reuse of intermediate results within the computation engine). This design significantly reduces the need to access high-power, high-latency external DRAM, directly reducing the total power consumption. Furthermore, it improves the actual performance by reducing the waiting time of the computation unit, thus achieving high-performance, low-power real-time inference acceleration for medical images. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0020] Figure 1 This is a flowchart of the hardware inference stage of Embodiment 1 of the present invention; Figure 2 This is a flowchart of the software training phase of Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the structure of a computer device according to Embodiment 4 of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0022] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] Example 1 This embodiment provides a medical image segmentation method based on an accelerated convolutional neural network.

[0024] Currently, there is an urgent need for a new technological approach that can deeply integrate efficient frequency domain computation, flexible reconfigurable architecture, and hardware-friendly model compression technology to break through the limitations of existing medical image segmentation methods based on accelerated convolutional neural networks.

[0025] This embodiment provides a medical image segmentation method based on accelerated convolutional neural networks. Based on frequency domain computation and a hybrid engine, it aims to break through the limitations of existing methods and provide a new solution for accelerating medical image segmentation based on FPGA.

[0026] This embodiment provides a medical image segmentation method based on accelerated convolutional neural networks. It employs an adaptive medical image processing convolutional neural network (CNN) accelerator based on frequency domain computation and a hybrid engine. This accelerator improves the traditional single computation mode into a dynamically reconfigurable hybrid computation architecture, including a frequency domain processing engine (FPE), a Winograd engine, and a GEMM (General Matrix to Matrix Multiplication) engine. The accelerator is directly mapped to an FPGA platform, overcoming the computational architecture mismatch, high memory access overhead, and low energy efficiency bottlenecks of existing accelerators when processing convolutional kernels of different sizes. The core hardware device utilizes a top-level scheduler to dynamically distribute tasks according to the characteristics of the network layers. For large-size convolutions, 8-bit frequency domain complex number operations are efficiently executed through DSP packing technology and bias correction mechanisms in the FPE. Combined with hardware and software collaborative offline structured pruning, mixed-precision quantization, and a unique mapped data stream, the computational throughput and DSP resource utilization are significantly improved, greatly alleviating the "memory wall" pressure and achieving high-performance, low-power real-time inference acceleration for medical images.

[0027] This embodiment provides a medical image segmentation method based on an accelerated convolutional neural network, which relies on: (1) FPGA hardware platform: Xilinx ZCU102 evaluation platform was selected. This platform is based on Zynq UltraScale+MPSoC, which integrates a powerful processing system (PS, containing ARM Cortex-A53 / R5 processor) and programmable logic (PL) on the chip, and has abundant hardware resources such as DSP, BRAM / URAM and high-speed DDR4 memory interface. (2) External DRAM memory: Onboard DDR4 memory, used to store the complete network weight data packet after offline preprocessing, as well as the input / output images and intermediate feature maps that need to be accessed globally during inference; (3) Embedded processor (PS end): As the host, it runs the PetaLinux embedded operating system, and is responsible for running the overall application software, parsing the network model structure, issuing instructions to the PL end, configuring and starting DMA for data transfer, and processing the final inference results; (4) Development and training environment (offline): PC equipped with NVIDIA RTX series GPUs, running the TensorFlow deep learning framework, used to complete preprocessing work such as training CNN models, structured pruning, mixed precision quantization and offline FFT of weights.

[0028] This embodiment provides a medical image segmentation method based on accelerated convolutional neural networks, employing an adaptive FPGA accelerator for CNN model inference in medical image processing, designed in a hardware-software co-design. Through hardware-friendly model preprocessing, a dynamically reconfigurable hybrid computing architecture, and a processing engine and data stream specifically optimized for 8-bit frequency domain operations, a balance between performance, power consumption, and flexibility is achieved.

[0029] This embodiment provides a medical image segmentation method based on an accelerated convolutional neural network, which includes the following steps: Step 1: Design a dynamically reconfigurable hybrid computing engine, which mainly consists of a top-level scheduler and three types of dedicated computing units optimized for different convolution types: (1) Frequency-domain Processing Engine (FPE): It is specifically designed for efficient processing of large-size convolution kernels (such as 5×5 and above), and uses frequency-domain complex multiplication for acceleration. This is the core innovation of this invention.

[0030] (2) Winograd engine: It is specifically designed to accelerate the 3×3 convolution that is very common in medical images. Its mathematical principle is to convert the convolution operation into a matrix operation and reduce the number of multiplications through a small number of linear transformations.

[0031] For a convolution with an output of m×m and a kernel of r×r, its matrix form is: ; in, For output data blocks; For input data blocks; For convolution kernel; G, and The transformation matrix is ​​fixed. This is element-wise multiplication.

[0032] For a typical scenario of computing a 3×3 convolution kernel (F(2,3)), this algorithm can reduce the number of multiplications by 2.25 times.

[0033] (3) GEMM engine: efficiently performs 1×1 convolution in the form of pulsatile array, which is essentially a general matrix multiplication.

[0034] (4) The top-level scheduler is responsible for parsing the network structure of the convolutional neural network, identifying the type of the current network layer (including the type of convolutional kernel and computational characteristics), and dynamically scheduling the computational tasks to the corresponding computational engine.

[0035] The accelerator interacts with external memory via direct memory access (DMA). The top-level scheduler configures the DMA and data path to send feature maps and weight data to the corresponding computing engines.

[0036] The following section details the design of the high-efficiency frequency domain processing engine (FPE).

[0037] FPE aims to solve the efficiency bottleneck of 8-bit fixed-point numbers in frequency domain computation. Its overall architecture includes an on-chip buffer, FFT / iFFT transform unit, controller, and FPE core computing unit. The FPE core computing unit consists of four key sub-components: Pack (packaging component), DSP (computation component), Acc (accumulation component), and Rec (recovery component).

[0038] Frequency domain processing is based on converting spatial domain signals to the frequency domain using the Discrete Fourier Transform (DFT) (usually implemented using the FFT algorithm). For a two-dimensional image (feature map)... The formula for transforming the pixel's spatial domain to the frequency domain is: ; in, For the input image in coordinates Pixel value at; For the frequency domain representation obtained after transformation, in frequency domain coordinates The value at; The height of the image; The width of the image; The imaginary unit, .

[0039] According to the convolution theorem, computing complex convolutions in the spatial domain is equivalent to performing simple element-wise multiplications in the frequency domain. Let... The frequency domain representation of the input feature map. If the frequency domain representation of the convolution kernel weights is given, then the output is... It can be calculated using the following formula: ; Since the values ​​in the frequency domain are complex numbers (containing both real and imaginary parts), the element-wise multiplication above is actually complex multiplication, expanding the complex number into its real part. and the virtual part Then we have: ; This formula is the standard expression for complex multiplication, used to describe the multiplication operation between a complex weight and a complex input activation value in frequency domain convolution; where It is the real part of the output activation value. It is the imaginary part of the output activation value. It is the real part of the input activation value. It is the imaginary part of the input activation value. It is the real part of the complex weight. It is the imaginary part of the complex weight.

[0040] Formula for calculating the real part of the output: ; Formula for calculating the imaginary part of the output: ; Performing a complex multiplication directly using the above formula requires four real multiplications and two real additions / subtractions. In hardware designs such as FPGAs, multipliers are computational units that consume more logic resources, have longer critical path delays, and higher power consumption than adders. Therefore, to achieve higher computational efficiency with lower hardware overhead, this invention employs an optimized complex multiplication algorithm that reduces the required number of real multiplications from four to three. This optimized algorithm introduces three intermediate variables. This allows us to restructure the computation process, thereby breaking down a complex multiplication into three real multiplications and five real additions / subtractions.

[0041] That is, the FPE core computing unit uses three-way real number multiplication to perform a complex number multiplication operation, thus transforming complex number multiplication into... Decomposed into three intermediate variables The calculation ultimately outputs the real part as The imaginary part is The specific calculation process is as follows: ; ; ; ; The above calculation transforms complex number multiplication into a simpler calculation. These three intermediate values ​​are then combined using simple addition to form the real part of the final complex number result. and the virtual part Although this structure, which requires three multiplications and five additions, superficially increases the number of addition operations, the benefits are significant. It not only reduces the resource consumption of the most critical multipliers, but its regular, step-by-step computation mode is also highly suitable for efficient implementation in FPGA DSP pipelines. The data path of the frequency domain processing engine (FPE) proposed in this invention is designed around this three-stage computation, by... The computation is mapped to different pipeline stages, thereby enabling highly parallel processing and significantly improving the overall engine's computational throughput and energy efficiency.

[0042] The FPE core computing unit adopts a DSP packing and bias correction mechanism: the FPE's pack component is responsible for coupling the real and imaginary parts of complex numbers. Taking an 8-bit fixed-point number as an example, it couples the real parts of two weighted numbers ( and ) and imaginary part ( and These are coupled to registers A and B of the 27-bit DSP, respectively. and Since the least significant 8 bits after the coupled register are considered as unsigned inputs, the final output needs to undergo a bit-to-bit conversion from unsigned to signed. In this embodiment, a bias D is added to the DSP for the conversion. The generation of D is based on the following formula and algorithm 1: ; in, and These are the least significant 8 bits of A and B, respectively. and (signed numbers); and These are the corresponding unsigned inputs; and They are and The sign bit, A l <0 means an-1 =1, B l <0 means b n-1 =1, A l ≥0 means a n-1 =0, B l ≥0 means b n-1 =0; The input is a signed multiplicand, and inmode indicates the operation mode.

[0043] Algorithm 1: Add a bias D to the DSP in the FAE, as follows: Input: A l B l C. Operation mode (inmode); Output: Bias D; Case 1: When the operation mode is subtraction (-): If A l <0 and B l <0, that is, a n-1 =1 and b n-1 =1, then D=0; If A l ≥0 and B l <0, or A l <0 and B l If ≥0, then: determine whether A is satisfied. l <0 and B l ≥0, if satisfied, then D= (C 8), that is, the output D is equal to C shifted left by 8 bits and then flipped; If A l ≥0 and B l If <0), then D=C 8, meaning the output D is equal to C shifted left by 8 bits; Case 2: When the operation mode is addition (+): If A l <0 and B l <0, then D= (C 9), that is, the output D is equal to C shifted left by 9 bits and then inverted; If A l ≥0 and B l <0, or A l <0 and B l If ≥0, then D= (C 8), that is, the output D is equal to the left shifted by 8 bits and then inverted; Otherwise, i.e., A l ≥0 and B l ≥0, i.e., a n-1 =0 and b n-1If =0, then D=0.

[0044] Step 2: Offline software training and preprocessing stage.

[0045] like Figure 2 As shown, the offline software training and preprocessing stages are mainly completed on high-performance PCs or GPU servers, where a series of hardware-friendly optimizations are performed on the original CNN model to generate model files and data packages adapted to hardware accelerators.

[0046] Step 201, Lightweight Design: Replace standard convolution with structures such as depthwise separable convolution to fundamentally reduce the number of parameters and computational load.

[0047] Step 202, Model Training and Optimization: First, train the original convolutional neural network model (e.g., U-Net) in a deep learning framework (e.g., TensorFlow); then, calculate the importance score of each layer filter according to the importance evaluation criterion based on L1 norm, and perform structured pruning on the convolutional neural network model accordingly, removing redundant filters to compress the model size.

[0048] Specifically, structured pruning employs a method that prunes entire filters or channels, ensuring that the pruned model retains a regular, dense matrix structure, facilitating efficient parallel processing by hardware. To provide precise and quantifiable criteria for structured pruning, the importance of each filter in a trained CNN model must first be evaluated. The L1 norm is used as the sole criterion for measuring the importance of a single filter. The L1 norm calculates the sum of the absolute values ​​of all elements in a vector or matrix, and its physical meaning directly reflects the overall activity or contribution of all weight parameters in that filter.

[0049] For the i-th filter in a convolutional layer It is a three-dimensional tensor with dimension . (in (where K is the number of input channels and K is the side length of the convolution kernel), filter Importance score Its L1 norm is defined and calculated as follows: ; in, Represents the i-th filter Importance score; Representative filter In the middle, located in the c-th input channel, with spatial coordinates of A single weight value; This represents the absolute value operation; This represents the summation of the absolute values ​​of all weights within the filter (traversing all input channels and all spatial locations).

[0050] A lower L1 norm value A small sum of the absolute values ​​of all weights in the filter indicates that the filter contributes little to the overall feature extraction of the network and is therefore considered a minor filter that can be pruned. Conversely, a high L1 norm indicates that the filter is a key component of the network feature extraction and must be retained.

[0051] After calculating all filters (F1, F2, ..., F) in this layer n After determining the importance scores, these scores will be sorted, and the filter with the lowest score will be removed as a whole according to the preset pruning rate to complete the pruning operation.

[0052] Step 203, Mixed Precision Quantization and Fine-tuning: Based on the differences in the sensitivity of each layer of the convolutional neural network to precision, mixed precision quantization is performed on the pruned convolutional neural network model, that is, different data bit widths (such as 4 bits and 8 bits) are adaptively allocated. While ensuring the accuracy of medical image segmentation, storage and computing resources are saved to the maximum extent. After quantization, fine-tuning training is performed to restore the model accuracy.

[0053] Step 204, Frequency Domain Transformation and Packaging: For large-size convolutional kernel layers in the convolutional neural network model that require frequency domain calculations, perform Fast Fourier Transform (FFT) offline to pre-convert the weights into frequency domain representations and store the frequency domain weights in external memory.

[0054] Specifically, for layers that require frequency domain computation (typically layers with large convolutional kernels), their pruned and quantized weight data are computed offline using Fast Fourier Transform (FFT), and the results are stored in DRAM.

[0055] Step 205: Generate deployment files: Finally, generate model configuration files from the processed network topology and quantization parameters, package the preprocessed weight data, and store these two core files on non-volatile storage media such as SD cards for hardware to access.

[0056] Step 3: Online hardware inference stage.

[0057] The core of this embodiment is a dynamically reconfigurable hybrid computing engine. The online inference hardware architecture can dynamically schedule the data stream to the most efficient hardware computing unit according to the computing characteristics of different layers in the CNN.

[0058] like Figure 1As shown, the online hardware inference stage is executed on the FPGA hardware platform, and the real-time processing of medical images is completed by the processing system (PS) and programmable logic (PL) working together.

[0059] Step 301, Initialization: After power-on, the PS terminal reads the deployment file package from the SD card; the FPGA bit stream is loaded to configure the PL terminal logic, the preprocessed weight data packet is loaded into the onboard DDR4 memory, and the model configuration file is parsed into the PS terminal memory, and then enters the standby state.

[0060] Step 302, Image Input and Buffer: When medical image data to be processed (such as CT slices) is received, the PS end transmits it to the dedicated input buffer of DDR4 via the bus.

[0061] Step 303, Dynamic Scheduling and Hybrid Computation: The PS-side scheduler parses the network structure layer by layer according to the model configuration file, and dynamically activates the corresponding hardware computing engine on the PL side based on the convolution kernel type of the current layer. If the current layer is a 1×1 convolution, it is scheduled to the GEMM engine to perform general matrix multiplication, i.e., systolic array multiplication; If the current layer is a 3×3 convolution, it is scheduled to the Winograd engine to reduce multiplication complexity, i.e., execute... Operations; If the current layer is a large-size convolution (such as 5×5 and above), it is scheduled to the Frequency Domain Processing Engine (FPE) to perform frequency domain complex multiplication. FPE uses DSP packing technology and bias correction mechanism to efficiently perform 8-bit complex multiplication operations.

[0062] Step 304, Data Transfer and Write-back: The top-level controller configures DMA to move the feature map and weight data to the on-chip buffer of the computing engine; after the calculation is completed, the result data is written back to the specified address in DDR4 via DMA; this "load-calculate-write-back" process is executed in a loop until all layers of the network have been processed.

[0063] Step 305, Result Output: The final inference result (such as the lesion segmentation mask) is generated in the DDR4 output buffer. The application on the PS side then accesses this address to read the result and executes subsequent application layer tasks, such as coloring and rendering the segmented region or calculating its clinically relevant indicators. Finally, the result is presented to the user through the display interface.

[0064] Step 4: Collaborative design of mapped data flow and data reuse.

[0065] To minimize the overhead of the DSP packing operation itself and alleviate DRAM bandwidth pressure, a mapping dataflow was designed to work closely with it. Its core is to maximize on-chip data reuse, which is mainly reflected in the following three aspects: Weight reuse: a set of multiple weights that are bundled together (e.g., ... , and , A feature map activation value can be reused by multiple feature map activation values ​​from different input tiles, or by the real and imaginary parts of the same activation value. Activation Reuse: After calculating a set of weights, a set of bundled activation values ​​(such as...) are reused. , These weights can be reused in the next set of weights from different output channels; Internal Result Reuse: In the FPE pipeline, the calculation results of its internal pre-adder / subtractor can be cached and reused directly in subsequent clock cycles, thereby avoiding redundant calculations and saving power.

[0066] Through the above hardware and software co-design, an FPGA accelerator solution was constructed that can adaptively switch computing modes according to network layer characteristics and achieve extremely high efficiency in frequency domain computing through DSP packaging and deep data reuse.

[0067] This embodiment is based on a series of precise parameters and configurations to ensure the feasibility and high performance of the method. At the hardware level, the method of this embodiment is implemented and verified on a Xilicnx ZCU102 evaluation platform. The Zynq UltraScale+ MPSoC chip on this platform has a core operating clock frequency of 200MHz for the programmable logic (PL) side, and all core multiply-accumulate operations are performed in the on-chip DSP48E2 module. Regarding data precision, 8-bit fixed-point numbers are used as the benchmark, and the data path and computing units are specially designed to be compatible with processing lower bit widths such as 4-bit data to execute mixed-precision quantized models. Specifically, in the accelerator's internal configuration, the FFT module in the frequency domain processing engine (FPE) has a transform size of 8x8 and a parallelism of 16; to minimize DRAM access, the capacities of the on-chip weight buffer and activation value-related buffer are precisely set to 0.13MB and 0.1MB, respectively. Before performing online inference, the model must be preprocessed using the offline toolchain provided in this embodiment, including structured pruning and quantization. The generated model configuration file and weight data package are necessary inputs for the system to perform inference.

[0068] To fully verify the actual effectiveness of the method in this embodiment, a typical case closely related to clinical application was designed and implemented: real-time nodule segmentation of lung CT images based on the U-Net network model. In this case, a U-Net model with 40% channel pruning and 8-bit fixed-point quantization was deployed on the aforementioned ZCU102 platform. When processing 512×512 pixel single-channel CT slices, precise measurements showed that the average end-to-end latency for segmenting a single image was only 23.8 milliseconds, corresponding to a real-time processing throughput of up to 42 FPS (frames per second). This speed demonstrates a significant performance advantage of more than 2.5 times compared to accelerators using traditional spatial domain optimization schemes on the same hardware (whose latency exceeds 60 milliseconds), fully meeting the needs of real-time interactive clinical applications.

[0069] While validating high performance, the impact on accuracy was also rigorously evaluated. Using the Dice coefficient, a widely accepted standard in medical segmentation, the output of the 8-bit quantization model in this embodiment was compared with the results of an unoptimized 32-bit floating-point model run on a PC. The average difference in the Dice coefficient was less than 1.2%. In summary, this fully demonstrates that the method in this embodiment can achieve a significant performance leap while keeping the accuracy loss within a clinically acceptable range, validating its advanced nature, practicality, and high reliability in real-world applications.

[0070] Those skilled in the art will understand that the method of this embodiment addresses the problems faced by existing FPGA accelerators when processing complex CNN models, such as rigid computing architecture, off-chip memory access bottlenecks, and low performance and energy efficiency caused by the disconnect between software and hardware design. It proposes a hybrid precision CNN accelerator with software and hardware co-design and dynamic reconfigurability. It innovatively matches different convolution operators through heterogeneous computing engines, improves computing density by utilizing depth-optimized frequency domain computing units, and combines a mapping data stream with data reuse as its core, thereby achieving a balance between performance and energy efficiency while ensuring high precision.

[0071] At the implementation level, the end-to-end performance of the method in this embodiment is more than 2.5 times better than traditional spatial domain solutions, and power efficiency is increased exponentially. When processing networks such as VGG16, hardware resource efficiency breaks through the theoretical limit of 1.0 GOP / s / multiplier. Meanwhile, after adopting 8-bit fixed-point quantization, the model accuracy loss is strictly controlled within 2%, meeting the high-precision requirements of medical diagnosis. The core DSP packaging technology delivers significant performance improvements while maintaining additional logic resource overhead of no more than 3%. This accelerator can efficiently support multiple mainstream networks such as U-Net and YOLOv3, achieving real-time processing capabilities exceeding 40 FPS, and can be directly embedded into high-end medical imaging equipment, enabling real-time diagnosis and analysis.

[0072] This embodiment provides a medical image segmentation method based on an accelerated convolutional neural network, which has the following technical effects: (1) Analysis of structural features and mechanism of action.

[0073] This embodiment enhances computational efficiency and versatility through a dynamically reconfigurable hybrid engine. Traditional FPGA accelerators often employ a rigid, "one-size-fits-all" architecture, which cannot efficiently handle the complex structures of modern CNNs that combine multiple convolution types (1×1, 3×3, large kernels, etc.). The dynamically reconfigurable hybrid computing engine designed in this embodiment includes a specially optimized Frequency Domain Processing (FPE) engine, a Winograd engine, and a GEMM engine. The top-level scheduler intelligently distributes tasks to the most efficient hardware units based on the specific computational characteristics of the network layers. This "divide and conquer" structural feature ensures that each type of convolution operation is executed on its optimal hardware path, avoiding resource waste and performance bottlenecks. This significantly improves the end-to-end inference speed of the entire network and enables the accelerator to be widely compatible with various mainstream network architectures such as ResNet, MobileNet, and Inception.

[0074] Frequency domain computation throughput is doubled through DSP packaging and bias correction mechanisms. Existing technologies struggle to efficiently map low-precision (e.g., 8-bit) complex number operations to the DSP physical units of an FPGA, resulting in low DSP resource utilization. This invention's FPE core innovatively employs DSP packaging technology and designs a corresponding dynamic bias correction mechanism. This structure can package two independent 8-bit complex number correlation operations and pipeline them in parallel within a single DSP module. This mechanism directly increases the throughput of DSP hardware resources used for frequency domain computation by nearly double, achieving deep exploitation of the DSP's computational potential with minimal additional logic overhead.

[0075] The co-designed mapping dataflow significantly reduces memory access bottlenecks. Traditional accelerator designs face a severe data "memory wall" problem, where high-frequency access to external DRAM leads to significant latency and power consumption. This invention designs a mapping dataflow tightly coupled with FPE computation. Its core mechanism maximizes three-level multiplexing of on-chip data (multiplication of packed weights, reuse of packed activation values, and reuse of intermediate results within the computation engine). This design drastically reduces the need for accessing high-power, high-latency external DRAM, directly reducing the total system power consumption, and further improving actual performance by reducing the latency of computation units.

[0076] (2) Theoretical analysis supports the effectiveness of the technology.

[0077] Co-design of software and hardware achieves global optimization. Theoretically, simply optimizing the algorithm or hardware can only achieve a local optimum. This embodiment adopts the concept of co-design of software and hardware: in the offline stage, through methods such as structured pruning and mixed precision quantization, the original CNN model is processed into a "hardware-friendly" model with lower computational cost, lower storage requirements, and a more regular structure. The data structure and computation mode of this optimized model can perfectly match the hardware characteristics of the hybrid computing engine and efficient data flow in this embodiment. This two-way collaboration between software and hardware allows the advantages at the algorithm level to be translated seamlessly into actual performance and energy efficiency benefits at the hardware level, thereby achieving global optimization.

[0078] To verify the practical effectiveness of the structural advantages of this embodiment, a complete hardware implementation of the scheme was performed on the Xilinx ZCU102 evaluation platform. U-Net, widely used in medical image segmentation, and YOLOv3, representative in object detection tasks, were selected for testing. The experimental results strongly demonstrate the comprehensive benefits of this embodiment.

[0079] A dual breakthrough in performance and energy efficiency. In actual testing, this embodiment demonstrates significant and quantifiable superiority in both performance and energy efficiency. In comparison, FPGA accelerators using traditional spatial domain convolution designs typically achieve energy efficiency of around 30 GOPs / W when processing networks of similar complexity. However, this embodiment, on the ZCU102 platform, achieves an energy efficiency of 75.8 GOPs / W when processing the U-Net model and 68.5 GOPs / W when processing the more complex YOLOv3 model, representing more than double the energy efficiency of traditional solutions. More importantly, in terms of hardware resource efficiency, thanks to the DSP packaging and data reuse technology of this embodiment, the peak equivalent computing power provided by a single multiplier reaches 1.15 GOPs / s / multiplier when processing computationally intensive layers in U-Net and YOLOv3 networks. This data breaks through the theoretical limitation that a physical multiplier can only complete one multiplication operation per clock cycle in traditional designs, powerfully demonstrating the advanced nature of this invention in terms of computing architecture and dataflow design.

[0080] Balancing high compression ratios with high accuracy, this embodiment successfully resolves the dilemma between high-performance optimization and high accuracy preservation. Through hardware and software co-design, using 8-bit fixed-point quantization, the accuracy degradation on relevant public validation sets (such as the LUNA16 dataset for U-Net) is strictly controlled to within 2% compared to the unoptimized 32-bit floating-point model. This result demonstrates that the technical solution of this embodiment can achieve significant hardware gains while maintaining the high fidelity required for demanding applications such as accurate medical image segmentation.

[0081] Extremely low hardware resource utilization. The core innovation of this embodiment is achieving a huge performance return at a very low cost. Experimental data shows that the additional hardware logic overhead required to implement the key technology of DSP packaging is extremely low. Compared with the basic design without this technology, this embodiment achieves nearly double the performance improvement while the incremental utilization of the logic lookup table (LUT) does not exceed 3%. This fully demonstrates that this invention is a high-performance, cost-effective, and easily engineerable advanced technology.

[0082] (3) Significant benefits in practical applications.

[0083] In addition to the aforementioned quantifiable technical advantages, the true value of this embodiment lies in its ability to break through the bottlenecks of existing technologies, bring revolutionary changes to actual medical application scenarios, and solve pain points in clinical practice.

[0084] This enables real-time analysis in clinical medicine. The ultra-high performance and extremely low latency provided by this embodiment mean that AI analysis of medical images is no longer an offline task that takes several minutes. During surgery, doctors can use devices equipped with the chip of this invention to perform real-time analysis of video streams from endoscopes or ultrasound probes, such as real-time annotation of suspicious lesions, automatic measurement of anatomical structures, or overlaying of organ outlines in augmented reality (AR) for surgical navigation. This "analysis as you see" capability greatly improves the accuracy and safety of surgery.

[0085] This approach promotes the portability of high-end medical devices. Traditional high-performance AI inference relies on high-power GPU servers, resulting in bulky devices that require dedicated server room environments. The superior energy efficiency of this embodiment allows powerful AI computing power to be integrated into low-power, even battery-powered devices. This means that AI-assisted diagnosis can move from the radiology departments of large hospitals to bedside clinics in community clinics, emergency scenes in ambulances, and even to handheld devices that doctors can carry (such as smart stethoscopes and portable ultrasound machines). This significantly expands the reach of advanced medical services and contributes to the balanced allocation of medical resources and tiered medical services.

[0086] Example 2 This embodiment provides a medical image segmentation system based on an accelerated convolutional neural network, comprising: An image receiving module is configured to receive medical images to be processed. The image segmentation module is configured to: for medical images, analyze the network structure of the convolutional neural network, and dynamically schedule the medical image segmentation task to the corresponding computing engine according to the type of convolution kernel in each network layer; wherein the computing engine includes: a frequency domain processing engine, used to process the calculation of large-size convolution kernels, using frequency domain complex multiplication for acceleration; a Winograd engine, used to accelerate the calculation of medium-size convolution kernels, reducing the number of multiplications through linear transformation; and a GEMM engine, used to perform small-size convolution and general matrix multiplication operations.

[0087] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same, so they will not be repeated here.

[0088] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a medical image segmentation method based on an accelerated convolutional neural network as described in Embodiment 1 above.

[0089] Example 4 This embodiment provides a computer device, such as... Figure 3 As shown, the system includes a computer-readable storage medium 1003, a processor 1001, a communication interface 1002, and a computer program stored on the computer-readable storage medium 1003 and executable on the processor 1001. The processor 1001, communication interface 1002, and computer-readable storage medium 1003 can be connected via a bus or other means. The communication interface 1002 is used to receive and send data. When the processor 1001 executes the program, it implements the steps in the medical image segmentation method based on an accelerated convolutional neural network as described in Embodiment 1 above.

[0090] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A medical image segmentation method based on an accelerated convolutional neural network, characterized in that, include: Receive medical images to be processed; For medical images, the network structure of the convolutional neural network is analyzed, and the medical image segmentation task is dynamically scheduled to the corresponding computing engine according to the type of convolution kernel in each network layer. The computing engines include: a frequency domain processing engine for processing the calculation of large-sized convolution kernels, which is accelerated by frequency domain complex multiplication; a Winograd engine for accelerating the calculation of medium-sized convolution kernels, which reduces the number of multiplications through linear transformation; and a GEMM engine for performing small-sized convolution and general matrix multiplication operations.

2. The medical image segmentation method based on an accelerated convolutional neural network as described in claim 1, characterized in that, The frequency domain complex multiplication includes: transforming the feature map from the spatial domain of the pixels to the frequency domain through discrete Fourier transform; performing element-wise multiplication in the frequency domain, and decomposing one complex multiplication into three real multiplications and five real additions / subtractions to obtain the real and imaginary parts of the complex result, thereby realizing convolution calculation in the spatial domain; coupling the real and imaginary parts of the two weighted multiplications to registers respectively, and adding a bias to the registers to realize the bit conversion from unsigned to signed.

3. The medical image segmentation method based on an accelerated convolutional neural network as described in claim 2, characterized in that, The bias is: ； in, and These are the least significant 8 bits of A and B, respectively; and These are the corresponding unsigned inputs; The input is a signed multiplicand, and inmode indicates the operation mode.

4. The medical image segmentation method based on an accelerated convolutional neural network as described in claim 2, characterized in that, The results of addition / subtraction calculations can be cached and reused directly in subsequent clock cycles.

5. The medical image segmentation method based on an accelerated convolutional neural network as described in claim 2, characterized in that, A set of packaged complex weights can be reused by multiple feature map activation values ​​from different input blocks, or by the real and imaginary parts of the same activation value respectively; After a set of weights has been calculated, a set of packaged complex activation values ​​is reused by the next set of weights from different output channels.

6. The medical image segmentation method based on an accelerated convolutional neural network as described in claim 1, characterized in that, The network structure, quantization parameters, and weight data of the convolutional neural network are stored in a non-volatile storage medium, and the convolutional neural network is pruned before storage based on the L1 norm to evaluate the importance of the filters.

7. The medical image segmentation method based on an accelerated convolutional neural network as described in claim 6, characterized in that, Before storage, the data bit width is adaptively allocated to the pruned convolutional neural network based on the sensitivity of the network layers.

8. A medical image segmentation system based on an accelerated convolutional neural network, characterized in that, include: An image receiving module is configured to receive medical images to be processed. The image segmentation module is configured to: for medical images, analyze the network structure of the convolutional neural network, and dynamically schedule the medical image segmentation task to the corresponding computing engine according to the type of convolution kernel in each network layer; wherein the computing engine includes: a frequency domain processing engine, used to process the calculation of large-size convolution kernels, using frequency domain complex multiplication for acceleration; a Winograd engine, used to accelerate the calculation of medium-size convolution kernels, reducing the number of multiplications through linear transformation; and a GEMM engine, used to perform small-size convolution and general matrix multiplication operations.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the medical image segmentation method based on an accelerated convolutional neural network as described in any one of claims 1-7.

10. A computer device comprising a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the medical image segmentation method based on an accelerated convolutional neural network as described in any one of claims 1-7.

Images