Miniature ultrasonic image reconstruction method, device and equipment based on deep learning

Abstract

The invention belongs to the technical field of image processing, and discloses a miniature ultrasonic image reconstruction method, device and equipment based on deep learning. The method comprises the following steps: constructing a lightweight U-Net model by a processed signal; a mixed loss function and a quantitative perception optimization and compression lightweight U-Net model are adopted, the processed model is deployed on an FPGA, a layered mapping strategy and a double-buffering mechanism are adopted to execute reasoning, and a reconstructed image is output; an up-sampling strategy is dynamically selected through a closed-loop feedback super-resolution module to optimize a reconstructed image, the ultrasonic imaging quality is improved through a lightweight U-Net architecture, hardware collaborative optimization and an end-to-end imaging pipeline, and a depth separable convolution, multi-scale attention gating and mixed loss function optimization algorithm is adopted; the power consumption is reduced through FPGA accelerated reasoning, dynamic voltage frequency adjustment and on-chip memory optimization, the imaging effect is improved through quantitative perception training, and the system is high in resolution, low in power consumption, high in real-time performance and suitable for portable medical equipment.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a micro-ultrasound image reconstruction method, device and equipment based on deep learning. Background Art

[0002] Traditional ultrasound imaging systems face numerous challenges in image reconstruction. Their complex algorithms require high computing power, making them difficult to run quickly on edge devices. Furthermore, they lack the ability to preserve detail in small lesions, impacting diagnostic accuracy. Furthermore, traditional systems consume high power, limiting their application in portable and wearable medical devices.

[0003] The above content is only used to assist in understanding the technical solution of the present invention and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of the present invention is to provide a micro-ultrasound image reconstruction method, device and equipment based on deep learning, aiming to solve the technical problems that it is difficult to run quickly on edge devices, the ability to retain details is insufficient when facing tiny lesions, affecting diagnostic accuracy, and the system power consumption is high, which limits its application in portable and wearable medical devices.

[0005] To achieve the above objectives, the present invention provides a micro-ultrasound image reconstruction method based on deep learning, which comprises the following steps:

[0006] Collecting raw ultrasonic radio frequency signals, preprocessing the raw ultrasonic radio frequency signals through adaptive beamforming technology embedded in a lightweight CNN, and outputting the processed signals;

[0007] Constructing a lightweight U-Net model using the processed signal, wherein the encoder and decoder of the lightweight U-Net model use depthwise separable convolutional layers, the convolution kernel weights are quantized to 4 to 8 bits, and a channel-spatial dual attention mechanism is embedded in the skip connections of the lightweight U-Net model to form multi-scale attention gating;

[0008] The lightweight U-Net model is optimized using a hybrid loss function that combines SSIM structural similarity loss, edge-sensitive loss, and perceptual loss.

[0009] The optimized model is compressed through quantization perception to obtain a quantized model, wherein the quantized model is compressed to INT8 precision, and the attention module retains 16-bit activation;

[0010] Deploy the quantization model on the FPGA, perform inference using a layered mapping strategy and a double buffering mechanism, and output a reconstructed image;

[0011] The reconstructed image is optimized by dynamically selecting an upsampling strategy through a closed-loop feedback super-resolution module, wherein the detail resolution of the optimized reconstructed image is 0.8 mm.

[0012] In some embodiments, constructing a lightweight U-Net model using the processed signal includes:

[0013] Constructing a depthwise separable convolutional layer using the processed signal, and replacing the standard separable convolutional layer of the initial lightweight U-Net model with the depthwise separable convolutional layer to obtain a lightweight U-Net model;

[0014] Perform global average pooling on the input sample feature map and calculate the attention weight through the fully connected layer and its activation function;

[0015] Multiplying the calculated attention weight by the sample feature map channel by channel to enhance the expressive power of the lightweight U-Net model;

[0016] Extracting multi-scale features of the sample feature map through convolution operations at different levels;

[0017] Concatenate or weightedly fuse feature maps of different scales to form a fused feature map;

[0018] At a specific network level, the semantic information and edge information in the fused feature map are enhanced through specific convolution operations and activation functions, and the processed feature map is multi-tasked to output classification, segmentation, or detection task results to verify the effectiveness of the lightweight U-Net model;

[0019] The resource monitoring hook function is used to check whether it is a specific hardware environment. If not, it returns directly to obtain the current hardware resource usage and convert it into the corresponding unit. If the resource usage exceeds the preset threshold, sparsity acceleration or structural optimization is activated. When the resource is exceeded, each layer of the model is traversed. If it is a convolutional layer, a random mask is generated and the convolution kernel weight is multiplied by the mask to make some weights zero, thereby achieving sparsification. At the same time, the network structure of the lightweight U-Net model is dynamically adjusted according to the characteristics of the input data.

[0020] In some embodiments, constructing a depthwise separable convolutional layer using the processed signal includes:

[0021] Quantize the convolution kernel weights of the convolution layer through hardware-aware initialization;

[0022] Generate a random tensor with the same shape as the convolution kernel weight of a preset lightweight U-Net model using the processed signal, wherein the value of the random tensor is in the range of [0, 1);

[0023] Generate a Boolean mask based on the random tensor according to the configured sparsity, and convert the Boolean mask to a floating-point type, where values ​​less than the sparsity are set to True, and values ​​other than the sparsity are set to False;

[0024] The generated mask is element-wise multiplied with the quantized convolutional layer’s kernel weights to construct a depth-wise separable convolutional layer.

[0025] In some embodiments, optimizing the lightweight U-Net model using a hybrid loss function includes:

[0026] Traverse each layer of the lightweight U-Net model, collect and count the feature statistics of each layer;

[0027] Calculate the mean of the feature statistics, calculate the dynamic sparsity based on the mean and preset parameters, limit the dynamic sparsity to the set range and return it;

[0028] Calculate the spatial domain importance score of the convolution kernel weight and the temporal domain importance score of the feature activity, and combine the spatial and temporal importance scores to generate a comprehensive score;

[0029] Sorting is performed according to the comprehensive score and the retained feature channels or convolution kernels are selected, and the network structure of the lightweight U-Net model is reconstructed using the retained feature channels or convolution kernels to complete the optimization.

[0030] In some embodiments, compressing the optimized model through quantization perception includes:

[0031] Get the target cache size, create a pruner based on the cache size, and use the pruner to traverse all layers of the model and perform structured pruning on the convolution kernel weights of each layer;

[0032] Obtaining a cache configuration of a target chip, and adjusting pruning, quantization, and sparse coding of the pruner according to the cache configuration; if the target chip is not of a specific type, returning to a default configuration;

[0033] Create a UNetPruner instance using the pruner, pass the UNetPruner instance into the optimized model, and use the method corresponding to the UNetPruner instance to prune the optimized model as a whole;

[0034] Apply dynamic quantization to the model and call the corresponding method for dynamic quantization to complete model compression.

[0035] In some embodiments, deploying the quantization model on an FPGA includes:

[0036] Traverse each layer of the model, obtain the convolution kernel weights of the current layer, convert the weights from dense format to CSR format, and obtain sparse weights, indices and pointers, and save the converted data to the layer;

[0037] Configure the hardware for a specific accuracy and sampling rate, set channel parameters, configure quantized sampling parameters, enable precision clocking and set the oversampling factor and clock interval, switch double buffering to achieve zero-wait switching, stop and restart hardware sampling, process completed buffer data within the specified time, start related pulse emission, set the drive pulse frequency, and start hardware continuous acquisition;

[0038] Traverse each feature unit, use SIMD vectorization to process the sum of the product of the convolution kernel weight and the input, process the non-vectorized part of the remaining elements, and determine whether the activation conditions are met based on the summation result;

[0039] If it is satisfied, the information about using SIMD optimization is output; if it is not satisfied, the information about using general implementation is output;

[0040] Use inline assembly to load convolution kernel weights and pulse data into registers, perform SIMD instruction-optimized multiplication and addition operations, and update feature maps. This loop continues until all data is processed.

[0041] Initialize the accumulator acc to 0, traverse each row of the convolution kernel weight matrix, and for each row, traverse each column of the convolution kernel weight matrix, calculate the product of the low-order weight and the high-order pulse data and add them to acc, and perform a shift operation on the accumulated result and store it in the high-order output array;

[0042] Input the compressed data pointer and output buffer pointer, as well as the maximum length. Every two bytes represent a value and the number of repetitions. They are decoded and filled into the output buffer in sequence until the maximum length is reached or the end of the data is reached.

[0043] The input is the quantized convolution kernel weight, pulse data, output pointer, scale factor and zero point. The convolution kernel weight matrix and pulse data are traversed by double loop, the weighted sum is calculated while considering the quantization zero point, and finally the output is dequantized.

[0044] Set up the model parser, error reporter, and memory pool, load model data, and allocate tensors to complete model deployment.

[0045] In some embodiments, the upsampling strategy is to use 8-bit quantized weights and activations for the initial convolution layer, 4-bit weight quantization for the depthwise separable convolution layer, 8-bit weights and 16-bit activation for the attention module, and construct a model distillation framework to migrate SwinTransformer knowledge.

[0046] In addition, to achieve the above objectives, the present invention also proposes a micro-ultrasound image reconstruction device based on deep learning, wherein the micro-ultrasound image reconstruction device based on deep learning comprises:

[0047] a signal processing module, configured to collect raw ultrasonic radio frequency signals, pre-process the raw ultrasonic radio frequency signals using adaptive beamforming technology embedded in a lightweight CNN, and output the processed signals;

[0048] A model construction module is used to construct a lightweight U-Net model using the processed signal, wherein the encoder and decoder of the lightweight U-Net model use depthwise separable convolutional layers, the convolution kernel weights are quantized to 4 to 8 bits, and a channel-spatial dual attention mechanism is embedded in the skip connections of the lightweight U-Net model to form multi-scale attention gating;

[0049] A model optimization module for optimizing the lightweight U-Net model using a hybrid loss function that combines SSIM structural similarity loss, edge-sensitive loss, and perceptual loss;

[0050] The model optimization module is used to compress the optimized model through quantization perception to obtain a quantized model, wherein the quantized model is compressed to INT8 precision and the attention module retains 16-bit activation;

[0051] An image reconstruction module, configured to deploy the quantization model on the FPGA, perform inference using a layered mapping strategy and a double buffering mechanism, and output a reconstructed image;

[0052] An image optimization module is used to dynamically select an upsampling strategy to optimize the reconstructed image through a closed-loop feedback super-resolution module, wherein the detail resolution of the optimized reconstructed image is 0.8 mm.

[0053] In some embodiments, the model construction module is configured to construct a depthwise separable convolutional layer using the processed signal, and replace the standard separable convolutional layer of the initial lightweight U-Net model with the depthwise separable convolutional layer to obtain a lightweight U-Net model;

[0054] Perform global average pooling on the input sample feature map and calculate the attention weight through the fully connected layer and its activation function;

[0055] Multiplying the calculated attention weight by the sample feature map channel by channel to enhance the expressive power of the lightweight U-Net model;

[0056] Extracting multi-scale features of the sample feature map through convolution operations at different levels;

[0057] Concatenate or weightedly fuse feature maps of different scales to form a fused feature map;

[0058] At a specific network level, the semantic information and edge information in the fused feature map are enhanced through specific convolution operations and activation functions, and the processed feature map is multi-tasked to output classification, segmentation, or detection task results to verify the effectiveness of the lightweight U-Net model;

[0059] The resource monitoring hook function is used to check whether it is a specific hardware environment. If not, it returns directly to obtain the current hardware resource usage and convert it into the corresponding unit. If the resource usage exceeds the preset threshold, sparsity acceleration or structural optimization is activated. When the resource is exceeded, each layer of the model is traversed. If it is a convolutional layer, a random mask is generated and the convolution kernel weight is multiplied by the mask to make some weights zero, thereby achieving sparsification. At the same time, the network structure of the lightweight U-Net model is dynamically adjusted according to the characteristics of the input data.

[0060] In addition, to achieve the above-mentioned objectives, the present invention also proposes a micro-ultrasound image reconstruction device based on deep learning, and the micro-ultrasound image reconstruction device based on deep learning includes: a memory, a processor, and a micro-ultrasound image reconstruction program based on deep learning stored on the memory and runnable on the processor, and the micro-ultrasound image reconstruction program based on deep learning is configured to implement the steps of the micro-ultrasound image reconstruction method based on deep learning as described above.

[0061] The present invention constructs a lightweight U-Net model through the processed signal; adopts a hybrid loss function and quantization perception to optimize and compress the lightweight U-Net model, deploys the processed model on the FPGA, and adopts a layered mapping strategy and a double buffering mechanism to perform inference and output a reconstructed image; dynamically selects an upsampling strategy through a closed-loop feedback super-resolution module to optimize the reconstructed image, improves the ultrasound imaging quality through a lightweight U-Net architecture, hardware collaborative optimization and an end-to-end imaging pipeline, and adopts a depth-separable convolution, multi-scale attention gating and hybrid loss function optimization algorithm; utilizes FPGA to accelerate inference, dynamic voltage and frequency adjustment and on-chip memory optimization to reduce power consumption, and improves imaging effects through quantization perception training. It has high resolution, low power consumption and strong real-time performance, and is suitable for portable medical devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a flowchart of the first embodiment of the micro-ultrasound image reconstruction method based on deep learning of the present invention;

[0063] Figure 2 Schematic diagram of the model architecture of the micro-ultrasound image reconstruction method based on deep learning of the present invention;

[0064] Figure 3 This is an energy efficiency curve diagram of the micro-ultrasound image reconstruction method based on deep learning of the present invention;

[0065] Figure 4 This is a schematic diagram of the layout of the FPGA accelerator logic unit in the micro-ultrasound image reconstruction method based on deep learning of the present invention;

[0066] Figure 5 This is a structural block diagram of the first embodiment of the micro-ultrasound image reconstruction device based on deep learning of the present invention.

[0067] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0068] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0069] The embodiment of the present invention provides a micro-ultrasound image reconstruction method based on deep learning, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of a micro-ultrasound image reconstruction method based on deep learning of the present invention.

[0070] In this embodiment, the micro-ultrasound image reconstruction method based on deep learning includes the following steps:

[0071] Step S10: collecting original ultrasonic radio frequency signals, preprocessing the original ultrasonic radio frequency signals by using adaptive beamforming technology embedded in a lightweight CNN, and outputting the processed signals.

[0072] In this embodiment, the executor of this embodiment is a micro-ultrasound image reconstruction device based on deep learning, wherein the micro-ultrasound image reconstruction device based on deep learning has functions such as data processing, data communication and program running. The micro-ultrasound image reconstruction device based on deep learning can be a computer terminal device or other network device, and of course it can also be other devices with similar functions, and this embodiment does not limit this.

[0073] It's important to note that traditional ultrasound imaging systems present numerous challenges in image reconstruction. Their algorithms are complex and require high computing power, making them difficult to run quickly on edge devices. Furthermore, they lack the ability to preserve detail in small lesions, impacting diagnostic accuracy. Furthermore, traditional systems consume high power, limiting their application in portable and wearable medical devices.

[0074] In order to solve the above technical problems, in this embodiment, a lightweight U-Net model is constructed through the processed signal; a hybrid loss function and quantization perception are used to optimize and compress the lightweight U-Net model, the processed model is deployed on the FPGA, and a hierarchical mapping strategy and a double buffering mechanism are used to perform inference and output the reconstructed image; the upsampling strategy is dynamically selected through the closed-loop feedback super-resolution module to optimize the reconstructed image, and the ultrasound imaging quality is improved through the lightweight U-Net architecture, hardware collaborative optimization and end-to-end imaging pipeline, and the depth-separable convolution, multi-scale attention gating and hybrid loss function optimization algorithm are adopted; FPGA is used to accelerate inference, dynamic voltage and frequency adjustment and on-chip memory optimization to reduce power consumption, and quantization perception training is used to improve imaging effects. It has high resolution, low power consumption and strong real-time performance, and is suitable for portable medical devices. Specifically, it can be implemented as follows.

[0075] It should be noted that the model architecture diagram in this embodiment can refer to Figure 2As shown in the figure, Transposed (Transposed Conv) represents transposed convolution, also known as deconvolution. It is used for upsampling operations in some tasks to increase the size of feature maps. It appears multiple times in the figure, indicating that in the lightweight U-Net architecture, transposed convolution is used multiple times for operations such as restoring the resolution of feature maps. Concatenate represents a splicing operation. In deep learning, feature maps of different layers are often spliced ​​along a certain dimension (such as the channel dimension) to integrate feature information at different levels. It appears multiple times in the figure, indicating that feature maps are spliced ​​in different parts of the architecture. Channel Attention represents an attention mechanism that focuses on the importance differences between different channels of a feature map. By performing operations such as weighting on the channel dimension, it enhances the features of important channels and suppresses the features of unimportant channels, allowing the network to focus more on key information. Spatial Attention represents another attention mechanism that focuses on the importance differences between different locations in the feature map along the spatial dimension. By weighting spatial locations and other operations, it highlights important spatial regions and ignores irrelevant areas, helping the network more accurately locate and process information such as objects. Channel / SperialAgientor represents a module that combines channel and spatial attention. It can be used to comprehensively consider the importance distribution of features in both the channel and spatial dimensions, enabling more comprehensive feature optimization and information screening.

[0076] The overall process in this embodiment includes the model architecture (algorithm innovation) design of a lightweight U-Net architecture, replacing the convolutional layers of the standard U-Net with depthwise separable convolutions to reduce computational complexity and parameter requirements. A channel-spatial dual attention mechanism is embedded in the skip connection to form multi-scale attention gating (MSAG), improving the ability to preserve details of small lesions. A hybrid loss function is designed, combining the SSIM structural similarity loss and the edge-sensitive loss to optimize image reconstruction quality. Hardware collaborative optimization (system innovation) involves designing a customized FPGA data flow architecture, hierarchically mapping the U-Net encoder / decoder to logic units to achieve fast real-time inference. Dynamic voltage frequency scaling (DVFS) technology is developed to dynamically adjust processor frequency based on image complexity, reducing power consumption. A double buffering mechanism is used to manage feature map storage, reducing DDR access times and bandwidth usage. In the end-to-end imaging pipeline, a lightweight CNN is embedded in the raw RF signal stage, replacing the traditional DAS algorithm, improving the signal-to-noise ratio and enabling adaptive beamforming preprocessing. Quantization-aware training is adopted, using 8-bit integer quantization (INT8) compatible training to ensure lossless model deployment on edge devices. Build an end-to-end ultrasound imaging pipeline, including signal acquisition, preprocessing, image reconstruction and post-processing, to achieve a complete imaging process.

[0077] The above process can refer to the following embodiments, such as embodiment 1, hardware collaborative optimization: design a customized FPGA data flow architecture, and map the U-Net encoder / decoder layer by layer to the logic unit. Implement inter-layer pipelining to improve inference speed. Develop dynamic voltage frequency scaling (DVFS) technology to dynamically adjust the processor frequency according to image complexity to balance performance and power consumption. Use a double buffering mechanism to manage feature map storage, reduce DDR access times, and reduce bandwidth usage. Complete hardware collaborative optimization and improve overall system performance. Using DVFS technology can effectively control advantages, Figure 3 This is an energy efficiency curve chart that depicts the relationship between power consumption and frequency under the DVFS mechanism. "DVFS Energy Efficiency Curve" represents the dynamic voltage and frequency scaling (DVFS) energy efficiency curve, "DVFS Power Curve" represents the dynamic voltage and frequency scaling power curve, which is the curve formed by the data points in the chart, and "Operating Points" represent operating points. "Power Consumption (W)" represents power consumption (watts), with the vertical axis representing processor power consumption in watts, which is used to measure energy consumption at different processor frequencies. "Processor Frequency (MHz)" represents processor frequency (megahertz), with the horizontal axis representing the processor's operating frequency. It is used to display relevant performance and energy consumption data under different frequency settings, helping to analyze the impact of frequency on power consumption, etc.

[0078] Example 2, end-to-end imaging pipeline construction. Embed a lightweight CNN in the raw RF signal stage to replace the traditional DAS algorithm, improve the signal-to-noise ratio, and realize adaptive beamforming preprocessing. Use quantization-aware training and 8-bit integer quantization (INT8) compatible training to ensure lossless deployment of the model on edge devices. Build an end-to-end ultrasound imaging pipeline, covering signal acquisition, preprocessing, image reconstruction, and post-processing to achieve a complete imaging process. Complete the end-to-end imaging pipeline construction to ensure imaging quality and efficiency.

[0079] Example 3, model architecture innovation and optimization. On the basis of lightweight U-Net, the Transformer module is further introduced to form a hybrid CNN-Transformer architecture, which enhances the global feature capture capability of the model while retaining local detail information and improving the recognition accuracy of complex lesions. An adaptive multi-scale fusion (AMSF) module is designed to dynamically fuse feature maps of different levels, and intelligently adjust the weights of each feature map according to the size and morphology of different lesions, thereby improving the explicit expression capability of tiny lesions (such as cysts less than 1mm) and complex structures (such as vascular bifurcations), making these tiny or complex structures more clearly identifiable in the image. The hybrid loss function is optimized, and the perceptual loss (Perceptual Loss) is added on the basis of the SSIM structural similarity loss and the edge-sensitive loss. By referring to high-quality medical image features, the visual quality and diagnostic value of the reconstructed image are further improved, so that the reconstructed image is closer to the visual effect of the real medical image while maintaining structural similarity.

[0080] Example 4: Hardware Collaborative Optimization and Upgrade. A custom ASIC was designed based on the RISC-V architecture. A dedicated instruction set was designed for key computing tasks in the ultrasound imaging pipeline, such as convolution operations and matrix multiplication. This improved instruction execution efficiency, significantly improving inference speed compared to general-purpose processors, reaching over 100 fps and meeting the requirements of real-time imaging. The memory hierarchy was optimized, adopting a three-level cache architecture (L1: registers, L2: SRAM, L3: DDR). Efficient data transmission channels were designed between each cache level to ensure fast data read and write. A reasonable cache allocation strategy improved memory bandwidth utilization by over 50% compared to traditional architectures, effectively alleviating the impact of memory bandwidth bottlenecks on system performance. A dynamic energy efficiency management (DEM) system was developed. This system combines a task scheduling algorithm with DVFS technology to dynamically adjust the processor voltage and frequency based on the priority and complexity of the current imaging task, achieving fine-grained power consumption control. During low-load tasks, the frequency is reduced to save energy; during high-load tasks, the frequency is increased to ensure performance, thereby minimizing system power consumption while ensuring imaging quality.

[0081] Example 5, data flow enhancement. The implementation of real-time adaptive beamforming (RABF) technology, by embedding a lightweight LSTM network in the preprocessing stage, analyzes the characteristics of the ultrasound signal in real time, dynamically optimizes the size and shape of the transmit / receive aperture, improves the beam directivity and signal focusing, and thus improves the lateral resolution and contrast of the image, making the tissue boundary clearer and the lesion contour sharper. A closed-loop feedback super-resolution (CFSR) module is designed, and a reinforcement learning algorithm is introduced in the post-processing stage. According to the content characteristics and quality evaluation indicators of the image, the optimal upsampling strategy is dynamically selected to achieve super-resolution reconstruction of the image. This module can automatically adjust the upsampling parameters according to different imaging scenarios and requirements, improve the image detail expression and visual effects, and reduce the generation of artifacts and noise.

[0082] Example 6, Quantization and Deployment Optimization. A mixed precision quantization (HPQ) strategy is implemented, and different quantization accuracies are used for different layers of the model according to their computational characteristics and sensitivity to precision: Initial convolution layer: Both weights and activations are quantized to 8-bit, reducing computational complexity and storage requirements while ensuring the accuracy of input feature extraction. Depthwise separable convolution layer: Weights are quantized to 4-bit, and activations remain at 8-bit. Since the computational characteristics of depthwise separable convolution have relatively low requirements for weight precision, reducing weight precision can significantly reduce the amount of computation, while maintaining high activation precision to maintain the quality of feature maps. Attention module: Weights are quantized to 8-bit, and activations are quantized to 16-bit. The attention module models the inter-channel relationship and spatial position relationship of the feature map and is more sensitive to precision. Therefore, the precision of activation is appropriately increased during quantization to ensure that the attention mechanism can accurately capture important information in the feature map and ensure the model's ability to recognize detailed features such as tiny lesions. A model distillation framework is constructed, and a large teacher model (such as Swin Transformer) is used to guide the training of a lightweight student model (such as a lightweight U-Net). By gradually migrating the knowledge of the teacher model (such as feature representation, attention distribution, etc.) to the student model, the performance of the student model is improved under limited computing resources, so that it can inherit some excellent features of the teacher model while maintaining a lightweight architecture, thereby improving the reconstruction quality and diagnostic accuracy of complex medical images.

[0083] Example 7, system-level performance improvement. An integrated energy harvesting module uses piezoelectric materials to convert the mechanical vibration of the ultrasonic probe during operation into electrical energy, and an efficient energy management system is designed to store the collected energy in a rechargeable battery or supercapacitor. This module can provide an additional 0.2W of continuous power when the device is working normally, effectively extending the battery life of the device and improving the ease of use and battery life of portable ultrasound equipment. Wireless collaborative imaging function is realized, and through the LoRa communication protocol, multiple miniature ultrasound devices can communicate with each other and share some computing tasks. In complex imaging scenarios, such as large-area tissue scanning or simultaneous detection of multiple lesions, a single device can distribute part of the computing load to other devices, thereby reducing its own computing pressure and power consumption, and improving the imaging efficiency and performance of the entire system. At the same time, the long-distance communication capability and low power consumption characteristics of the LoRa protocol ensure that the devices can still maintain a long battery life and stable communication connection when working together.

[0084] Example 8, clinical verification enhancement. Pathology perception training (PAT) is carried out, and various simulated lesions (including cysts, tumors, etc. of different sizes, shapes and types) are injected into the ultrasound image data set, and the model is trained in a targeted manner in combination with real pathological diagnosis results. In this way, the model can learn more features and patterns related to lesions, significantly improve the sensitivity and recognition accuracy of abnormal structures, reduce the risk of missed diagnosis and misdiagnosis, and improve the reliability and practicality of the system in clinical diagnosis. A real-time quality monitoring (RQM) module is deployed, a lightweight CNN (model size is less than 100KB) is designed, and the quality indicators of the reconstructed image (such as signal-to-noise ratio, contrast, resolution, etc.) are evaluated in real time. When the image quality does not meet the preset standards, the module can promptly trigger the system's resampling mechanism or automatically adjust the parameters of the imaging algorithm to optimize the image quality and ensure that the output image always meets the requirements of clinical diagnosis, thereby improving the stability and diagnostic value of the entire ultrasound imaging system.

[0085] Example 9, performance comparison after optimization. Compare the system performance indicators before and after optimization to verify the effectiveness of the optimization scheme: Spatial resolution: increased from 1.0mm to 0.8mm, the image detail resolution ability is improved by 20%, and it can more clearly display the boundaries of tiny lesions and tissue structures, providing stronger support for early disease diagnosis. Peak signal-to-noise ratio (PSNR): increased from 38.2dB to 42.7dB, the noise level of the image is significantly reduced, the visual quality is significantly improved, and the accuracy and confidence of the doctor's interpretation of the image are improved. Inference delay: reduced from 5.0ms to 2.3ms, the system response speed is accelerated by 54%, achieving faster real-time imaging, and meeting the needs of clinical application scenarios with high real-time requirements such as interventional surgery. Power consumption: Continuous working power consumption is reduced from 0.9W to 0.6W. While maintaining high performance, the power consumption is reduced by 33%, further extending the battery life of the device and improving portability and ease of use. Model size: Compressed from 2.0MB to 0.8MB, a 60% reduction in model size, making it easier to deploy and update on resource-constrained embedded devices, reducing device storage costs and hardware requirements.

[0086] In practice, this embodiment requires collecting raw ultrasonic RF signals, preprocessing them using adaptive beamforming technology embedded in a lightweight CNN, and outputting the processed signals. Embedding a lightweight CNN in the raw RF signal replaces the traditional DAS algorithm, improving the signal-to-noise ratio and enabling adaptive beamforming preprocessing.

[0087] Step S20: constructing a lightweight U-Net model using the processed signal.

[0088] In the specific implementation, in this embodiment, it is necessary to first use the processed signal to construct a lightweight U-Net model, specifically to build a hardware-aware lightweight U-Net model. The hardware-aware lightweight U-Net model includes the following processes: hardware-aware initialization, construction of a depth-separable convolution layer, introduction of an attention gating mechanism, realization of network forward propagation, real-time optimization, and dynamic structural adjustment. Among them, hardware-aware initialization is to quantize the encoding convolution kernel weights and register a resource monitoring hook. Quantizing the encoding convolution kernel weights is to quantize the convolution kernel weights of the convolution layer, and the number of quantization bits is specified by the configuration. Registering a resource monitoring hook registers a forward propagation hook for a specific layer to monitor resource usage. Constructing a depth-separable convolution layer is to generate a depth-separable convolution mask and apply a depth-separable convolution mask.

[0089] Furthermore, constructing a depthwise separable convolutional layer involves generating a depthwise separable convolution mask and applying it. Generating the depthwise separable convolution mask involves creating a random tensor with the same shape as the convolution kernel weights, whose values ​​are in the range [0, 1). A Boolean mask is then generated based on the configured sparsity, setting values ​​less than the sparsity to True and the rest to False. Finally, the Boolean mask is converted to a floating-point type. Applying the depthwise separable convolution mask involves element-wise multiplication of the generated mask with the convolution kernel weights of the convolution layer, thereby partially zeroing out the kernel weights and achieving sparsification. The introduction of an attention gating mechanism involves calculating attention weights across feature map channels and applying the attention weights to the feature maps. Calculating the attention weights across feature map channels involves performing global average pooling on the input feature map, and then calculating the attention weights through a fully connected layer and its activation function. Applying the attention weights to the feature map involves channel-wise multiplication of the calculated attention weights with the input feature map, enhancing the expressiveness of the feature map. Implementing the network's forward propagation involves multi-scale feature extraction, feature fusion, semantic information enhancement, edge information preservation, dynamic feature selection, and multi-task output processing. Multi-scale feature extraction involves extracting multi-scale features from the input image through convolution operations at different levels. Feature fusion involves concatenating or weightedly fusing feature maps at different scales to form a fused feature map. Semantic information enhancement involves enhancing semantic information in feature maps at specific network levels through specific convolution operations and activation functions. Edge information preservation involves preserving edge information in feature maps through specific convolution kernel designs and activation functions at specific network levels. Dynamic feature selection involves dynamically selecting important features in feature maps during the inference phase based on the characteristics of the input data. Multi-task output processing involves performing multi-task processing on the fused feature map, outputting multi-task results such as classification, segmentation, or detection. Real-time optimization involves a resource monitoring hook function that checks for a specific hardware environment. If not, it returns directly, obtains current hardware resource usage, and converts it to the corresponding units. If resource usage exceeds a preset threshold, sparsity acceleration or structural optimization is activated. Dynamic structural adjustment is to traverse each layer of the model when resources are exceeded. If it is a convolutional layer, a random mask is generated and the convolution kernel weight is multiplied by the mask to make some weights zero, thereby achieving sparsification. At the same time, the network structure is dynamically adjusted according to the characteristics of the input data.

[0090] In a specific implementation, the model construction process is to use the processed signal to construct a depth-wise separable convolution layer, and replace the standard separable convolution layer of the initial lightweight U-Net model with the depth-wise separable convolution layer to obtain a lightweight U-Net model;

[0091] Perform global average pooling on the input sample feature map and calculate the attention weight through the fully connected layer and its activation function;

[0092] Multiplying the calculated attention weight by the sample feature map channel by channel to enhance the expressive power of the lightweight U-Net model;

[0093] Extracting multi-scale features of the sample feature map through convolution operations at different levels;

[0094] Concatenate or weightedly fuse feature maps of different scales to form a fused feature map;

[0095] At a specific network level, the semantic information and edge information in the fused feature map are enhanced through specific convolution operations and activation functions, and the processed feature map is multi-tasked to output classification, segmentation, or detection task results to verify the effectiveness of the lightweight U-Net model;

[0096] The resource monitoring hook function is used to check whether it is a specific hardware environment. If not, it returns directly to obtain the current hardware resource usage and convert it into the corresponding unit. If the resource usage exceeds the preset threshold, sparsity acceleration or structural optimization is activated. When the resource is exceeded, each layer of the model is traversed. If it is a convolutional layer, a random mask is generated and the convolution kernel weight is multiplied by the mask to make some weights zero, thereby achieving sparsification. At the same time, the network structure of the lightweight U-Net model is dynamically adjusted according to the characteristics of the input data.

[0097] Among them, the processed signal is used to construct a depth-separable convolution layer, including: quantizing the convolution kernel weights of the convolution layer through hardware-aware initialization; using the processed signal to generate a random tensor with the same shape as the convolution kernel weights of a preset lightweight U-Net model, and the value of the random tensor is in the range of [0,1); generating a Boolean mask based on the random tensor according to the configured sparsity, and converting the Boolean mask to a floating-point type, wherein values ​​less than the sparsity are set to True, and values ​​of other sparsities are set to False; multiplying the generated mask by the convolution kernel weights of the quantized convolution layer element by element to construct a depth-separable convolution layer.

[0098] Step S30: Optimizing the lightweight U-Net model using a hybrid loss function.

[0099] Step S40: compressing the optimized model through quantization perception to obtain a quantized model.

[0100] It should be noted that the overall model optimization process in this embodiment includes a pruning strategy based on feature graph activity and collecting feature statistics of each layer of the lightweight U-Net. The pruning strategy based on feature graph activity includes obtaining feature statistics of each layer, traversing each layer, dynamically adjusting the pruning rate (if dynamic pruning is enabled), calculating the spatiotemporal joint pruning score, sorting and selecting the retained feature channels or convolution kernels according to the scores, performing spatiotemporal joint pruning, progressive gradient-aware pruning, and then reconstructing the network layer. Wherein, according to claim 20, the feature statistics of each layer of the lightweight U-Net are collected, characterized in that the statistical dictionary is initialized, each sample is traversed, the feature data of each layer is obtained through the forward propagation of the model, the data of each layer is accumulated, the feature activity is recorded, and the average activity of each layer is calculated. Wherein, the network layer is reconstructed to obtain the original layer object, the convolution kernel weights and biases are clipped according to the retained index, and the clipped weights and biases are returned. The pruning rate of each layer is dynamically adjusted to calculate the mean of the feature statistics, the dynamic sparsity is calculated according to the mean and preset parameters, the dynamic sparsity is limited to a certain range and returned. Joint spatiotemporal pruning calculates the spatial importance score of the convolution kernel weights and the temporal importance score of the feature activity, combining the spatial and temporal importance scores to generate a comprehensive score. Progressive gradient-aware pruning traverses the model modules, finds all convolutional layers, generates a mask based on the gradient, updates the existing mask, selects the locations of the top k maximum values, and generates a new mask.

[0101] Furthermore, the process of model compression is to obtain the target cache size, create a pruner based on the cache size, and use the pruner to traverse all layers of the model to perform structured pruning on the convolution kernel weights of each layer; obtain the cache configuration of the target chip, and adjust the pruning, quantization and sparse coding of the pruner according to the cache configuration. If the target chip is not of a specific type, the default configuration is returned; use the pruner to create a UNetPruner instance, pass the UNetPruner instance into the optimized model, and use the corresponding method of the UNetPruner instance to perform overall pruning on the optimized model; apply dynamic quantization to the model, call the corresponding method for dynamic quantization to complete model compression.

[0102] In specific implementations, hardware-aware pruning optimization, joint optimization, dynamic pruning, quantization compression, sparse coding, and format conversion are required before model deployment. Hardware-aware pruning optimization involves obtaining the target cache size, creating a cache-size-based pruner, traversing all layers of the model, and performing structured pruning on the convolution kernel weights at each layer. Joint optimization involves obtaining the cache configuration of the target chip (L1 and L2 cache sizes) and adjusting pruning, quantization, and sparse coding strategies accordingly. If the target chip is not of a specific type, the default configuration is returned. Dynamic pruning involves applying pruning operations to reduce model complexity. A UNetPruner instance is created, the current model and pruning rate are passed in, and the model is pruned and updated using its methods. Quantization compression involves applying dynamic quantization to the model, calling methods to perform dynamic quantization, and updating the model. Sparse coding involves creating an encoder, traversing the model layers, and run-length encoding any sparse data in a layer, saving the result. Format conversion involves converting the instruction format based on the target chip type and converting the convolution kernel weights at each layer of the model to the CRS sparse storage format. The instruction format is converted according to the target chip type. If the target device is a specific chip type, the method is called for format conversion, and then the convolution kernel weights of each layer are converted to the CRS sparse storage format. It is characterized by traversing each layer of the model, obtaining the convolution kernel weights of the current layer, converting the weights from dense format to CSR format, and obtaining sparse weights, indexes, and pointers. The format of the update layer is 'csr', and the converted data is saved to the layer.

[0103] Step S50: deploying the quantization model on the FPGA, and performing inference using a layered mapping strategy and a double buffering mechanism to output a reconstructed image.

[0104] In the specific implementation, the deployment process is to traverse each layer of the model, obtain the convolution kernel weights of the current layer, convert the weights from dense format to CSR format, and obtain sparse weights, indices and pointers, and save the converted data to the layer; configure the hardware to a specific accuracy and sampling rate, and set the channel parameters, configure the quantized sampling parameters, enable the precision clock and set the oversampling multiple and clock interval, switch the double buffer to achieve zero-wait switching, stop and restart the hardware sampling, process the buffer data completed within the specified time, start the relevant pulse emission, set the driving pulse frequency, and start the hardware continuous acquisition; traverse each feature unit, use SIMD vectorization to process the sum of the product of the convolution kernel weight and the input, process the non-vectorized part of the remaining elements, and judge whether the activation conditions are met based on the summation result; if it is met, output the information using SIMD optimization, if not, output the information using general implementation; use inline assembly to load the convolution kernel weights and pulse data into registers, execute The multiplication and addition operation optimized by the row SIMD instruction is performed and the feature map is updated. The loop is processed until all data is processed; the accumulator acc is initialized to 0, and each row of the convolution kernel weight matrix is ​​traversed. For each row, each column of the convolution kernel weight matrix is ​​traversed, and the product of the low-order weight and the high-order pulse data is calculated and added to acc. The accumulated result is shifted and stored in the high-order output array; the input is the compressed data pointer and output buffer pointer, as well as the maximum length. Every two bytes represent a value and the number of repetitions. They are decoded and filled into the output buffer in sequence until the maximum length is reached or the data ends; the input is the quantized convolution kernel weight, pulse data, output pointer, scaling factor and zero point. The convolution kernel weight matrix and pulse data are traversed through a double loop to calculate the weighted sum and consider the quantization zero point. Finally, dequantization is performed to obtain the output; the model parser, error reporter and memory pool are set up to load the model data and allocate tensors to complete the model deployment. The layout of the FPGA accelerator logic unit is as follows Figure 4 As shown, it reflects the distribution of each processing unit and the data flow.

[0105] In some embodiments, deployment includes initializing deployment monitoring, data processing, and hardware configuration. Among them, initializing deployment monitoring is to initialize PPOAgent to initialize a reinforcement learning-based agent PPOAgent, pass in the environment configuration, and set the action space to multiple discrete actions for adjusting the pruning rate. Automatic pruning strategy search is to obtain the initial state of the model, and repeat the following steps in multiple training cycles: select an action according to the current state; prune the model according to the selected action; evaluate the performance of the pruned model and calculate the reward; update the state, action, and reward of the agent. Select pruning is to define a pruning rate list, select the corresponding pruning rate according to the action index, and return the pruned model. Monitoring storage is to check whether the resource usage exceeds a certain proportion. If it exceeds, the emergency pruning function is called to process the model, and finally the processed model is returned.

[0106] Furthermore, the hardware configuration in data processing and hardware configuration includes hardware modules, sparse U-Net inference SIMD instruction optimization, initializing the U-Net library, processing pulse events, mixed precision inference, decoding data, quantized inference, initializing the neural network engine, processing sensor data and performing inference.

[0107] The hardware module is configured to a specific precision and sampling rate, and channel parameters are set. Quantized sampling parameters are configured, precision clocks are enabled, and the oversampling factor and clock interval are set. DMA interrupts are configured and interrupt service functions are registered. The DMA interrupt is configured to switch to double buffering for zero-wait switching, stop and restart hardware sampling, process buffered data within the specified time, initiate relevant pulse emission, set the drive pulse frequency, and then start hardware continuous acquisition. The SIMD instruction optimization for sparse U-Net inference involves iterating through each feature unit, using SIMD vectorization to perform the product sum of the convolution kernel weights and input, processing the non-vectorized portion of the remaining elements, and determining whether the activation condition is met based on the summation result. The U-Net library is initialized to output SIMD optimization information if the compilation environment is a specific environment type, otherwise outputting information that a general implementation is used. Pulse event processing involves using inline assembly to load the convolution kernel weights and pulse data into registers, perform SIMD-optimized multiplication and addition operations, and update the feature map. This process is looped until all data has been processed. Mixed-precision inference initializes the accumulator acc to 0, iterates over each row of the convolution kernel weight matrix, and for each row, iterates over each column of the convolution kernel weight matrix. The product of the low-order weight and the high-order spike data is calculated and added to acc. The accumulated result is bit-shifted and stored in the high-order output array. The decoded data consists of the compressed input data pointer and output buffer pointer, as well as the maximum length. Each two bytes represents a value and the number of repetitions. The decoded data is sequentially decoded and padded to the output buffer until the maximum length is reached or the end of data is reached. Quantized inference takes as input the quantized convolution kernel weights, spike data, output pointer, scale factor, and zeros. A double loop iterates over the convolution kernel weight matrix and spike data, calculates the weighted sum, accounting for quantization zeros, and finally dequantizes the output. Initializing the neural network engine involves setting up the model parser, error reporter, and memory pool, loading model data, and allocating tensors. Processing sensor data and performing inference involves copying the input data to the input tensor, performing inference, logging the error and returning if inference fails, and obtaining the output tensor and sending the result if inference succeeds.

[0108] Step S60: dynamically selecting an upsampling strategy through a closed-loop feedback super-resolution module to optimize the reconstructed image.

[0109] It should be noted that the upsampling strategy is to use 8-bit quantized weights and activations in the initial convolutional layer, 4-bit weight quantization in the depthwise separable convolutional layer, 8-bit weights and 16-bit activation in the attention module, and build a model distillation framework to transfer Swin Transformer knowledge.

[0110] It's also important to emphasize that defining a hardware-aware lightweight U-Net model, optimizing it, and deploying it follow a sequential and progressive relationship. After defining a model, optimizing it, and then flexibly adjusting its configuration based on the target model and actual requirements (such as input data format and sampling rate), hardware-aware initialization, building depthwise separable convolutional layers, introducing an attention gating mechanism, implementing network forward propagation, real-time optimization, and dynamic structural adjustment all follow a sequential and progressive relationship. This sequential and progressive relationship is characterized by hardware-aware initialization serving as the foundation for the remaining actions. Next, building depthwise separable convolutional layers, introducing an attention gating mechanism, implementing network forward propagation, and then real-time optimization, dynamically adjusting the structure based on training progress to adapt to different training stages. Quantizing the encoded convolution kernel weights and registering resource monitoring hooks form a synergistic relationship. Quantizing the weights reduces resource usage, while the resource monitoring hooks further optimize resource management, ensuring efficient resource usage during model execution. Multi-scale feature extraction, feature fusion, semantic information enhancement, edge information preservation, dynamic feature selection, and multi-task output processing follow a sequential and progressive relationship. Multi-scale feature extraction provides the initial feature representation for forward propagation, feature fusion preliminarily integrates features of different scales, semantic information enhancement improves the semantic expressiveness of features, edge information preservation captures edge information of the data, supporting detail processing, dynamic feature selection further optimizes feature utilization, and multi-task output processing implements multi-task output during the inference phase of forward propagation. The sequential and progressive relationship involves collecting feature statistics from each layer of the lightweight U-Net and implementing a pruning strategy based on feature graph activity. The pruning strategy based on feature graph activity obtains this collected feature statistics and calculates a spatiotemporal joint pruning score. Feature channels or convolution kernels to be retained are then sorted and selected. After pruning, the network layers are reconstructed. Hardware-aware pruning optimization, joint optimization, dynamic pruning, quantization compression, sparse coding, format conversion, initialization deployment monitoring, data processing, and hardware configuration follow a sequential and progressive relationship. Pruning is performed based on hardware requirements, followed by joint optimization using various optimization techniques such as quantization and sparse coding. Dynamic pruning complements joint optimization, further improving the model's efficiency during the inference phase. After converting the model to a format supported by the target hardware, the model is deployed to the target device and initialization deployment monitoring is performed. The model's performance on the target device is verified, and hardware configuration optimization is performed based on hardware characteristics. Initializing PPOAgent, automatically searching for pruning strategies, selecting pruning strategies, and monitoring storage follow a sequential and progressive relationship. Initializing PPOAgent initializes the network structure, automatically searching for pruning strategies, selecting the optimal pruning strategy based on the search results, and monitoring the model's storage usage during the pruning process.Configuring the hardware module, initializing the neural network engine, initializing the U-Net library, optimizing the SIMD instructions for sparse U-Net inference, processing sensor data and performing inference, processing pulse events, decoding data, mixed precision inference, and quantized inference are sequential and progressive relationships. The sequential and progressive relationship is that after the hardware module configuration is completed, the neural network engine is initialized to prepare for inference, the U-Net library is initialized to provide support for sparse U-Net inference, and after the U-Net library is initialized, the sparse U-Net inference is optimized, and the SIMD (single instruction multiple data) instruction set is used to accelerate the inference process. The sensor data collected by the hardware module is sent to the neural network engine to start executing the inference task. During the U-Net inference process, pulse events are processed and the processed pulse events are decoded into a data format that can be understood by subsequent processing units. In order to further optimize the inference performance, mixed precision inference technology is adopted. On the basis of mixed precision inference, the model is further quantized to further reduce storage requirements and computational complexity.

[0111] This embodiment constructs a lightweight U-Net model using the processed signal; adopts a hybrid loss function and quantization-aware optimization and compression to the lightweight U-Net model, deploys the processed model on an FPGA, and uses a layered mapping strategy and a double-buffering mechanism to perform inference and output a reconstructed image; dynamically selects an upsampling strategy through a closed-loop feedback super-resolution module to optimize the reconstructed image, improves ultrasound imaging quality through a lightweight U-Net architecture, hardware collaborative optimization, and an end-to-end imaging pipeline, and adopts a depthwise separable convolution, multi-scale attention gating, and hybrid loss function optimization algorithm; utilizes FPGA-accelerated inference, dynamic voltage and frequency adjustment, and on-chip memory optimization to reduce power consumption, and improves imaging effects through quantization-aware training. The embodiment has high resolution, low power consumption, and strong real-time performance, making it suitable for portable medical devices.

[0112] Reference Figure 5 , Figure 5 This is a structural block diagram of the first embodiment of the micro-ultrasound image reconstruction device based on deep learning of the present invention.

[0113] like Figure 5 As shown, the micro-ultrasound image reconstruction device based on deep learning proposed in an embodiment of the present invention includes:

[0114] The signal processing module 10 is used to collect the original ultrasonic radio frequency signal, pre-process the original ultrasonic radio frequency signal by using the adaptive beamforming technology embedded in the lightweight CNN, and output the processed signal;

[0115] a model construction module 20 for constructing a lightweight U-Net model using the processed signal, wherein the encoder and decoder of the lightweight U-Net model use depthwise separable convolutional layers, the convolution kernel weights are quantized to 4 to 8 bits, and a channel-spatial dual attention mechanism is embedded in the skip connections of the lightweight U-Net model to form multi-scale attention gating;

[0116] A model optimization module 30 is configured to optimize the lightweight U-Net model using a hybrid loss function that combines SSIM structural similarity loss, edge-sensitive loss, and perceptual loss;

[0117] The model optimization module 30 is used to compress the optimized model through quantization perception to obtain a quantized model, wherein the quantized model is compressed to INT8 precision and the attention module retains 16-bit activation;

[0118] An image reconstruction module 40 is configured to deploy the quantization model on the FPGA, perform inference using a layered mapping strategy and a double buffering mechanism, and output a reconstructed image;

[0119] The image optimization module 50 is configured to dynamically select an upsampling strategy to optimize the reconstructed image through a closed-loop feedback super-resolution module, wherein the detail resolution of the optimized reconstructed image is 0.8 mm.

[0120] This embodiment constructs a lightweight U-Net model using the processed signal; adopts a hybrid loss function and quantization-aware optimization and compression to the lightweight U-Net model, deploys the processed model on an FPGA, and uses a layered mapping strategy and a double-buffering mechanism to perform inference and output a reconstructed image; dynamically selects an upsampling strategy through a closed-loop feedback super-resolution module to optimize the reconstructed image, improves ultrasound imaging quality through a lightweight U-Net architecture, hardware collaborative optimization, and an end-to-end imaging pipeline, and adopts a depthwise separable convolution, multi-scale attention gating, and hybrid loss function optimization algorithm; utilizes FPGA-accelerated inference, dynamic voltage and frequency adjustment, and on-chip memory optimization to reduce power consumption, and improves imaging effects through quantization-aware training. The embodiment has high resolution, low power consumption, and strong real-time performance, making it suitable for portable medical devices.

[0121] In some embodiments, the model construction module 20 is configured to construct a depthwise separable convolutional layer using the processed signal, and replace the standard separable convolutional layer of the initial lightweight U-Net model with the depthwise separable convolutional layer to obtain a lightweight U-Net model;

[0122] Perform global average pooling on the input sample feature map and calculate the attention weight through the fully connected layer and its activation function;

[0123] Multiplying the calculated attention weight by the sample feature map channel by channel to enhance the expressive power of the lightweight U-Net model;

[0124] Extracting multi-scale features of the sample feature map through convolution operations at different levels;

[0125] Concatenate or weightedly fuse feature maps of different scales to form a fused feature map;

[0126] At a specific network level, the semantic information and edge information in the fused feature map are enhanced through specific convolution operations and activation functions, and the processed feature map is multi-tasked to output classification, segmentation, or detection task results to verify the effectiveness of the lightweight U-Net model.

[0127] The resource monitoring hook function is used to check whether it is a specific hardware environment. If not, it returns directly to obtain the current hardware resource usage and convert it into the corresponding unit. If the resource usage exceeds the preset threshold, sparsity acceleration or structural optimization is activated. When the resource is exceeded, each layer of the model is traversed. If it is a convolutional layer, a random mask is generated and the convolution kernel weight is multiplied by the mask to make some weights zero, thereby achieving sparsification. At the same time, the network structure of the lightweight U-Net model is dynamically adjusted according to the characteristics of the input data.

[0128] In some embodiments, the model building module 20 is configured to quantize the convolution kernel weights of the convolution layer through hardware-aware initialization;

[0129] Generate a random tensor with the same shape as the convolution kernel weight of a preset lightweight U-Net model using the processed signal, wherein the value of the random tensor is in the range of [0, 1);

[0130] Generate a Boolean mask based on the random tensor according to the configured sparsity, and convert the Boolean mask to a floating-point type, where values ​​less than the sparsity are set to True, and values ​​other than the sparsity are set to False;

[0131] The generated mask is element-wise multiplied with the quantized convolutional layer’s kernel weights to construct a depth-wise separable convolutional layer.

[0132] In some embodiments, the model optimization module 30 is used to traverse each layer of the lightweight U-Net model, collect and count the feature statistics of each layer;

[0133] Calculate the mean of the feature statistics, calculate the dynamic sparsity based on the mean and preset parameters, limit the dynamic sparsity to the set range and return it;

[0134] Calculate the spatial domain importance score of the convolution kernel weight and the temporal domain importance score of the feature activity, and combine the spatial and temporal importance scores to generate a comprehensive score;

[0135] Sorting is performed according to the comprehensive score and the retained feature channels or convolution kernels are selected, and the network structure of the lightweight U-Net model is reconstructed using the retained feature channels or convolution kernels to complete the optimization.

[0136] In some embodiments, the model optimization module 30 is configured to obtain a target cache size, create a pruner based on the cache size, and use the pruner to traverse all layers of the model to perform structured pruning on the convolution kernel weights of each layer;

[0137] Obtaining a cache configuration of a target chip, and adjusting pruning, quantization, and sparse coding of the pruner according to the cache configuration; if the target chip is not of a specific type, returning to a default configuration;

[0138] Create a UNetPruner instance using the pruner, pass the UNetPruner instance into the optimized model, and use the method corresponding to the UNetPruner instance to prune the optimized model as a whole;

[0139] Apply dynamic quantization to the model and call the corresponding method for dynamic quantization to complete model compression.

[0140] In some embodiments, the image reconstruction module 40 is used to traverse each layer of the model, obtain the convolution kernel weights of the current layer, convert the weights from dense format to CSR format, and obtain sparse weights, indices and pointers, and save the converted data to the layer;

[0141] Configure the hardware for a specific accuracy and sampling rate, set channel parameters, configure quantized sampling parameters, enable precision clocking and set the oversampling factor and clock interval, switch double buffering to achieve zero-wait switching, stop and restart hardware sampling, process completed buffer data within the specified time, start related pulse emission, set the drive pulse frequency, and start hardware continuous acquisition;

[0142] Traverse each feature unit, use SIMD vectorization to process the sum of the product of the convolution kernel weight and the input, process the non-vectorized part of the remaining elements, and determine whether the activation conditions are met based on the summation result;

[0143] If it is satisfied, the information of using SIMD optimization is output; if it is not satisfied, the information of using general implementation is output;

[0144] Use inline assembly to load convolution kernel weights and pulse data into registers, perform SIMD instruction-optimized multiplication and addition operations, and update feature maps. This loop continues until all data is processed.

[0145] Initialize the accumulator acc to 0, traverse each row of the convolution kernel weight matrix, and for each row, traverse each column of the convolution kernel weight matrix, calculate the product of the low-order weight and the high-order pulse data and add them to acc, and perform a shift operation on the accumulated result and store it in the high-order output array;

[0146] The input is the compressed data pointer and output buffer pointer, as well as the maximum length. Every two bytes represent a value and the number of repetitions. They are decoded and filled into the output buffer in sequence until the maximum length is reached or the data ends.

[0147] The input is the quantized convolution kernel weight, pulse data, output pointer, scale factor and zero point. The convolution kernel weight matrix and pulse data are traversed by double loop, the weighted sum is calculated while considering the quantization zero point, and finally the output is dequantized.

[0148] Set up the model parser, error reporter, and memory pool, load model data, and allocate tensors to complete model deployment.

[0149] In some embodiments, the upsampling strategy is to use 8-bit quantized weights and activations in the initial convolution layer, 4-bit weight quantization in the depthwise separable convolution layer, 8-bit weights and 16-bit activation in the attention module, and construct a model distillation framework to migrate Swin Transformer knowledge.

[0150] An embodiment of the present application also provides a micro-ultrasound image reconstruction device based on deep learning, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus, and the memory is used to store a micro-ultrasound image reconstruction program based on deep learning; the processor is used to implement the above-mentioned micro-ultrasound image reconstruction method based on deep learning when executing the program stored in the memory.

[0151] The communication bus mentioned in the deep learning-based micro-ultrasound image reconstruction device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0152] The communication interface is used for communication between the above-mentioned deep learning-based micro ultrasound image reconstruction device and other devices.

[0153] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located remote from the processor.

[0154] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0155] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0156] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0157] Each embodiment in this specification is described in a related manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For related parts, refer to the description of the method embodiment.

[0158] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

[0159] It should be understood that the above is only an example and does not constitute any limitation to the technical solution of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any limitation on this.

[0160] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of it according to actual needs to achieve the purpose of the embodiment scheme, and no limitation is made here.

[0161] In addition, for technical details not fully described in this embodiment, please refer to the deep learning-based micro-ultrasound image reconstruction method provided in any embodiment of the present invention, and will not be repeated here.

[0162] In addition, it should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0163] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0164] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, or of course by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0165] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

[0166] It is understandable that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts of the above method.

Claims

1. A micro-ultrasound image reconstruction method based on deep learning, characterized in that: The micro-ultrasound image reconstruction method based on deep learning includes: Collecting raw ultrasonic radio frequency signals, preprocessing the raw ultrasonic radio frequency signals through adaptive beamforming technology embedded in a lightweight CNN, and outputting the processed signals; Constructing a lightweight U-Net model using the processed signal, wherein the encoder and decoder of the lightweight U-Net model use depthwise separable convolutional layers, the convolution kernel weights are quantized to 4 to 8 bits, and a channel-spatial dual attention mechanism is embedded in the skip connections of the lightweight U-Net model to form multi-scale attention gating; The lightweight U-Net model is optimized using a hybrid loss function that combines SSIM structural similarity loss, edge-sensitive loss, and perceptual loss. The optimized model is compressed through quantization perception to obtain a quantized model, wherein the quantized model is compressed to INT8 precision, and the attention module retains 16-bit activation; Deploy the quantization model on the FPGA, perform inference using a layered mapping strategy and a double buffering mechanism, and output a reconstructed image; The reconstructed image is optimized by dynamically selecting an upsampling strategy through a closed-loop feedback super-resolution module, wherein the detail resolution of the optimized reconstructed image is 0.8 mm.

2. The micro-ultrasound image reconstruction method based on deep learning according to claim 1, characterized in that: The method of constructing a lightweight U-Net model using the processed signal includes: Constructing a depthwise separable convolutional layer using the processed signal, and replacing the standard separable convolutional layer of the initial lightweight U-Net model with the depthwise separable convolutional layer to obtain a lightweight U-Net model; Perform global average pooling on the input sample feature map and calculate the attention weight through the fully connected layer and its activation function; Multiplying the calculated attention weight by the sample feature map channel by channel to enhance the expressive power of the lightweight U-Net model; Extracting multi-scale features of the sample feature map through convolution operations at different levels; Concatenate or weightedly fuse feature maps of different scales to form a fused feature map; At a specific network level, the semantic information and edge information in the fused feature map are enhanced through specific convolution operations and activation functions, and the processed feature map is multi-tasked to output classification, segmentation, or detection task results to verify the effectiveness of the lightweight U-Net model; The resource monitoring hook function is used to check whether it is a specific hardware environment. If not, it returns directly to obtain the current hardware resource usage and convert it into the corresponding unit. If the resource usage exceeds the preset threshold, sparsity acceleration or structural optimization is activated. When the resource is exceeded, each layer of the model is traversed. If it is a convolutional layer, a random mask is generated and the convolution kernel weight is multiplied by the mask to make some weights zero, thereby achieving sparsification. At the same time, the network structure of the lightweight U-Net model is dynamically adjusted according to the characteristics of the input data.

3. The micro-ultrasound image reconstruction method based on deep learning according to claim 2, characterized in that: The step of constructing a depth-wise separable convolutional layer using the processed signal comprises: Quantize the convolution kernel weights of the convolution layer through hardware-aware initialization; Generate a random tensor with the same shape as the convolution kernel weight of a preset lightweight U-Net model using the processed signal, wherein the value of the random tensor is in the range of [0, 1); Generate a Boolean mask based on the random tensor according to the configured sparsity, and convert the Boolean mask to a floating-point type, where values ​​less than the sparsity are set to True, and values ​​other than the sparsity are set to False; The generated mask is element-wise multiplied with the quantized convolutional layer’s kernel weights to construct a depth-wise separable convolutional layer.

4. The micro-ultrasound image reconstruction method based on deep learning according to claim 1, characterized in that: The use of a hybrid loss function to optimize the lightweight U-Net model includes: Traverse each layer of the lightweight U-Net model, collect and count the feature statistics of each layer; Calculate the mean of the feature statistics, calculate the dynamic sparsity based on the mean and preset parameters, limit the dynamic sparsity to the set range and return it; Calculate the spatial domain importance score of the convolution kernel weight and the temporal domain importance score of the feature activity, and combine the spatial and temporal importance scores to generate a comprehensive score; Sorting is performed according to the comprehensive score and the retained feature channels or convolution kernels are selected, and the network structure of the lightweight U-Net model is reconstructed using the retained feature channels or convolution kernels to complete the optimization.

5. The micro-ultrasound image reconstruction method based on deep learning according to claim 1, characterized in that: The optimized model is compressed by quantization perception, including: Get the target cache size, create a pruner based on the cache size, and use the pruner to traverse all layers of the model and perform structured pruning on the convolution kernel weights of each layer; Obtaining a cache configuration of a target chip, and adjusting pruning, quantization, and sparse coding of the pruner according to the cache configuration; if the target chip is not of a specific type, returning to a default configuration; Create a UNetPruner instance using the pruner, pass the UNetPruner instance into the optimized model, and use the method corresponding to the UNetPruner instance to prune the optimized model as a whole; Apply dynamic quantization to the model and call the corresponding method for dynamic quantization to complete model compression.

6. The micro-ultrasound image reconstruction method based on deep learning according to claim 1, characterized in that: The deploying the quantization model on the FPGA includes: Traverse each layer of the model, obtain the convolution kernel weights of the current layer, convert the weights from dense format to CSR format, and obtain sparse weights, indices and pointers, and save the converted data to the layer; Configure the hardware for a specific accuracy and sampling rate, set channel parameters, configure quantized sampling parameters, enable precision clocking and set the oversampling factor and clock interval, switch double buffering to achieve zero-wait switching, stop and restart hardware sampling, process completed buffer data within the specified time, start related pulse emission, set the drive pulse frequency, and start hardware continuous acquisition; Traverse each feature unit, use SIMD vectorization to process the sum of the product of the convolution kernel weight and the input, process the non-vectorized part of the remaining elements, and determine whether the activation conditions are met based on the summation result; If it is satisfied, the information of using SIMD optimization is output; if it is not satisfied, the information of using general implementation is output; Use inline assembly to load convolution kernel weights and pulse data into registers, perform SIMD instruction-optimized multiplication and addition operations, and update feature maps. This loop continues until all data is processed. Initialize the accumulator acc to 0, traverse each row of the convolution kernel weight matrix, and for each row, traverse each column of the convolution kernel weight matrix, calculate the product of the low-order weight and the high-order pulse data and add them to acc, and perform a shift operation on the accumulated result and store it in the high-order output array; Input the compressed data pointer and output buffer pointer, as well as the maximum length. Every two bytes represent a value and the number of repetitions. Decode and fill them into the output buffer in sequence until the maximum length is reached or the data ends. The input is the quantized convolution kernel weight, pulse data, output pointer, scale factor and zero point. The convolution kernel weight matrix and pulse data are traversed by double loop, the weighted sum is calculated while considering the quantization zero point, and finally the output is dequantized. Set up the model parser, error reporter, and memory pool, load model data, and allocate tensors to complete model deployment.

7. The micro-ultrasound image reconstruction method based on deep learning according to claim 1, characterized in that: The upsampling strategy is to use 8-bit quantized weights and activations in the initial convolutional layer, 4-bit weight quantization in the depthwise separable convolutional layer, 8-bit weights and 16-bit activations in the attention module, and build a model distillation framework to transfer Swin Transformer knowledge.

8. A micro-ultrasound image reconstruction device based on deep learning, characterized in that: The micro-ultrasound image reconstruction device based on deep learning includes: a signal processing module, configured to collect raw ultrasonic radio frequency signals, pre-process the raw ultrasonic radio frequency signals using adaptive beamforming technology embedded in a lightweight CNN, and output the processed signals; A model construction module is used to construct a lightweight U-Net model using the processed signal, wherein the encoder and decoder of the lightweight U-Net model use depthwise separable convolutional layers, the convolution kernel weights are quantized to 4 to 8 bits, and a channel-spatial dual attention mechanism is embedded in the skip connections of the lightweight U-Net model to form multi-scale attention gating; A model optimization module for optimizing the lightweight U-Net model using a hybrid loss function that combines SSIM structural similarity loss, edge-sensitive loss, and perceptual loss; The model optimization module is used to compress the optimized model through quantization perception to obtain a quantized model, wherein the quantized model is compressed to INT8 precision and the attention module retains 16-bit activation; An image reconstruction module, configured to deploy the quantization model on the FPGA, perform inference using a layered mapping strategy and a double buffering mechanism, and output a reconstructed image; An image optimization module is used to dynamically select an upsampling strategy to optimize the reconstructed image through a closed-loop feedback super-resolution module, wherein the detail resolution of the optimized reconstructed image is 0.8 mm.

9. The deep learning-based micro-ultrasound image reconstruction device according to claim 1, wherein: The model construction module is used to construct a depthwise separable convolutional layer using the processed signal, and replace the standard separable convolutional layer of the initial lightweight U-Net model with the depthwise separable convolutional layer to obtain a lightweight U-Net model; Perform global average pooling on the input sample feature map and calculate the attention weight through the fully connected layer and its activation function; Multiplying the calculated attention weight by the sample feature map channel by channel to enhance the expressive power of the lightweight U-Net model; Extracting multi-scale features of the sample feature map through convolution operations at different levels; Concatenate or weightedly fuse feature maps of different scales to form a fused feature map; At a specific network level, the semantic information and edge information in the fused feature map are enhanced through specific convolution operations and activation functions, and the processed feature map is multi-tasked to output classification, segmentation, or detection task results to verify the effectiveness of the lightweight U-Net model; The resource monitoring hook function is used to check whether it is a specific hardware environment. If not, it returns directly to obtain the current hardware resource usage and convert it into the corresponding unit. If the resource usage exceeds the preset threshold, sparsity acceleration or structural optimization is activated. When the resource is exceeded, each layer of the model is traversed. If it is a convolutional layer, a random mask is generated and the convolution kernel weight is multiplied by the mask to make some weights zero, thereby achieving sparsification. At the same time, the network structure of the lightweight U-Net model is dynamically adjusted according to the characteristics of the input data.

10. A micro-ultrasound image reconstruction device based on deep learning, characterized in that: The deep learning-based micro-ultrasound image reconstruction device includes: a memory, a processor, and a deep learning-based micro-ultrasound image reconstruction program stored on the memory and executable on the processor. The deep learning-based micro-ultrasound image reconstruction program is configured to implement the steps of the deep learning-based micro-ultrasound image reconstruction method as described in any one of claims 1 to 7.

Images