Image processing method, device and equipment based on hardware perception and dynamic sparsification

Abstract

The invention provides an image processing method, device and equipment based on hardware perception and dynamic rarefaction, and relates to the technical field of image processing, and the method comprises the steps: inputting hardware operation state data into a preset PPO strategy network, so as to obtain the sparse rate of each convolution layer; reserving a specified number of channels as effective calculation channels; performing saliency region analysis on an input to-be-processed image to generate a pixel-level saliency map corresponding to the to-be-processed image; dividing the to-be-processed image into a key target area and a background area; allocating a first preset precision to the key target area for image processing, and allocating a second preset precision to the background area for image processing; according to the sparse rate of each convolutional layer, the first preset precision allocated to the key target region and the second preset precision allocated to the background region, collaborative image processing is carried out on the to-be-processed image; according to the invention, the large-scale landing capability and continuous optimization performance of the edge intelligent vision system can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image processing method, device and equipment based on hardware perception and dynamic sparsification. BACKGROUND

[0002] In edge computing scenarios such as intelligent security and protection, vehicle-mounted perception, and industrial quality inspection, lightweight deployment of visual models is urgently needed to balance inference efficiency and task accuracy. However, current lightweight deployment techniques for visual models on edge devices (such as vehicle-mounted chips and security cameras) have significant drawbacks: existing solutions mostly use static pruning or uniform quantization strategies, which fix the model structure and precision allocation before deployment. These solutions cannot respond to real-time resource states such as CPU load fluctuations, memory bandwidth changes, and chip temperature rises during the inference process, leading to a significant increase in inference delay or even model failure in high-load or high-temperature environments, which severely affects system real-time performance and stability. Meanwhile, globally uniform quantization methods (such as converting the entire model to 4-bit or 8-bit width) ignore the differentiated precision needs of key target regions and background regions in images, which can easily cause a significant decrease in key target recognition accuracy in complex scenarios such as rain, fog, occlusion, and low light, with a precision loss often exceeding 30%. Furthermore, sparsification and quantization processes are mostly independent and lack a collaborative mechanism, which can amplify low-precision calculation errors in certain scenarios (such as a small-scale target detection precision loss exceeding 12%). In addition, most lightweight models (such as MobileNetV3) rely on offline training and manual hyperparameter adjustment, which makes it difficult to adapt to long-term changes such as hardware aging and running environment switching after deployment, lacking the ability to continuously optimize. These problems collectively restrict the large-scale deployment and performance improvement of edge intelligent visual systems. SUMMARY

[0003] To address the above technical problems, the technical solution adopted by the present application is as follows: According to a first aspect of the present application, a method for image processing based on hardware perception and dynamic sparsification is provided, comprising: obtaining hardware running state data of an edge device; the hardware running state data at least includes CPU utilization, memory bandwidth, and chip temperature; inputting the hardware running state data after anti-interference processing into a preset PPO policy network to output the sparsity of each convolutional layer of the image processing model in the edge device through the PPO policy network; determining the number of channels to be retained for each convolutional layer according to the sparsity of each convolutional layer, and sorting the channels of each convolutional layer according to their importance based on channel norms, and retaining a specified number of channels with high ranking as effective calculation channels; performing salient region analysis on the input image to be processed to generate a pixel-level saliency map corresponding to the image to be processed; According to the saliency map, the image to be processed is divided into a key target region and a background region; The key target region is assigned a first preset accuracy for image processing, and the background region is assigned a second preset accuracy for image processing; the first preset accuracy is greater than the second preset accuracy; According to the sparsity of each convolutional layer and the first preset accuracy assigned to the key target region and the second preset accuracy assigned to the background region, the image to be processed is cooperatively processed.

[0004] Further, after obtaining the hardware running parameters of the edge device, the method comprises: Obtaining the information entropy of the hardware state data sequence in the sliding window; In the case where the information entropy is lower than a preset information entropy threshold, it is determined that the corresponding sensor is invalid, and the hardware state data of the last sampling period is enabled; In combination with the ambient temperature collected by the ambient temperature sensor, the chip temperature is linearly compensated according to a preset compensation coefficient; the value of the preset compensation coefficient is such that the error of the compensated chip temperature is controlled within ±1℃.

[0005] Further, before inputting the anti-interference processed hardware running state data into the preset PPO policy network, the method comprises: Combining the current values of CPU utilization, memory bandwidth and chip temperature with the CPU utilization rate of change, memory bandwidth rate of change to form a multi-dimensional state vector for inputting into the PPO policy network; wherein the chip temperature is included in the multi-dimensional state vector in the form of the ratio of the chip temperature to the chip frequency reduction temperature threshold.

[0006] Further, the PPO policy network is a three-layer fully connected network structure, the three-layer fully connected network comprises 128, 64 and 32 neurons respectively, and ReLU is used as the activation function; the output layer of the PPO policy network adopts a grouped activation method, so that each output corresponds to the sparsity of a single convolutional layer of the image processing model, and the sparsity value range is limited to 0-70% through the activation function.

[0007] Further, it further comprises: Periodically collecting execution trajectory data of the PPO policy network under the current hardware state; the execution trajectory data comprises a hardware state vector, an output sparsity and a corresponding reward value, and the reward value comprehensively considers image processing delay, recognition accuracy, sparsity fluctuation degree and chip temperature; The generalized advantage estimation algorithm is used to determine the relative value of each sparsity decision; Based on a preset number of policy update rounds and a policy change threshold, a joint loss function comprising a policy loss, a value function error and a policy entropy is constructed; The parameters of the PPO policy network are updated through gradient clipping and back propagation.

[0008] Further, the image processing of the background region with the second preset precision comprises: The convolution kernel weights of the image processing model are aggregated into preset number of cluster centers by using a K-Means clustering algorithm. When the background region is processed, the cluster centers are called by a lookup table to replace the original convolution kernel weights to perform convolution calculation, so as to reduce the algorithmic power consumption of the background region processing.

[0009] Further, the cooperative image processing of the to-be-processed image comprises: According to the sparsity of each convolution layer, the corresponding convolution layer of the image processing model is subjected to effective calculation channel screening. According to the precision allocation result, the screened convolution layer is subjected to differential precision calculation, and the image processing process is monitored in real time. In the case that the single-layer processing time exceeds a preset threshold or the chip temperature exceeds a warning value, a preset hardware protection mechanism is triggered to ensure stable execution of the image processing.

[0010] Further, the preset hardware protection mechanism comprises at least one of the following: The maximum sparsity is enabled to perform channel pruning on each convolution layer, skip non-key feature extraction layers in the image processing model, switch to a unified first preset precision mode and close high-frequency calculation cores, cache the current image processing model state to perform breakpoint continuation after hardware recovery.

[0011] According to another aspect of the present application, an image processing device based on hardware perception and dynamic sparsification is also provided, and the device comprises: A hardware state perception module is configured to acquire hardware running state data of an edge device; the hardware running state data at least comprises CPU utilization, memory bandwidth and chip temperature. A PPO dynamic decision module is configured to input the anti-interference processed hardware running state data into a preset PPO policy network, so as to output the sparsity of each convolution layer of the image processing model in the edge device through the PPO policy network; according to the sparsity of each convolution layer, the number of channels to be reserved for each convolution layer is determined, and the channels of each convolution layer are sorted according to importance according to the channel norm, and the specified number of channels at the front of the sorting are reserved as effective calculation channels. The input guiding mixed precision module is used for significant region analysis on an input image to be processed to generate a pixel-level saliency map corresponding to the image to be processed; the image to be processed is divided into a key target region and a background region according to the saliency map; the key target region is assigned a first preset precision for image processing, and the background region is assigned a second preset precision for image processing; the first preset precision is greater than the second preset precision. The joint scheduling execution module is used for cooperative image processing of the image to be processed according to the sparsity of each convolution layer and the first preset precision assigned to the key target region and the second preset precision assigned to the background region.

[0012] According to another aspect of the present application, an electronic device is also provided, which includes a processor and the above-mentioned image processing device based on hardware perception and dynamic sparsification.

[0013] The present application has at least the following beneficial effects: The image processing method based on hardware perception and dynamic sparsification of the present application effectively solves the problem that the existing static pruning / uniform quantization scheme cannot adapt to real-time resource fluctuations of hardware, avoids a sharp increase in inference delay and model failure in a high-load or high-temperature environment, and guarantees the real-time performance and stability of the system; by performing significant region analysis on the image and dividing the key target region and the background region, higher first preset precision and lower second preset precision are respectively assigned for processing, thereby breaking through the limitation of global uniform quantization that ignores the differentiated accuracy requirements of different regions, and significantly reducing the accuracy loss of key target recognition in complex scenes such as rain, fog, occlusion, and low light; at the same time, the cooperative image processing is performed in combination with the sparsity of each convolution layer and the differentiated accuracy assignment, thereby realizing the linkage regulation of sparsification and quantization, avoiding the problem that the sparse pruning amplifies the low-precision calculation error when processed independently, and the dynamic sparsification strategy can be adaptively adjusted by the PPO policy network without relying on offline training and manual hyperparameter adjustment, which can better adapt to long-term changes such as hardware aging and environment switching after deployment, and further improves the large-scale landing ability and continuous optimization performance of the edge intelligent vision system. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0015] Figure 1A flowchart of the image processing method based on hardware perception and dynamic sparsification provided for the embodiment of the present application is shown in FIG. 1. Figure 2 A core architecture schematic diagram provided for the embodiment of the present application is shown in FIG. 2. Figure 3 A hardware running state perception schematic diagram provided for the embodiment of the present application is shown in FIG. 3. Figure 4 A PPO policy network architecture schematic diagram provided for the embodiment of the present application is shown in FIG. 4. Figure 5 A lightweight visual Transformer structure schematic diagram provided for the embodiment of the present application is shown in FIG. 5. Figure 6 A mixed precision execution logic schematic diagram provided for the embodiment of the present application is shown in FIG. 6. Figure 7 A training process schematic diagram provided for the embodiment of the present application is shown in FIG. 7. Figure 8 An image processing device schematic diagram based on hardware perception and dynamic sparsification provided for the embodiment of the present application is shown in FIG. 8. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0017] It should be noted that, based on the present disclosure, those skilled in the art should understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, an apparatus and / or a method can be implemented using any number of the aspects set forth herein. In addition, this apparatus and / or method can be implemented using other structures and / or functionality in addition to or other than one or more of the aspects set forth herein.

[0018] The traditional method of lightweight deployment of current visual models on edge devices usually adopts static pruning or uniform quantization strategy, which fixes the model structure and precision allocation before deployment, lacking dynamic adaptation capability at runtime. For example, pruning schemes based on channel pruning or preset sparsity can reduce computational load to some extent, but cannot respond to real-time resource fluctuations during inference, leading to a sharp decline in performance in high-load or energy-constrained scenarios. At the same time, global uniform quantization strategies, such as converting the entire model to 4-bit or 8-bit width, also ignore the sensitivity of different regions in the image to precision, which can easily cause loss of key target detection accuracy. In addition, the current post-training quantization mechanism often relies on static data calibration and lacks robustness to input distribution changes, with serious error accumulation when switching precision. In the face of device aging, temperature fluctuations or external environmental changes, existing lightweight model architectures such as MobileNetV3 need to be retrained, lacking long-term adaptability. Sparse and quantization processing are still executed as independent processes in most systems, lacking cross-module linkage scheduling mechanisms, making it difficult to achieve optimal performance at the overall level.

[0019] Disadvantages of the prior art: 1. Current static compression strategies cannot adapt to runtime hardware resource fluctuations, such as a 200% increase in inference delay when CPU is overloaded, which can easily lead to model failure in high-load or hot environments, severely affecting the real-time performance and stability of the system; 2. Uniform quantization strategies treat key regions and non-key regions in images with the same precision, lacking precision differentiation mechanisms, which can lead to a significant decline in recognition accuracy in complex scenarios such as rain, fog, occlusion, and low light, with some task precision loss greater than 30%; 3. The separate sparse and quantization processes lack coordination mechanisms, and sparse pruning can amplify low-precision calculation errors in certain scenarios, such as a 12% loss in detection accuracy for small-scale targets in images; 4. Most lightweight models rely on a large number of manual adjustments and hyperparameter searches during design, even with NAS technology, they still require offline training and deployment, lacking dynamic adaptation capability for long-term changes such as hardware aging and runtime environment switching, limiting the potential for continuous optimization in actual industrial scenarios.

[0020] Based on the above background, this embodiment proposes an image processing method, apparatus, and device based on hardware-aware dynamic sparsity. Through a hardware-aware dynamic sparsity control (HASC) mechanism, the PPO algorithm is used to analyze the device's operating status (CPU load / memory bandwidth / chip temperature) in real time, dynamically generating sparsity decision trees (0-70%) for each layer of the network, achieving load-sensitive adaptive computational acceleration. Simultaneously, an input-guided mixed precision (IGMP) module is constructed, using lightweight ViT to parse salient regions of the image, retaining 8-bit precision for key targets while using 4-bit cluster center approximation for the background. Gradient alignment loss is used to constrain precision switching errors, ensuring the core accuracy of the visual task. After deployment, the system supports online updates of policy network parameters, continuously adapting to device aging and environmental changes. This solution breaks through the traditional fixed compression mode, achieving a dynamic balance between computational efficiency and accuracy in edge scenarios.

[0021] The solution in this embodiment can solve the following problems: 1. The system uses the PPO reinforcement learning algorithm to analyze the operating indicators of edge devices in real time, such as CPU load, memory bandwidth and chip temperature, and dynamically generates sparsity rate adjustment strategies (range 0-70%) for each layer of the network. Under high load, the sparsity ratio is automatically increased to ensure that the inference latency is controlled, while under low load, accuracy is prioritized.

[0022] 2. In terms of precision allocation, the image saliency heatmap is extracted based on the lightweight visual Transformer (ViT) module. Key target regions in the image (such as faces and vehicles) are processed with 8-bit high precision, while the background region is processed with 4-bit cluster center quantization. Combined with the gradient alignment loss function, the error propagation caused by precision switching is suppressed, which effectively improves the recognition accuracy in complex scenes (such as severe weather).

[0023] 3. This system supports on-device online optimization and can continuously update the PPO policy network after deployment. It can autonomously adjust the sparsity policy through the reward function to achieve rapid response to changes in hardware status. 4. By using a joint decision-making execution and dynamic switching mechanism of sparsification and mixed precision quantization, the utilization of computing resources is maximized while ensuring recognition accuracy, thereby achieving a dynamic balance between computing efficiency and inference accuracy.

[0024] The following will refer to Figure 1 The flowchart shown introduces an image processing method based on hardware awareness and dynamic sparsity.

[0025] From the product side, by constructing the "hardware-aware dynamic sparse control (HASC)" and "input-guided mixed precision (IGMP)" two core mechanisms, the model can realize real-time sensing of device state and image input characteristics during runtime, and automatically adjust the network structure and calculation precision. In the HASC module, the system periodically monitors the CPU utilization, memory bandwidth and chip temperature, and automatically decides the sparsity of each convolution layer based on the strategy network, realizing load-sensitive inference calculation. In the IGMP module, the saliency analyzer assigns 8-bit high-precision calculation to the key areas of the image (such as the target edge), and applies 4-bit low-bitwidth compression to the background area, realizing precision-energy balance.

[0026] From the technical side, as shown in Figure 2 Three key modules are constructed: first, from the bottom sensing level, the "hardware-aware dynamic sparse control (HASC)" module is proposed to realize the strong coupling control mechanism of sparsity and device state; through the "input-guided mixed precision (IGMP)" module, the saliency map is extracted using a lightweight visual Transformer to implement differential bit width control in different areas; finally, in the "joint scheduling engine", the fusion execution path is constructed to integrate the above two types of strategies at the operator level, realizing sustainable, efficient and stable visual inference execution flow on the edge side.

[0027] The image processing method based on hardware sensing and dynamic sparsification can include the following steps: S100, obtaining hardware running state data of an edge device; the hardware running state data at least includes CPU utilization, memory bandwidth and chip temperature.

[0028] In this embodiment, a "hardware-aware dynamic sparse control" mechanism (Hardware-Aware Sparse Control, HASC) for edge devices is proposed, which can realize real-time sensing of the running state of the device during model inference, and dynamically adjust the sparsity of each network layer based on this, thereby achieving the goal of delay control and power saving. This mechanism revolves around three key resource dimensions, including processor load (CPU utilization), memory bandwidth and chip junction temperature. By constructing a unified sampling → fusion → decision pipeline, the system can realize millisecond-level resource sensing capability.

[0029] Data acquisition implementation: use a sampling period matched with the video frame rate corresponding to the image to be processed to acquire hardware running state data; for example: use a 33ms sampling period matched with a 30FPS video frame rate to ensure that the hardware state update is synchronized with the image inference period, avoiding frame-to-frame state drift.

[0030] In this embodiment, as Figure 3As shown, a set of multi-source heterogeneous sensor data collection → real-time filtering → feature fusion hardware state perception pipeline is designed, and the main implementation process includes the following three parts: (1) Multi-source data collection method: CPU utilization: Read each core time counter through the Linux kernel interface, calculate the "running time" and "idle time" of all cores in the adjacent sampling period, and calculate the global CPU utilization U cpu : ; Among them, ΔT user is the time increment of the CPU core for executing "user state task" in the adjacent sampling period (i.e. the time change consumed by the user-initiated application-related computing task); ΔT nice is the time increment of the CPU core for executing "low-priority user state task" in the adjacent sampling period (i.e. the time change consumed by the user state task with lower priority adjusted by the nice command); ΔT system is the time increment of the CPU core for executing "kernel state task" in the adjacent sampling period (i.e. the time change consumed by the tasks at the operating system kernel level, such as hardware drivers, memory management, process scheduling, etc.); ΔT idle is the time increment of the CPU core in the "idle state" in the adjacent sampling period (i.e. the time change of the CPU core in the waiting state without any executable tasks); N core is the number of CPU cores.

[0031] (2) Accurate measurement of memory bandwidth: Call perf_event_open system call, register PERF_COUNT_HW_CACHE_DTLB and PERF_COUNT_HW_CACHE_LLC performance events, and cooperate with memory controller PMU (Performance Monitoring Unit) to obtain read and write throughput, and the bandwidth calculation formula is as follows: ; Among them, C rd is the read cache line count, C wr is the write cache line count; S line is the cache line size, such as: S line =64B (DDR4 cache line size), T sample is the sampling period, such as: T sample =33ms.

[0032] (3) Chip temperature anti-interference collection The system collects the chip temperature by accessing / sys / class / thermal / thermal_zone0 / temp, and adopts a sliding window median filtering algorithm to suppress the influence of high-frequency temperature fluctuations and system abnormal jumps.

[0033] ; wherein T chip （filtered） is the effective value of the chip temperature obtained after median filtering (used for subsequent hardware state analysis and decision-making); Median() is a median operation function that takes the median of a group of input data to suppress outliers and high-frequency noise in the temperature data; T chip （k） is the original chip temperature data collected at the kth sampling time (k is the sampling time index); n is the index of the current sampling time; t is the length parameter of the sliding window, i.e. the window contains temperature data of t+1 sampling times from "n-t" to "n".

[0034] Further, after obtaining the hardware running parameters of the edge device, the method comprises: obtaining the information entropy of the hardware state data sequence in the sliding window.

[0035] In the case where the information entropy is lower than a preset information entropy threshold, it is determined that the corresponding sensor is invalid, and the hardware state data of the previous sampling period is enabled.

[0036] In combination with the ambient temperature collected by the ambient temperature sensor, the chip temperature is linearly compensated according to a preset compensation coefficient; the value of the preset compensation coefficient is such that the error of the compensated chip temperature is controlled within ±1℃.

[0037] Real-time filtering and anti-interference design: Considering the complex deployment environment of the edge device, the collected data may have sudden changes, misreading or sensor failure, etc., the present application constructs the following two key anti-interference mechanisms: (1) Sliding window entropy filtering: The system calculates the information entropy of the hardware running state data sequence in the sampling window: ; wherein E t is the information entropy of the hardware state data at time t, which is used to measure the uncertainty and fluctuation degree of the hardware state feature vector. The lower the entropy value, the more the data tends to be single and the smaller the fluctuation, and the more likely it is that there is a sensor anomaly.

[0038] p i is the normalized proportion of the i-th dimension of the hardware state fusion feature vector, which represents the "contribution weight" of this dimension in the overall feature; H t，iThe original value of the i-th dimension of the hardware state fusion feature vector at time t (for example, CPU utilization, memory bandwidth change rate, etc., a total of 5 dimensions, corresponding to the upper limit of the sum in the formula is 5); The sum of the absolute values of all 5 dimensions of the hardware state fusion feature vector at time t is used as the denominator of the normalization, so that the proportion of each dimension meets .

[0039] By calculating the information entropy through the formula, it can be judged whether the hardware state data has lost the fluctuation characteristic due to sensor abnormality, and a basis is provided for subsequent "anti-interference processing (such as replacing abnormal data)".

[0040] If E t <E min (0.7), it means that the state is single and lacks fluctuation, and it is extremely likely that the sensor is stuck or the data is abnormal, so the sensor is determined to be invalid, and the last period data is enabled to prevent abnormal values from being propagated to the control logic.

[0041] (2) Temperature drift compensation: Considering that the junction temperature is highly dependent on the ambient temperature, in order to eliminate the influence of external environment, the present application introduces a system to introduce the temperature measured by the environmental sensor, and constructs the following linear compensation relationship: ; Among them, T compensate is the compensated chip temperature effective value, which is a temperature value that can accurately reflect the actual heating state of the chip after eliminating the influence of environmental temperature drift, and is used for subsequent hardware state perception and strategy decision; is the original chip temperature after anti-interference filtering processing (for example, through sliding window median filtering, etc., to suppress the high-frequency noise and abnormal jump of the original temperature data); α is a preset temperature compensation coefficient, which is obtained through a large number of environmental experiments, so that the error of the compensated chip temperature is controlled within ±1℃, reflecting the influence of environmental temperature on chip temperature measurement; T amb is the real-time environmental temperature collected by the environmental temperature sensor, that is, the external temperature of the environment where the chip is located (such as the air temperature around the edge device); 25℃ is the reference environmental temperature, which is used as the benchmark value of the environmental temperature, and when the actual environmental temperature T amb deviates from this value, the chip temperature is adjusted through the compensation coefficient α to eliminate the measurement error caused by environmental temperature drift; α=0.15.

[0042] Through the compensation formula, the chip temperature measurement can be more accurate, the hardware load misjudgment caused by environmental heat interference can be avoided, and the reliability of subsequent sparse rate decision based on hardware state is improved; The problems of "hardware state data containing abnormal values" and "chip temperature misjudgment caused by environmental interference" in the prior art are solved; Sensor failure detection avoids outputting incorrect sparse rate of PPO strategy due to data abnormalities (such as misjudging CPU low load and reducing sparseness, resulting in high load delay); Temperature drift compensation ensures accurate chip temperature reading, avoids misjudgment of hardware overload caused by high temperature environment, improves the reliability of sparse rate decision, and ensures long-term stable operation of the system.

[0043] Further, before the hardware running state data after the anti-interference processing is input into the preset PPO strategy network, the following steps are included: The current values of CPU utilization rate, memory bandwidth and chip temperature are combined with the CPU utilization rate change rate and the memory bandwidth change rate to form a multi-dimensional state vector for inputting the PPO strategy network; wherein the chip temperature is included in the multi-dimensional state vector in the form of the ratio of the chip temperature to the chip frequency reduction temperature threshold.

[0044] The spatio-temporal feature fusion method comprises the following steps: In order to capture the current value and change trend of the device running state, three basic resource indicators and their short-term change trends are combined into a five-dimensional state vector in the present embodiment : ; Among them, U CPU (t) is the CPU utilization rate at time t, reflecting the load degree of the current CPU computing task; B mem （t） is the memory bandwidth at time t, indicating the real-time rate of memory data transmission; ε is a minimum value, avoiding the undefined operation of logarithm when the memory bandwidth is 0, and at the same time, the memory bandwidth data is smoothed and denoised, and the feature distribution is optimized; T chip （t） is the chip temperature at time t; T th is the frequency reduction temperature threshold of the chip, which is the critical temperature value of the chip triggering frequency reduction protection (ensuring hardware safety), and is used to convert the chip temperature into "relative risk degree"; U CPU (t-1) is the CPU utilization rate at time t-1; B mem （t-1） is the memory bandwidth at time t-1, that is, the memory data transmission rate at the previous sampling time, which is used to calculate the change rate; Δt is the sampling period (such as 33ms, matching the 30FPS video frame rate), which is the time interval between adjacent two hardware state samplings, and is used to calculate the "change rate" feature.

[0045] Through the vector, the PPO policy network can simultaneously perceive the current load state (CPU utilization, memory bandwidth, chip temperature relative risk) and dynamic change trend (CPU utilization rate of change, memory bandwidth rate of change) of the hardware, so as to more accurately adapt to the real-time resource fluctuation of the hardware and output reasonable convolution layer sparsity.

[0046] Through the above steps, the problems of "no anti-interference mechanism, low data precision, and asynchronous sampling" in the prior art are solved, high confidence input is provided for subsequent sparsity decision; the 33ms sampling period matches the video frame rate, avoiding state delay affecting inference real-time performance; temperature compensation and entropy filtering ensure reliable hardware data, avoiding policy decision deviation caused by sensor failure or environmental interference, laying a foundation for dynamic adaptation of hardware.

[0047] S200, inputting the hardware running state data after the anti-interference processing to a preset PPO policy network, to output the sparsity of each convolution layer of the image processing model in the edge device through the PPO policy network.

[0048] In this embodiment, a PPO dynamic decision mechanism is used, that is, after completing the multi-dimensional perception and expression of the device running state, the key problem that the system needs to solve is: how to map these complex hardware features to the sparsity decision of each layer in the network structure, to realize the dynamic balance of "neither over-compression affecting accuracy, nor redundant calculation wasting power consumption". Therefore, this embodiment introduces the idea of reinforcement learning, and designs a dynamic sparsity control strategy based on Proximal Policy Optimization (PPO).

[0049] Further, the PPO policy network is a three-layer fully connected network structure, the three-layer fully connected network respectively contains 128, 64 and 32 neurons, and ReLU is used as the activation function; the output layer of the PPO policy network adopts a grouped activation method, so that each output corresponds to the sparsity of a single convolution layer of the image processing model, and the sparsity is limited to the range of 0-70% through the activation function.

[0050] As shown in Figure 4 The policy network architecture includes a three-layer perception mapping policy network, which takes the real-time collected hardware state vector as input, extracts high-order features through the three-layer fully connected network (with 128, 64 and 32 neurons respectively, and the activation function is ReLU) in turn, and converts to the sparsity of each convolution layer, so as to dynamically adjust the inference structure to adapt to the system resource state.

[0051] The output layer adopts a grouped Sigmoid activation function, and each output α iThe channel sparsity corresponding to the i-th convolutional layer is limited in the range [0, 0.7]. The output adopts the following grouping Sigmoid activation form: ; Wherein, σ() is a Sigmoid activation function, which nonlinearly maps the input value to the interval (0, 1), realizing range constraint and nonlinear transformation; W o （i） is the weight matrix corresponding to the i-th convolutional layer sparsity prediction in the PPO policy network output layer, which is used for linear transformation of the features of the previous layer; h 32 is the output vector of the third fully connected layer (containing 32 neurons) of the PPO policy network, which is the feature representation processed by the previous two layers (128 neurons and 64 neurons); b o （i） is the bias term corresponding to the i-th convolutional layer sparsity prediction in the PPO policy network output layer, which is used to adjust the linear transformation result and enhance the model expression ability.

[0052] Through this structure, the system can complete the nonlinear mapping from the device state to the sparse policy within milliseconds, realize the micro-granularity and dynamic channel pruning control, and avoid the precision sacrifice caused by the global pruning strategy.

[0053] S300, according to the sparsity of each convolutional layer, determine the number of channels to be retained for each convolutional layer, and according to the channel norm, sort the importance of the channels of each convolutional layer, and retain the specified number of channels at the front of the sorting as effective calculation channels.

[0054] Sparse decision execution flow: After generating the sparsity of each layer, the system uses it to dynamically prune the convolutional channels of each layer, and the specific process includes the following three steps: (1) Channel number calculation: according to the current sparsity α i , calculate the number of channels to be retained for each layer: ; Wherein, is the number of effective channels to be retained after sparsification of the i-th convolutional layer in the image processing model (the result is an integer rounded down to ensure that the number of channels is a reasonable integer value); is the original total number of channels of the i-th convolutional layer in the image processing model (the total number of channels of this layer before sparsification).

[0055] The formula obtains the number of effective channels to be retained after dynamic sparsification of the convolutional layer by "original channel number x retention ratio (1-sparsity)" and rounding down, realizing the channel pruning regulation and control of the convolutional layer.

[0056] (2) Top-K channel selection: using channel norm as an importance indicator, select Top-K channels for reservation: ; wherein, Idx keep is the effective channel index set to be reserved after Top-K screening (i.e., which channels will be reserved to participate in subsequent calculations); TopK is a Top-K selection operation that selects the "top K elements with the largest values" from the input elements and returns their indices; here, K is determined by the number of effective channels to be reserved in the convolution layer (i.e., the number of effective channels calculated by the sparsity rate ); represents the range of channel indices traversed in the convolution layer, and represents the norm calculation and screening for each of the 1st channel to the channel; is the original total number of channels of the convolution layer (total number of channels before sparsification); W :,:,c is the convolution kernel sub-tensor corresponding to the cth channel in the convolution layer (the sub-tensor of the cth channel after slicing the convolution kernel tensor by the channel dimension); || || F is the Frobenius norm, which is used to quantify the "energy / importance" of W :,:,c The greater the norm, the higher the contribution of the channel's convolution kernel to feature extraction, and the more valuable it is to be reserved.

[0057] (3) Only the selected channels are executed in the actual inference process, saving resource overhead, and the code example is as follows: pruned_weight=weight[Idx_keep, :, :, :] #select active channel weight output=conv2d(input,pruned_weight) #perform sparse convolution.

[0058] Through this step, the problems of "global pruning sacrificing accuracy" and "independent sparsification amplifying low-precision error" in the prior art can be solved; by screening effective channels through channel norm, the largest contribution channels are ensured to be reserved, saving 50%-70% of computing power while avoiding loss of key features (such as a decrease in small-scale target detection accuracy loss from traditional 12% to less than 5%); sparse convolution only calculates effective channels, further reducing CPU and memory load, and adapting to the resource-limited characteristics of edge devices.

[0059] S400, performing salient region analysis on the input image to be processed to generate a pixel-level saliency map corresponding to the image to be processed.

[0060] In this embodiment, in addition to the execution load difference caused by the device state, the content complexity of the input image itself also directly affects the inference resource consumption. Taking face recognition or defect detection tasks as an example, a large number of background regions contribute little to the final discrimination, but consume the same precision operation resources. To solve this problem, this embodiment proposes an “input-guided mixed precision (IGMP)” module, which uses a lightweight ViT to extract image saliency regions and applies a differential bit width strategy to different pixel blocks: the key area maintains 8-bit precision, and the background area is reduced to 4-bit cluster center approximation, thereby further compressing the inference resource consumption while ensuring accuracy.

[0061] Lightweight ViT saliency resolver: To efficiently identify the “key area” in the image, the present application proposes a lightweight visual Transformer structure, the architecture of which is as shown in Figure 5

[0062] Compared with the traditional multi-layer deep Transformer, only 4 Transformer blocks are used in this embodiment, and a space-channel decoupling attention mechanism is designed to reduce the computational complexity while retaining key perception capabilities. The attention mechanism is defined as follows: Spatial attention: ; wherein A s is the spatial attention weight matrix, and the matrix element represents the attention correlation probability between different spatial positions (such as pixel blocks, feature positions) of the image, which is used to strengthen the key target area features and suppress background interference; Q s is the query (Query) matrix, which is obtained by linearly transforming the input image features, and is used to calculate the similarity between the “key matrix” and the space position; K s is the key (Key) matrix, which is obtained by linearly transforming the input image features, and is multiplied by Q s to generate the original similarity score matrix; K s T is the transpose of the key matrix, which is multiplied by Q s to obtain the similarity matrix of different spatial positions; d k is the feature dimension of the key / query matrix; divided by d k 1 / 2 is to scale the similarity score to avoid the distribution being too steep after Softmax or gradient vanishing due to too large dimension, so that the attention weight distribution is more stable.

[0063] Channel attention: ; wherein || is channel concatenation, A c ​is the attention weight vector for the channel, and the output result is the "importance weight" of each feature channel (value range (0, 1), used to measure the contribution degree of the channel to the image key target (such as pedestrians, vehicles) features; MLP () is a perception layer of the MLP; AvgPool (Q) is an average pooling operation on the query matrix Q; Q is the feature matrix corresponding to the "query" in the visual Transformer (obtained by linear projection of the image block), and the average pooling is used to compress the spatial dimension, extract the global average features of the channel, and capture the overall information of the channel; MaxPool (K) is a maximum pooling operation on the key matrix K; K is the feature matrix corresponding to the "key" in the visual Transformer, and the maximum pooling is used to extract the global maximum features of the channel and capture the most discriminative local key information (such as target edges, textures, etc.) in the channel.

[0064] The channel attention weight generated by the formula can assist the subsequent saliency region analysis to more accurately distinguish "key target channels" and "background channels", and provide channel-level importance basis for the differential processing of "the first preset precision (such as 8-bit) of the key region and the second preset precision (such as 4-bit) of the background region".

[0065] Output: ; wherein, is the element-level point-by-point multiplication, Attn (Q, K, V) is the output feature of the joint attention mechanism, which is the image feature representation after being weighted by "spatial attention + channel attention" twice, and is used for subsequent generation of pixel-level saliency map to guide the division of key target region and background region; Q, K and V are query matrix, key matrix and value matrix respectively; the channel attention is weighted on the value matrix, that is, the feature value V of each channel is scaled according to the channel attention weight A c to highlight the features of the key channel and weaken the features of the background channel.

[0066] This structure introduces spatial-channel two-level attention paths, respectively calculates the attention weights of image positions and semantic dimensions, and finally outputs by Hadamard multiplication, so that the saliency result is more discriminative.

[0067] S500, according to the saliency map, the image to be processed is divided into a key target region and a background region.

[0068] Mixed precision execution engine: Based on the above saliency map, a "dynamic bit width allocation engine" is introduced in this embodiment, which allocates different quantization precisions for different saliency regions. The execution logic is as shown in Figure 6 . After the saliency map is output, the attention value M(x, y) of each pixel point represents the degree of contribution to the task; x and y are the X-axis coordinate and Y-axis coordinate of the pixel, respectively. According to the empirical threshold τ = 0.8 (optimized by the ROC curve), the following dynamic bit width strategy is implemented: ; BitWidth(x, y) is the quantization bit width (bit width determines data representation precision, 8-bit precision is higher, and 4-bit precision is lower but more power saving) adopted by the pixel point (or feature point) with coordinates (x, y) in the image.

[0069] The above operation is completed by the quantization control engine and will be executed in the joint scheduling stage in combination with the sparse strategy. This method ensures that the salient targets such as faces and vehicles are processed with high precision, while the sky and background areas are processed with lower 4-bit approximation, thereby significantly reducing the power requirement.

[0070] S600, a first preset precision is assigned to the key target area for image processing, and a second preset precision is assigned to the background area for image processing; the first preset precision is greater than the second preset precision.

[0071] Further, the image processing of the background area with the second preset precision comprises: The K-Means clustering algorithm is used to aggregate the convolution kernel weights of the image processing model into a preset number of cluster centers.

[0072] When the background area is processed, the cluster centers are called by table lookup to replace the original convolution kernel weights to perform convolution calculation, so as to reduce the power consumption of the background area processing.

[0073] Offline clustering stage: all the convolution kernel weights of the image processing model are expanded into a one-dimensional vector set, and the K-Means clustering algorithm is used to aggregate them into 16 cluster centers (4-bit corresponds to 2 4 = 16 discrete values, the number of cluster centers matches the bit width), and the clustering target is to minimize the sum of squares of the Euclidean distances between all weights in the cluster and the cluster center, so as to ensure that the cluster center can represent the distribution characteristics of the original weight.

[0074] Cluster center storage: the 16 cluster centers are stored as a "cluster center table" and loaded into the memory of the edge device during deployment, so as to facilitate fast table lookup.

[0075] In this embodiment, for the non-key area, in order to avoid the significant drop in precision caused by 4-bit quantization, the system uses a weight clustering strategy to aggregate the high-dimensional convolution kernel parameters into 16 representative cluster centers c1, …, c 16 , and then perform forward propagation through table lookup index replacement, as follows: Expanded convolution weights ; It is the real number field, W is the weight tensor of the convolutional layer, which stores the parameters of the convolution kernel and is used to perform convolution operations on the input features; C out C represents the number of output channels of the convolutional layer, i.e., the number of channels in the feature map generated by that convolutional layer; in K represents the number of input channels for the convolutional layer, i.e., the number of channels in the feature map of the previous layer (or the number of channels in the input data, such as 3 for an RGB image); h The height of the convolution kernel (the vertical dimension, such as K for a 3×3 convolution kernel). h =3;K w The width of the convolution kernel (the horizontal dimension, such as K for a 3×3 convolution kernel). w =3; The set of one-dimensional vectors obtained by expanding the convolution weight tensor W (flattening the multi-dimensional convolution kernel parameters, so that each convolution kernel parameter is transformed into a vector, which is convenient for subsequent clustering processing).

[0076] Minimize intra-cluster variance: ; where, {c k Let} be the set of all cluster centers, c k This represents the center vector of the k-th cluster; Let vector w be the cluster center c. k The square of the Euclidean distance measures the "dissimilarity" of a single weight vector to the cluster center; The optimization objective is to find the optimal set of cluster centers such that the sum of the squared distances from all weight vectors to their respective cluster centers is minimized (i.e., "minimize intra-cluster variance"). k For each weight vector w, find the nearest cluster center c. k (This determines the cluster to which w belongs).

[0077] Online inference implementation: Weight replacement: When performing convolution calculations on the background region, the original convolution kernel weights are not loaded. Instead, the Euclidean distance between the original weights and each cluster center is calculated, the index of the nearest cluster center is selected, and the corresponding cluster center is called from the cluster center table to replace the original weights.

[0078] Convolution calculation: Perform convolution with the input features of the background region using the cluster center weights. Code example is shown below: def quant_conv(x, weight_clusters): idx = argmin(|weight - clusters|, axis=1) # Get the index of the nearest cluster w_quant = clusters[idx] # Reconstruct quantization weights return conv2d(x, w_quant).

[0079] This method transforms the multiplication in convolution operations into a table lookup operation with almost no loss of precision, thereby improving computational efficiency.

[0080] Gradient alignment training method: Since the mixed-precision strategy introduces additional quantization error, we design the following joint loss function to suppress training-deployment bias: ; Among them, L total The overall loss function is a weighted sum of the main task loss, accuracy alignment loss, and regularization term, serving as the optimization objective for model training (which needs to be minimized to simultaneously meet multiple requirements); λ is the weight coefficient of the accuracy alignment loss, controlling the importance of "accuracy alignment" in the overall optimization (the larger the coefficient, the more emphasis is placed on accuracy alignment with the original model); μ is the weight coefficient of the L2 regularization term, controlling the regularization strength (the larger the coefficient, the stronger the constraint on the weights, and the less prone to overfitting, but may sacrifice some fitting ability).

[0081] Since switching between mixed precision introduces gradient discontinuities between high and low precision regions, potentially affecting the stability of saliency guidance, this embodiment constructs a gradient alignment loss function to strongly constrain the weight update trends on both paths: ; Among them, L align The gradient alignment loss measures the difference in weight gradients between the 4-bit quantization model and the 8-bit model, enabling the quantization model to inherit the gradient characteristics of the high-precision model during training and ensuring accuracy; |P| is the number of elements in set P, which is a set of weight parameters (or their indices) participating in gradient calculation, used to average the gradient differences and reduce the impact of fluctuations in a single sample / parameter. In a 4-bit quantization model, the weight W (p) The corresponding loss function gradient reflects the direction and magnitude of the optimization of the 4-bit model on that weight; In an 8-bit model, the weights W (p) The corresponding gradient of the loss function represents the optimization requirement of the high-precision model for that weight. The training process employs methods such as... Figure 7 The "two-stage strategy" shown.

[0082] S700 performs collaborative image processing on the image to be processed based on the sparsity of each convolutional layer, a first preset precision allocated to the key target region, and a second preset precision allocated to the background region.

[0083] Joint scheduling engine execution logic: Instruction receiving: the scheduling engine synchronously receives the "convolution layer sparsity" output by the HASC module and the "pixel-level precision mask" output by the IGMP module in each image processing cycle.

[0084] Collaborative execution process: ①Structural-level sparse pruning: according to the current convolution layer sparsity, the channel screening logic in the above steps is called to retain the Top-K effective channels, and the layer structure is simplified.

[0085] ②Pixel-level precision control: based on the precision mask, 8-bit convolution calculation is performed on the key area pixels in the effective channel, and 4-bit cluster center convolution calculation is performed on the background area pixels, realizing "layer structure-pixel precision" two-dimensional optimization.

[0086] ③Data caching: the output feature map of each layer is directly transmitted to the next layer or written into the shared memory buffer area to avoid repeated data copying.

[0087] Real-time monitoring and protection: integrate "watchdog mechanism" to monitor the single-layer processing time and chip temperature in real time: If the single-layer processing time is greater than the 20ms threshold, or the chip temperature is greater than 90% of the threshold (such as 94.5°C for Ascend 310), the hardware protection mechanism is triggered.

[0088] Efficiency optimization: adopt "compute graph fusion" technology to fuse "channel pruning, quantization calculation, and activation function" into a single mixed operator (Sparse Mixed Conv), reduce instruction flow redundancy; build a "zero-copy data path", and share memory for hardware state, saliency map, and channel mask, to avoid PCIe / DRAM intermediate copying and improve execution throughput.

[0089] Through the method of this step, the problems of "independent execution of sparsification and quantization, lack of collaboration", "uncontrollable inference delay", and "system crash caused by hardware exception" in the prior art are solved; the scheduling engine is combined to realize "sparsity-precision" collaborative regulation, avoid conflicts between the two (such as sparse pruning causing a large amount of quantization error), and reduce overall power consumption by more than 70%; real-time monitoring and mixed operator optimization ensure that the inference delay is ≤20ms (matching 30FPS frame rate); the zero-copy data path reduces the data transmission time by 30%, further improving the inference efficiency of edge devices.

[0090] Further, the collaborative image processing of the to-be-processed image comprises: According to the sparsity of each convolution layer, the effective calculation channel of the corresponding convolution layer of the image processing model is screened.

[0091] Efficient channel screening implementation: The scheduling engine follows the "layer-by-layer execution" logic. For each convolution layer of the image processing model, it first reads the sparsity rate of the layer, calls the "channel norm sorting + Top-K retention" logic, generates the "channel retention mask" of the layer (marks the effective channels), and only loads the weights of the effective channels to the computing unit, skipping the pruned channels.

[0092] Differential precision calculation for screened convolution layers according to precision allocation results, while real-time monitoring of image processing.

[0093] Differential precision calculation implementation: Precision mask application: For the screened effective channels, load the "pixel-level precision mask" generated by the IGMP module. For key area pixels marked as "1" in the mask, enable 8-bit arithmetic logic units (ALUs) to perform convolution. For background area pixels marked as "0", enable 4-bit ALUs and call the cluster center table to perform convolution.

[0094] Real-time monitoring indicators: Single-layer processing time: Record the start / end time of each convolution layer through a timer to calculate the processing time.

[0095] Chip temperature: Real-time read the compensated chip temperature, and compare with the chip frequency reduction threshold (such as 105℃).

[0096] In the case of detecting that the single-layer processing time exceeds the preset threshold or the chip temperature exceeds the warning value, trigger the preset hardware protection mechanism to ensure stable execution of image processing.

[0097] Abnormal trigger logic: If the single-layer processing time is greater than 20ms, or the chip temperature is greater than 90% of the frequency reduction threshold (such as 94.5℃), immediately trigger the hardware protection mechanism, interrupt the current layer calculation and perform protection actions.

[0098] Efficiency optimization: Operator fusion: Fuse the "channel screening → precision allocation → convolution → activation" series operators into a single mixed operator (Sparse Mixed Conv), reducing the number of data interactions between operators (from 4 to 1).

[0099] Shared memory: Construct a struct Shared Memory data structure to store hardware status, saliency map, and channel mask. Each module directly accesses the shared memory to avoid PCIe / DRAM data copying (such as from memory to GPU memory), improving data transmission efficiency by 30%.

[0100] By the above method, the problems of "independent execution of sparsity and quantization, low efficiency", "no protection for reasoning exception, system easy to crash" in the prior art can be solved; the layer sequence execution and operator fusion ensure smooth collaborative processing process, and the overall reasoning efficiency is improved by more than 60%; real-time monitoring and abnormal triggering avoid delay of whole frame reasoning caused by single layer timeout (such as delay of more than 50 ms in the traditional scheme); shared memory reduces data transmission bottleneck, further adapts to the characteristics of limited memory bandwidth of edge device, and ensures stable operation of the system.

[0101] Further, the preset hardware protection mechanism includes at least one of the following: Enabling the maximum sparsity to perform channel pruning on each convolutional layer, skipping non-key feature extraction layers in the image processing model, switching to a unified first preset precision mode and closing the high-frequency calculation core, caching the current image processing model state to perform breakpoint continuation after hardware recovery.

[0102] Enabling maximum sparsity pruning: the maximum sparsity is fixed at 70%, and all convolutional layers are forced to reselect channels at a sparsity of 70%, reducing more than 60% of the calculation amount and quickly reducing the hardware load.

[0103] Skip non-key feature extraction layers: pre-label the "non-key layers" (such as intermediate redundant feature fusion layers) in the model, trigger protection to directly skip these layers, and only keep the simplified reasoning path of "input layer → core convolutional layer → output layer", and compress the calculation graph structure.

[0104] Switch to a unified 8-bit mode: close the high-frequency calculation core of the chip (such as from 2.0 GHz to 1.2 GHz), and all regions use 8-bit precision calculation (avoid additional control overhead of 4-bit quantization), balance accuracy and power consumption.

[0105] Breakpoint continuation: store the "effective channel mask, precision mask, intermediate feature map" of the current layer to non-volatile memory (such as eMMC), and continue reasoning from the stored state after the hardware state is restored (such as the temperature drops to 85°C), without the need to recalculate the whole frame image.

[0106] Additional protection actions: if the edge device supports external fans, actively start the fans (fan speed is adjusted to the maximum) when triggering protection; if it is a multi-core device, migrate part of the convolution task to the idle core (such as CPU has 8 cores, migrate the task from the fully loaded core to the idle core), further relieve the hardware pressure.

[0107] By the method in this step, the problem of "no recovery mechanism for hardware exception, easy to cause system restart" in the prior art can be solved; the multi-dimensional protection mechanism covers "computing power reduction, path compression, power consumption control, breakpoint recovery", ensuring that the system does not crash in extreme scenarios (such as CPU overload, high chip temperature); breakpoint continuation avoids frame loss caused by protection actions (such as direct discarding of timeout frames in traditional solutions), ensuring the continuity of video stream inference; additional fans and multi-core migration further improve protection effect, adapting to edge scenarios such as vehicle-mounted and security that require extremely high stability.

[0108] Further, the method further comprises the following steps: Periodically collect the execution trajectory data of the PPO policy network under the current hardware state; the execution trajectory data includes a hardware state vector, an output sparsity, and a corresponding reward value, and the reward value comprehensively considers image processing delay, recognition accuracy, sparsity fluctuation degree, and chip temperature.

[0109] The generalized advantage estimation algorithm is used to determine the relative value of each sparsity decision.

[0110] Based on the preset policy update rounds and the policy change threshold, a joint loss function including policy loss, value function error, and policy entropy is constructed.

[0111] The parameters of the PPO policy network are updated by gradient clipping and back propagation.

[0112] In order to make the strategy have self-learning ability, the present application updates the policy network based on the PPO-Clip algorithm. This algorithm limits the change range of the probability ratio of new and old strategies (i.e. "proximal update") to prevent policy shock from causing unstable model performance. The core idea is to maximize the cumulative reward function while ensuring that the difference between the new and old strategies is not too large. The pseudo code of the specific algorithm main process is as follows: for_in range(K_epochs): # Step 1: Collect the execution trajectory of the policy network under the current hardware state, including: # states (hardware feature vector), actions (sparsity), and immediate rewards (from IoU and delay) states, actions, rewards = rollout(policy_net).

[0113] Here, the environment simulator / actual device is called to collect the state-action-reward triplets of the inference process under the current policy, providing an experience basis for subsequent policy gradient.

[0114] # Step 2: Calculate the relative value of each action using Generalized Advantage Estimation (GAE) advantages = generalized_advantage_estimation(rewards)。

[0115] GAE calculates the advantage of each action under the current policy relative to the baseline (value function), balancing bias and variance to stabilize the training process.

[0116] # Step 3: Build the PPO-Clip objective function based on probability ratios ratios = exp(new_log_probs - old_log_probs) surr1 = ratios * advantages surr2 = clip(ratios, 1-ε, 1+ε) * advantages policy_loss = -mean(min(surr1, surr2))。

[0117] The core of this section is to introduce the clip operation to limit the step size of policy updates, avoiding the collapse of the policy due to excessive gradient. ratios is the probability ratio of the current policy to the old policy action.

[0118] # Step 4: Construct a joint loss function, adding value function error and policy entropy total_loss = policy_loss+c1*value_loss-c2*entropy where value_loss is used to regress the expected value of the state; entropy enhances the exploratory nature of the policy, preventing it from falling into a local optimum; c1 and c2 control the relative importance of each term.

[0119] # Step 5: Gradient backpropagation and clipping, update policy network parameters optimizer.zero_grad() total_loss.backward() clip_grad_norm_(parameters, max_norm=0.5) optimizer.step()。

[0120] The clipping gradient is the guarantee of the stability of PPO, and max_norm=0.5 limits the parameter update range to prevent gradient explosion, which is particularly suitable for unstable data flow in edge scenarios.

[0121] wherein K epochs =4 is the update round number of the policy; c1=0.5, c2=0.01: value function and entropy coefficient.

[0122] In order to ensure that the dynamic sparse strategy always has stability and robustness in different edge environments, a multi-objective decision optimization mechanism is further introduced in the embodiment to guide the policy network to more balancedly consider reasoning delay, recognition accuracy, state stability and hardware security in the reinforcement learning training process, so as to avoid the problem of system out of control caused by aggressive compression. The mechanism is developed from three levels of reward function design, chip temperature control protection and burst load response, and strengthens the system level steady-state performance.

[0123] (1) Multi-objective reward function design In the reinforcement learning training phase, in order to realize multi-objective performance optimization, the present application constructs a reward function in the following form: ; Wherein, a is the basic weight coefficient of delay reward, which controls the importance of the "low delay target" in the total reward (positive number, the larger the value, the more emphasis on delay optimization), which can be set to a=1.0; β is the penalty coefficient of delay, which determines the decay rate of "delay growth" on the reward (positive number, the larger the value, the faster the reward decays under the same delay increment), which can be set to β=0.05; Δt is the reasoning delay increment (the difference between the current reasoning delay and the "baseline delay"), the larger the delay, the lower the reward (reflecting "low delay is better").

[0124] γ is the weight coefficient of accuracy reward, which controls the importance of the "high accuracy target" in the total reward (positive number, the larger the value, the more emphasis on accuracy), which can be set to γ=0.8; IoU t is the intersection over union (Intersection over Union), which measures the overlap between the model reasoning result (such as the bounding box of target detection, segmentation mask) and the true label, and the higher the IoU, the better the accuracy, the higher the reward.

[0125] η is the weight coefficient of stability penalty, which controls the punishment intensity of "policy fluctuation" (positive number, the larger the value, the more aversion to policy mutation), which can be set to η=0.2; α t is the sparse / quantization policy parameter at the current time (such as the sparsity of each convolutional layer, the quantization bit width configuration of the pixel); α t-1 is the sparse / quantization policy parameter at the last time.

[0126] ζ is the weight coefficient of overheat penalty, which controls the intensity of punishment for "hardware overheat" (positive number, the larger the value, the more emphasis on hardware thermal management), and ζ can be set to 0.5; I() is an indicator function, which returns 1 if the condition is met, otherwise returns 0; T is the real-time temperature of hardware (such as the temperature of NPU / GPU of edge device); T th is the temperature threshold, when the hardware temperature T exceeds T th , the indicator function returns 1, triggering this penalty (to avoid hardware overheating damage or performance drop).

[0127] Among them, the first term reflects the constraint of inference time on sparsity, the lower the delay, the greater the reward; the second term is the model output and the labeled IoU (or mAP), which reflects the detection / identification accuracy; the third term encourages the output of the strategy network at continuous time to be smooth, avoiding large fluctuations in sparsity; the fourth term judges whether the chip is overheated through binary logic and imposes a penalty term in the strategy.

[0128] (2) Temperature adaptive protection mechanism To prevent the risk of chip overheating triggered by too low sparsity, causing frequency degradation or even system restart, a protection mechanism based on temperature ratio is set in this embodiment: When ; represents that it is effective for all convolutional layers, that is, when the temperature approaches the threshold, the sparsity of all convolutional layers will be adjusted according to this rule.

[0129] This strategy adds adjustment operations after the output of the strategy network, forcing to increase the upper limit of sparsity and thus reducing the operation intensity. In addition, the system can also link the following emergency mechanisms: ① Temporarily skip the shallow calculation block to compress the early calculation graph structure; ② Migrate part of the convolution task to the idle core (multi-core device); ③ If the external fan is supported, actively start the air cooling system for heat dissipation compensation.

[0130] (3) Burst load response mechanism In some burst situations, such as sudden increase of background system calls, video stream burst upstream, etc., the CPU usage rate may change dramatically within a few milliseconds. At this time, in order to prevent the response delay from rising or the power consumption from exploding, the system is designed with the following response logic: ; Among them is the load change rate threshold, with the unit of % / ms. If the load rises, the sparsity is increased to relieve the pressure; if the load drops, the strategy is automatically withdrawn to improve the accuracy. This dynamic adjustment strategy can maintain the stability of model performance in the case of dramatic changes in system load.

[0131] In this embodiment, by acquiring the hardware running state data of the edge device in real time, such as CPU utilization, memory bandwidth and chip temperature, and inputting the anti-interference processed data into the PPO strategy network to dynamically output the sparsity of each convolution layer, the problem that the existing static pruning / quantization scheme cannot adapt to the real-time resource fluctuation of hardware is effectively solved, the inference delay is avoided from increasing sharply and the model is avoided from failing in high load or high temperature environment, the real-time performance and stability of the system are ensured, the key target area and the background area are divided by analyzing the saliency area of the image, and higher first preset accuracy and lower second preset accuracy are respectively allocated for processing, thereby breaking through the limitation of global unified quantization ignoring the accuracy differentiation requirement of different areas, and the accuracy loss of key target recognition in complex scenes such as rain, fog, occlusion and low light is significantly reduced. At the same time, the sparsity and quantization are cooperatively processed by combining the sparsity of each convolution layer and the differentiated accuracy allocation, the problem that the low-precision calculation error is amplified by independent processing of sparsity pruning is avoided, the dynamic sparsity strategy can be adaptively adjusted by the PPO strategy network, does not need to rely on offline training and manual super parameter adjustment, can better adapt to long-term changes such as hardware aging and environment switching after deployment, and further improves the large-scale landing ability and continuous optimization performance of the edge intelligent vision system.

[0132] In addition, although the steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be divided into multiple steps, etc.

[0133] The embodiments of the present application also provide an image processing device based on hardware perception and dynamic sparsity as shown in Figure 8 The device comprises: A hardware state perception module is configured to acquire hardware running state data of the edge device, and the hardware running state data at least includes CPU utilization, memory bandwidth and chip temperature.

[0134] A PPO dynamic decision module is configured to input the anti-interference processed hardware running state data into a preset PPO strategy network, so as to output the sparsity of each convolution layer of the image processing model in the edge device through the PPO strategy network, determine the number of channels to be reserved for each convolution layer according to the sparsity of each convolution layer, and sort the channels of each convolution layer according to importance, and reserve a specified number of channels at the front of the sorting as effective calculation channels.

[0135] The input guiding mixed precision module is configured to perform saliency region analysis on the input image to be processed to generate a pixel-level saliency map corresponding to the image to be processed; divide the image to be processed into a key target region and a background region according to the saliency map; assign the first preset precision to the key target region for image processing, and assign the second preset precision to the background region for image processing; and the first preset precision is greater than the second preset precision.

[0136] The joint scheduling execution module is configured to perform collaborative image processing on the image to be processed according to the sparsity of each volume layer and the first preset precision assigned to the key target region and the second preset precision assigned to the background region.

[0137] Further, the hardware state sensing module is further configured to: Obtain the information entropy of the hardware state data sequence in the sliding window.

[0138] In a case where the information entropy is lower than a preset information entropy threshold, it is determined that the corresponding sensor is invalid, and the hardware state data of the previous sampling period is enabled.

[0139] In combination with the ambient temperature collected by the ambient temperature sensor, the chip temperature is linearly compensated according to a preset compensation coefficient; the value of the preset compensation coefficient is such that the error of the compensated chip temperature is controlled within ±1℃.

[0140] The current values of the CPU utilization rate, the memory bandwidth and the chip temperature are combined with the CPU utilization rate change rate and the memory bandwidth change rate to form a multi-dimensional state vector for inputting the PPO policy network; wherein the chip temperature is included in the multi-dimensional state vector in the form of a ratio of the chip temperature to a chip frequency reduction temperature threshold.

[0141] Further, the device further comprises a parameter updating module configured to: Periodically collect execution trajectory data of the PPO policy network under the current hardware state; the execution trajectory data includes a hardware state vector, an output sparsity and a corresponding reward value, and the reward value comprehensively considers image processing delay, recognition accuracy, sparsity fluctuation degree and chip temperature.

[0142] The generalized advantage estimation algorithm is used to determine the relative value of each sparsity decision.

[0143] Based on a preset policy update number and a policy change threshold, a joint loss function including a policy loss, a value function error and a policy entropy is constructed.

[0144] The parameters of the PPO policy network are updated by gradient clipping and back propagation.

[0145] Further, the input guiding mixed precision module is further configured to: According to the sparsity of each convolutional layer, the corresponding convolutional layer of the image processing model is subjected to effective calculation channel screening.

[0146] According to the precision distribution result, the screened convolutional layer is subjected to differential precision calculation, and the image processing process is monitored in real time.

[0147] In the case that the single-layer processing time exceeds a preset threshold or the chip temperature exceeds a warning value, a preset hardware protection mechanism is triggered to ensure stable execution of the image processing.

[0148] Embodiments of the present application also provide an electronic device comprising a processor and the aforementioned non-transitory computer-readable storage medium.

[0149] The electronic device is merely an example, and should not limit the function and use range of the embodiments of the present application.

[0150] The electronic device is in the form of a general-purpose computing device. The components of the electronic device can include, but are not limited to, the above-mentioned at least one processor, the above-mentioned at least one memory, a bus connecting different system components including the memory and the processor.

[0151] The memory stores program codes, which can be executed by the processor, so that the processor executes the steps in various embodiments described in the specification.

[0152] The memory can include a readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory, and can further include a read-only memory (ROM).

[0153] The memory can also include programs / utilities with a set of (at least one) program modules, such as operating systems, one or more application programs, other program modules, and program data, each of which or some combination of which can include the implementation of a network environment.

[0154] The bus can represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures.

[0155] The electronic device can also communicate with one or more external devices such as a keyboard or a pointing device, through an I / O interface. The electronic device can communicate with one or more devices that enable a user to interact with it through a communication interface. The electronic device can also communicate with one or more devices or networks (e.g., LANs, WANs, and / or the Internet) through a network adapter. The network adapter can communicate with the other modules of the electronic device through the bus. It should be appreciated that other hardware and / or software modules can be used in conjunction with the electronic device. For example, a microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc. can be used with the electronic device.

[0156] From the above description of the embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to make a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) execute the method according to the embodiments of the present disclosure.

[0157] The embodiments of the present disclosure also provide a computer program product, which includes program codes for causing an electronic device to perform the steps in the method according to various example embodiments of the present disclosure described above in the specification when the program product is run on the electronic device.

[0158] Although some specific embodiments of the present disclosure have been described in detail by way of examples, it should be appreciated that the above examples are only for illustration, but not to limit the scope of the present disclosure. It should also be appreciated by those skilled in the art that various modifications can be made to the embodiments without departing from the scope and spirit of the present disclosure.

Claims

1. An image processing method based on hardware awareness and dynamic sparsity, characterized in that, include: Acquire hardware operating status data of edge devices; The hardware operating status data includes at least CPU utilization, memory bandwidth, and chip temperature. The hardware operating status data after anti-interference processing is input into the preset PPO policy network, so as to output the sparsity of each convolutional layer of the image processing model in the edge device through the PPO policy network. The number of channels to be retained in each convolutional layer is determined based on the sparsity of each convolutional layer, and the channels of each convolutional layer are ranked according to their importance based on their channel norm. The specified number of channels ranked first are retained as effective computational channels. The input image to be processed is subjected to salient region parsing to generate a pixel-level salient map corresponding to the image to be processed. The image to be processed is divided into key target regions and background regions based on the saliency map; Image processing is performed on key target areas with a first preset precision, and image processing is performed on background areas with a second preset precision. The first preset precision is greater than the second preset precision; Based on the sparsity of each convolutional layer and the first preset precision assigned to the key target region and the second preset precision assigned to the background region, collaborative image processing is performed on the image to be processed.

2. The image processing method based on hardware awareness and dynamic sparsity according to claim 1, characterized in that, After obtaining the hardware operating parameters of the edge device, the process includes: Obtain the information entropy of the hardware status data sequence within the sliding window; If the information entropy is lower than the preset information entropy threshold, the corresponding sensor is determined to be faulty, and the hardware status data of the previous sampling period is activated. Based on the ambient temperature collected by the ambient temperature sensor, the chip temperature is linearly compensated according to a preset compensation coefficient; the value of the preset compensation coefficient is selected to control the chip temperature error after compensation within ±1℃.

3. The image processing method based on hardware awareness and dynamic sparsity according to claim 1, characterized in that, Before inputting the anti-interference processed hardware operating status data into the preset PPO policy network, the following steps are included: The current values ​​of CPU utilization, memory bandwidth, and chip temperature are combined with the rate of change of CPU utilization and the rate of change of memory bandwidth to form a multidimensional state vector for inputting the PPO policy network; wherein, the chip temperature is included in the multidimensional state vector in the form of the ratio of chip temperature to the chip throttling temperature threshold.

4. The image processing method based on hardware awareness and dynamic sparsity according to claim 1, characterized in that, The PPO policy network is a three-layer fully connected network structure, which contains 128, 64, and 32 neurons respectively, and uses ReLU as the activation function. The output layer of the PPO policy network adopts a group activation method, so that each output corresponds to the sparsity of a single convolutional layer of the image processing model, and the sparsity value is limited to 0-70% by the activation function.

5. The image processing method based on hardware awareness and dynamic sparsity according to claim 1, characterized in that, Also includes: Periodically collect execution trajectory data of the PPO policy network under the current hardware state; The execution trajectory data includes a hardware state vector, the output sparsity, and the corresponding reward value. The reward value takes into account image processing latency, recognition accuracy, sparsity fluctuation, and chip temperature. The generalized advantage estimation algorithm is used to determine the relative value of each sparsity decision; Based on the preset number of policy update rounds and policy change threshold, a joint loss function is constructed that includes policy loss, value function error, and policy entropy; The parameters of the PPO policy network are updated through gradient pruning and backpropagation.

6. The image processing method based on hardware awareness and dynamic sparsity according to claim 1, characterized in that, The image processing of assigning a second preset precision to the background region includes: The K-Means clustering algorithm is used to aggregate the convolution kernel weights of the image processing model into a predetermined number of cluster centers; When processing the background region, the cluster center is used to replace the original convolution kernel weights to perform convolution calculations by looking up a table, thereby reducing the computational cost of background region processing.

7. The image processing method based on hardware awareness and dynamic sparsity according to claim 1, characterized in that, The collaborative image processing of the image to be processed includes: The effective computational channels of the corresponding convolutional layer of the image processing model are selected based on the sparsity of each convolutional layer. Differential precision calculations are performed on the selected convolutional layers according to the precision allocation results, while the image processing process is monitored in real time. If the single-layer processing time exceeds a preset threshold or the chip temperature exceeds a warning value, a preset hardware protection mechanism is triggered to ensure stable execution of image processing.

8. The image processing method based on hardware awareness and dynamic sparsity according to claim 1, characterized in that, The default hardware protection mechanisms include at least one of the following: Enable maximum sparsity to perform channel pruning on each convolutional layer, skip non-critical feature extraction layers in the image processing model, switch to a unified first preset precision mode and disable high-frequency computing kernels, and cache the current image processing model state to resume execution from the breakpoint after hardware recovery.

9. An image processing device based on hardware sensing and dynamic sparsity reduction, characterized in that, The device includes: The hardware status awareness module is used to acquire hardware operating status data of the edge device; the hardware operating status data includes at least CPU utilization, memory bandwidth and chip temperature. The PPO dynamic decision module is used to input the hardware operating status data after anti-interference processing into the preset PPO policy network, so as to output the sparsity of each convolutional layer of the image processing model in the edge device through the PPO policy network; determine the number of channels to be retained in each convolutional layer according to the sparsity of each convolutional layer, and sort the channels of each convolutional layer according to their importance based on the channel norm, and retain the specified number of channels with the highest ranking as effective calculation channels. The input-guided mixed-precision module is used to perform salient region analysis on the input image to be processed, and generate a pixel-level salient map corresponding to the image to be processed; the image to be processed is divided into key target regions and background regions according to the salient map; a first preset precision is assigned to the key target regions for image processing, and a second preset precision is assigned to the background regions for image processing; the first preset precision is greater than the second preset precision. The joint scheduling execution module is used to perform collaborative image processing on the image to be processed based on the sparsity of each convolutional layer, the first preset precision allocated to the key target region, and the second preset precision allocated to the background region.

10. An electronic device, characterized in that, It includes a processor and the image processing apparatus based on hardware awareness and dynamic sparsification as described in claim 9.

Images