Fish-eye image multi-branch adaptive enhancement method based on hardware state perception

Abstract

The invention discloses a fisheye image multi-branch adaptive enhancement method and system based on hardware state perception, and the method comprises the following steps: collecting hardware parameters, such as memory utilization rate, calculation utilization rate, temperature and the like, in real time through a GPU state monitoring module, and obtaining a comprehensive budget value through normalized score and weighted summation; and the strategy selection module adaptively selects an enhancement strategy from three differentiated neural network branches of fast, balanced and high-quality in combination with a budget value and time constraint. Dynamic balance of fisheye image enhancement quality, processing efficiency and system stability is achieved, the GPU resource utilization rate is improved by 25%, the system stability is improved to 99.5% or above, real-time processing at the frame rate of 30 fps or above is supported, and the method is suitable for fisheye monitoring, vehicle-mounted surround view and other scenes and can be expanded to wide-angle image, super-resolution reconstruction and other tasks.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, specifically relating to a hardware state-aware multi-branch adaptive enhancement method for fisheye images. Background Technology

[0002] Fisheye lenses are widely used in security monitoring, vehicle systems, panoramic photography, and virtual reality due to their ultra-wide field of view (typically greater than 180°). However, fisheye images generally suffer from quality problems such as edge distortion, uneven brightness, and blurred details, which seriously affect visual effects and subsequent image analysis tasks, necessitating improvements through image enhancement techniques.

[0003] Existing fisheye image enhancement technologies mainly suffer from the following technical problems: Fixed and singular processing strategy: Traditional fisheye image enhancement methods use a single processing algorithm, which cannot be dynamically adjusted according to actual hardware conditions and application requirements. When GPU resources are sufficient, it cannot achieve optimal performance, resulting in resource waste; when the GPU load is too high, it is prone to system crashes due to memory overflow or overheating, affecting system stability and reliability.

[0004] Lack of hardware resource awareness mechanisms: Existing fisheye image processing systems lack real-time monitoring of hardware status such as GPU memory usage, computational load, and device temperature, making it impossible to optimize processing strategies based on hardware conditions. This results in a high failure rate in resource-constrained environments (such as edge computing devices and embedded systems), while failing to fully utilize hardware performance in resource-rich environments.

[0005] Unpredictable processing time: Existing methods cannot select appropriate fisheye image enhancement strategies based on real-time requirements (such as video frame rate requirements and response latency limitations), resulting in problems such as unstable processing latency, frame loss, and delayed response in real-time application scenarios, which affects user experience.

[0006] There is an inherent contradiction between quality and efficiency: high-quality fisheye image enhancement algorithms (such as deep neural network-based methods) have high computational complexity and long processing time, while fast processing algorithms (such as traditional image processing methods) have poor enhancement quality. Existing technologies lack a mechanism to intelligently balance quality and efficiency under different application scenarios and hardware conditions.

[0007] Lack of adaptive optimization capability: Existing systems mostly use fixed parameters and thresholds, which cannot be adaptively adjusted and optimized according to actual operating conditions, making it difficult to adapt to the differentiated needs of different hardware platforms and application scenarios.

[0008] Existing technologies cannot solve the above problems simultaneously, and a new adaptive enhancement scheme is urgently needed. Summary of the Invention

[0009] The core technical problem that this invention aims to solve is: how to achieve a dynamic balance between fisheye image enhancement quality and processing efficiency by intelligently selecting processing strategies among multiple complexity enhancement branches through real-time monitoring of GPU hardware resource status, while ensuring system stability and controllable latency.

[0010] To address the aforementioned technical problems, this invention provides a multi-branch adaptive enhancement method for fisheye images based on hardware state awareness, comprising the following steps: S1: Real-time monitoring of GPU hardware state parameters, including at least memory usage, computational utilization, and temperature; calculating the normalized score of each item based on the hardware state parameters, and obtaining a comprehensive hardware state budget value through weighted summation; S2: Based on the hardware state budget value described in step S1, select one of at least three fisheye image processing strategies, including fast enhancement, balanced enhancement, and high-quality enhancement. The different strategies correspond to different target processing time, maximum resource consumption, and expected image quality. S3: Enhance the fisheye image using the neural network branch corresponding to the selected strategy; S4: Evaluate the quality of the enhanced fisheye image and feed back the processing time and quality score to the strategy selection module.

[0011] Furthermore, the calculation process of the hardware state budget value in step S1 includes the following steps: S1.1: Calculate the memory score, which is negatively correlated with memory usage; S1.2: Calculate the utilization score that is negatively correlated with GPU utilization, with reference to the safe utilization threshold of 80%-95%; S1.3: Calculate the temperature score that is negatively correlated with the current temperature, with reference to the temperature safety threshold of 80°C-90°C; S1.4: The initial budget value is calculated using a weighted summation method, with a total weight of 1; S1.5: Normalize the initial budget value to a preset range to ensure that minimum processing capacity is retained in extreme cases.

[0012] Furthermore, the specific calculation formulas for each step are as follows: The memory score formula is: ,in, Indicates the memory score. Indicates the current memory usage. Indicates the total amount of available memory; Utilization score formula: ,in, This represents the GPU utilization score. This indicates the current GPU utilization rate; Temperature score formula: ,in, Indicates the temperature score. Indicates the current temperature; Comprehensive budget formula: ,in, Indicates the budget value. Indicates the memory score. This represents the GPU utilization score. Indicates the temperature score; The budget value is normalized to the range [0.1, 1.0].

[0013] Furthermore, the selection rule for the fisheye image enhancement strategy in step S2 is as follows: Select the rapid enhancement branch when the budget value is below the first threshold; Select the balanced enhancement branch when the budget value is between the first and second thresholds; Select the high-quality enhancement branch when the budget value is higher than the second threshold; When time constraints exist, the strategy selection is adjusted according to the time limit.

[0014] Furthermore, the first threshold is 0.2-0.4, preferably 0.3; the second threshold is 0.6-0.8, preferably 0.7; The target processing time for the fast enhancement branch is 2-5ms, and the maximum memory is 50-150MB; the target processing time for the balanced enhancement branch is 8-15ms, and the maximum memory is 300-700MB; the target processing time for the high-quality enhancement branch is 20-30ms, and the maximum memory is 1000-2000MB.

[0015] Furthermore, when the time constraint is no greater than the first time threshold, the fast enhancement branch is forcibly selected; when the time constraint is between the first and second time thresholds, the highest restriction is the balanced enhancement branch; when the time constraint is greater than the second time threshold, the strategy selection based on the budget value is maintained.

[0016] Furthermore, the three fisheye image enhancement network branches include: fast enhancement branch, balanced enhancement branch, and high-quality enhancement branch; The fast enhancement branch employs a shallow convolutional network with depthwise separable convolutions, selectively reducing processing resolution, introducing residual connections, 3 convolutional layers, 32 channels, depthwise separable convolutions, 0.5x resolution reduction, and residual weights of 0.2-0.4. The balanced enhancement branch adopts an encoder-decoder architecture, integrates an attention mechanism, performs multi-scale feature fusion, has 6 layers of encoder-decoder, has [64, 128, 128] channels, channel and spatial attention mechanisms, and residual weights of 0.4-0.6. The high-quality enhancement branch adopts a deep complex network, integrates multiple enhancement modules, uses a dense connection or multi-path architecture, has a 12-layer network, has a channel number of [64, 128, 256], multi-head attention and feature refinement modules, and residual weights of 0.6-0.8.

[0017] Furthermore, the fisheye image quality assessment in step S4 is specifically implemented as follows: S4.1: Variance Evaluation: Calculate and normalize the variance of the grayscale image; , in, Indicates the variance score. Represents the variance of a grayscale image; S4.2: Sharpness assessment: Calculate and normalize edge strength based on gradient operators (such as the Laplacian operator); , in, Indicates the sharpness score. This represents the variance of the Laplace operator; S4.3: When an original reference image is available, calculate similarity metrics such as PSNR or SSIM; ,in Peak signal-to-noise ratio; S4.4: Obtain the overall quality score through weighted combination.

[0018] in Represents the overall quality score. Represents the variance score. Represents the sharpness score. This represents the PSNR score.

[0019] Furthermore, a fisheye image enhancement system includes: a GPU status monitoring module: real-time monitoring of GPU hardware status parameters, collecting GPU memory usage, computational utilization, and temperature parameters, and outputting hardware status data; a strategy selection module: receiving hardware status data, calculating a comprehensive budget value, and outputting enhancement strategy selection results based on budget rules and time constraints; a multi-branch enhancement network module: containing three neural network branches—fast enhancement, balanced enhancement, and high-quality enhancement—to enhance fisheye images; a quality evaluation module: calculating a comprehensive quality score using variance, sharpness, and PSNR metrics to evaluate the quality of the enhanced fisheye image; and a feedback optimization module: feeding back processing time, resource consumption, and quality score to the strategy selection module to optimize decision-making.

[0020] Furthermore, the GPU status monitoring module is implemented based on the GPUtil library or the nvidia-smi interface, with a monitoring cycle of 50-200ms. The monitored parameters include GPU memory usage, computing utilization, and temperature. The parameter update frequency is synchronized with the image acquisition frame rate. The system supports real-time processing of video streams at frame rates of 30fps and above, and is suitable for real-time enhancement scenarios of fisheye images.

[0021] The present invention has the following advantages: Compared with the prior art, the present invention has the following advantages: 1. Adaptive Hardware Resource Utilization: By monitoring GPU memory, utilization, temperature, and other hardware statuses in real time, the system dynamically calculates the overall budget value and automatically selects the optimal fisheye image enhancement strategy based on hardware conditions. Experiments show that compared to fixed strategy methods, this invention can improve GPU utilization by more than 25% while avoiding system crashes due to insufficient resources (the crash rate is reduced from 5-10% in traditional methods to below 0.5%).

[0022] 2. Controllable fisheye image enhancement quality: The three-branch architecture provides different quality levels of enhancement (quality scores of approximately 0.75, 0.83, and 0.89 respectively), allowing for flexible selection based on application needs. The fast branch is used in real-time scenarios such as security monitoring, while the high-quality branch is used in high-quality scenarios such as image archiving, meeting the differentiated needs of various applications.

[0023] 3. Predictable processing time: Each branch has a clear time target (3ms, 12ms, 25ms), and the actual processing time fluctuates little (standard deviation <2ms), meeting the requirements of real-time fisheye image processing. In 30fps video stream processing, frame latency is controlled within 33ms to avoid frame drops and stuttering.

[0024] 4. Significantly improved system stability: Hardware safety mechanisms (temperature threshold 85°C, utilization threshold 90%) prevent system crashes caused by GPU overheating or memory overflow. In long-term operation tests (72 hours of continuous operation), the system stability reached over 99.5%, significantly better than the 85-90% stability of fixed strategy methods.

[0025] 5. Dynamic balance between quality and efficiency: Through intelligent strategy selection, processing efficiency is maximized while ensuring image quality. Compared to always using the high-quality branch, the average processing time is reduced by 40%; compared to always using the fast branch, the average image quality is improved by 15%.

[0026] 6. Wide range of applications: Although designed for fisheye images, the technical solution of this invention can also be applied to other image enhancement tasks that require real-time processing, such as panoramic images and super-resolution reconstruction. Attached Figure Description

[0027] Figure 1 This is a diagram showing the overall architecture of the fisheye image enhancement system of this application.

[0028] Figure 2 This is a performance comparison chart of processing time in Example 2.

[0029] Figure 3 This is a performance comparison chart of the quality scores in Example 2.

[0030] Figure 4 This is a performance comparison chart of memory usage in Example 2.

[0031] Figure 5 This is a performance comparison chart of the efficiency analysis in Example 2.

[0032] Figure 6 This is a visualization of the GPU budget calculation weight allocation in Example 2.

[0033] Figure 7 A visualization of the hardware security threshold settings for Example 2.

[0034] Figure 8 A visualization of the strategy selection boundary for Example 2.

[0035] Figure 9 This is a visualization of the quality assessment weight allocation for Example 2. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0038] Example 1

[0039] This invention provides a hardware-state-aware multi-branch adaptive enhancement method for fisheye images, comprising the following steps: Step S1: Real-time monitoring of GPU hardware status and comprehensive budget calculation

[0040] 1. Hardware status parameter acquisition: GPU hardware status parameters are monitored in real time through GPU monitoring libraries (such as GPUtil, nvidia-smi, etc.), including but not limited to: (1) GPU memory usage (used memory / total memory), (2) GPU computing utilization (current load percentage), and (3) GPU temperature (degrees Celsius).

[0041] 2. Individual Hardware Status Score: A normalized score is calculated for each hardware status parameter. The score is negatively correlated with resource utilization. (1) Memory score calculation: , Indicates the memory score. Indicates the current memory usage. This represents the total amount of available memory. The score ranges from [0, 1], with higher memory usage resulting in a lower score.

[0042] (2) Utilization score calculation: , This represents the GPU utilization score. This indicates the current GPU utilization rate. This represents the safety utilization rate threshold. The safety utilization rate threshold can be set between 80% and 95%, with 90% being preferred.

[0043] (3) Temperature score calculation: , Indicates the temperature score. Indicates the current temperature. This indicates the temperature safety threshold, which can be set between 80°C and 90°C, preferably 85°C.

[0044] 3. Comprehensive Budget Calculation: The comprehensive budget is obtained by weighted summation. , Among them, the weighting coefficient , , satisfy: .

[0045] 4. Budget value normalization: Map the calculated budget value B to the range [0.1, 1.0] to ensure that minimum processing capacity is retained even in extreme cases. The calculation formula is as follows: ,in Let represent the normalized budget value, B represent the original budget value, α is 0.1, and β is 0.9.

[0046] Step S2: Adaptive selection of fisheye image enhancement strategy

[0047] 1. Budget-based strategy selection: Based on a comprehensive budget calculation, a segmented mapping method is used to select the fisheye image enhancement strategy. (1) Quickly enhance branch selection criteria: budget value <First threshold; Target processing time: 2-5ms; Maximum memory usage: 50-150MB; Applicable scenarios: GPU resource shortage, high real-time requirements.

[0048] (2) Balanced enhancement branch selection condition: First threshold ≤ budget value <Second threshold; Target processing time: 8-15ms; Maximum memory usage: 300-700MB; Applicable scenarios: Moderate resources, balancing quality and efficiency.

[0049] (3) Selection criteria for high-quality enhanced branches: budget value ≥ Second threshold; Target processing time: 20-30ms; Maximum memory usage: 1000-2000MB; Applicable scenarios: Sufficient GPU resources, high-quality requirements.

[0050] 2. Time Constraint Adjustment Mechanism: When external time constraints exist (such as frame rate requirements or real-time limitations), the strategy selection is adjusted a second time. (1) Forced fast strategy condition: time constraint ≤ first time threshold, forced selection of fast enhancement branch, ignoring budget value.

[0051] (2) Conditions for limiting the balancing strategy: First time threshold < time constraint ≤ Second time threshold, the highest limit is the balancing enhancement branch, even if the budget value allows for high-quality branches.

[0052] (3) Maintain basic strategy conditions: time constraint > second time threshold, and maintain strategy selection results based on budget value.

[0053] Step S3: Multi-branch fisheye image enhancement network processing

[0054] Three parallel deep neural network branches are constructed, each designed for different computational budgets and quality requirements: 1. Quickly enhance branch network architecture: Network layers: 3-layer convolutional neural network (shallow architecture), channel configuration [32, 32, 32] (low number of channels); Convolution type: depthwise separable convolution, significantly reducing the number of parameters and computational cost; Resolution strategy: 0.5x down-resolution processing (e.g., scaling 1920×1080 to 960×540), followed by upsampling restoration; Residual connections: Residual connections are introduced, with weight coefficients of 0.2-0.4; Activation function: ReLU; Maximum memory usage: limited to 50-150MB; Technical features: Sacrificing some detail retention capabilities in exchange for extremely fast processing speed.

[0055] 2. Balanced Enhanced Branch Network Architecture: Number of Network Layers: 6-layer encoder-decoder network (U-Net-like structure); Channel Configuration: Encoder [64, 128, 128], decoder symmetrical design; Attention Mechanism: Integration of channel attention and spatial attention; Feature fusion: Multi-scale feature fusion, integrating features from different levels through skip connections; Residual connections: Weight coefficients 0.4-0.6; Activation function: ReLU; Maximum memory usage: Limited to 300-700MB; Technical features: Achieving a good balance between quality and speed.

[0056] 3. High-quality enhanced branch network architecture: Number of network layers: 12-layer complex deep network; Channel configuration: [64, 128, 256]; Enhancement Module: Integrates multi-head attention, feature refinement module, and edge enhancement module (for edge distortion in fisheye images); Connection method: dense connection; Residual connection: weight coefficient 0.6-0.8; Activation function: GELU; Maximum memory usage: 1000-2000MB; Technical features: Maximizes quality enhancement and comprehensively improves all indicators of fisheye images.

[0057] Shared technical features: The three branches can share the underlying feature extractor, reducing the total number of parameters; batch normalization is adopted, and the output layer uses Sigmoid or Tanh activation to ensure that the pixel value range is legal; a special loss function (such as edge-weighted loss) is designed for the characteristics of fisheye images.

[0058] Step S4: Fisheye Image Quality Assessment and Feedback Optimization

[0059] 1. Multi-dimensional quality assessment indicators: Variance score ,in, Indicates the variance score. Represents the variance of a grayscale image. This represents the normalization factor 1. The normalization factor 1 can be set to 500-2000 to ensure that the score falls within a reasonable range. Sharpness rating ,in, Indicates the sharpness score. This represents the variance of the Laplace operator. This represents the normalization factor 2, which can be set to 5000-20000. The Laplacian operator can use a 3×3 or 5×5 convolution kernel.

[0060] PSNR calculation: , Score = (Normalized to the [0,1] interval, assuming) Typically between 20-50 dB), where Peak signal-to-noise ratio, The maximum pixel value. Mean square error: .

[0061] 2. Calculation of overall quality score: ,in Represents the overall quality score. Represents the base score. Represents variance weights. Represents the variance score. Represents the sharpness weight. Represents the sharpness score. Represents PSNR weights. This represents the PSNR score.

[0062] Parameter configuration: Base score: 0.2-0.4, ensuring minimum quality approval; variance weight α: 0.2-0.4; sharpness weight β: 0.3-0.5; PSNR weight γ: 0.2-0.4; total weights: α + β + γ ≤ 1; the overall score range is usually between [0.3, 1.0].

[0063] Feedback mechanism: Record the actual processing time (milliseconds), memory usage (MB), and overall quality score, and feed them back to the strategy selection module to optimize subsequent branch selection decisions.

[0064] Example 2

[0065] This embodiment is based on the previous embodiment. This embodiment applies the present invention to a fisheye surveillance camera system to achieve real-time image enhancement processing at a resolution of 1920×1080 and a frame rate of 30fps, which is suitable for the high real-time performance and stability requirements of security monitoring scenarios.

[0066] I. Hardware Configuration: GPU model: NVIDIA RTX 4060Ti (or equivalent GPU, such as RTX 3060Ti, RTX 4060, etc.); VRAM capacity: 16GB; Cooling system: Standard air cooling.

[0067] II. Parameter Settings: Hardware security thresholds: GPU temperature security threshold: 85°C; GPU security utilization threshold: 90%; Available video memory: 14GB.

[0068] Strategy selection thresholds: Fast / Balanced boundary (first threshold): 0.3; Balanced / High-Quality boundary (second threshold): 0.7; Time constraint thresholds: Forced fast threshold: 5ms; Constrained balance threshold: 15ms; Weight configuration: Memory weights : 0.4; Utilization weight : 0.3; Temperature weighting : 0.3; Quality assessment configuration: variance normalization factor 1000, sharpness normalization factor 10000, variance weight 0.3, sharpness weight 0.4, PSNR weight 0.3, base score 0.3.

[0069] III. Fisheye Image Processing Workflow 1. The surveillance camera captures fisheye images in real time (resolution 1920×1080, frame rate 30fps).

[0070] 2. The GPU status monitoring module obtains the current hardware status through the GPUtil library: memory usage 11.2GB / 14GB, utilization 65%, temperature 72°C; 3. Calculate the score for each item: Memory score = 1.0 - 11.2 / 14 = 0.20; Utilization score = 1.0 - 65 / 90 ≈ 0.28; Temperature score = 1.0 - 72 / 85 ≈ 0.15; 4. Calculate the overall budget: 0.4×0.20+0.3×0.28+0.3×0.15=0.209; Normalized budget value: 0.1+0.9×0.209≈0.288; 5. Strategy Selection: Based on the budget value of 0.288 < 0.3, select the rapid enhancement branch; 6. Rapidly enhanced branch processing: Through a 3-layer depthwise separable convolutional network, the resolution is downgraded by 0.5 times to 960×540, then upsampled to restore the original resolution, with residual connection weights of 0.3; 7. Output the enhanced fisheye image; 8. Quality assessment: Variance score 0.28, sharpness score 0.35, PSNR score 0.78, overall quality score 0.3 + 0.3×0.28 + 0.4×0.35 + 0.3×0.78 ≈ 0.756; 9. Feedback Optimization: Record processing time of 2.8ms and memory usage of 85MB, and feed back to the strategy selection module.

[0071] Table 1 Performance Comparison Table

[0072] Experimental results (based on RTX 4060 Ti testing), quality index comparison (based on high-quality branches): (1) Fast branching: PSNR is reduced by about 4dB, and processing speed is increased by 8.5 times. (2) Balanced branch: PSNR is reduced by about 2dB, and processing speed is increased by 2.1 times. System stability test: After running continuously for 72 hours, a total of 7,776,000 frames were processed, with 7,772,850 frames successfully processed (99.96%), 18,456 policy switches, 0 system crashes, an average GPU temperature of 76°C, and an average GPU utilization of 73%.

[0073] Adaptive effect verification: • As the GPU load increased from 50% to 85%, the system automatically switched from a high-quality branch to a balanced branch, preventing performance degradation; • As the GPU temperature rises from 70°C to 82°C, the system gradually degrades to fast branch to prevent overheating; • After the external program releases video memory (the available video memory increases from 8GB to 12GB), the system automatically upgrades to a high-quality branch to make full use of resources.

[0074] Comparative Experiment 1: Using a fixed high-quality algorithm (1) Average processing time: 24.1 ms; (2) When the GPU load is high (>85%), an out-of-memory error occurs, with a failure rate of 8.3%; (3) Average GPU temperature: 81°C (high).

[0075] Comparative Experiment 2: Fixed Use of Fast Algorithm (1) Average processing time: 2.9ms; (2) Image quality score: 0.758 (9.1% lower than the adaptive method); (3) The hardware performance was not fully utilized when resources were plentiful.

[0076] The adaptive method of this invention: (1) Average processing time: 14.6ms (dynamically adjusted according to hardware status); (2) Average quality score: 0.817; (3) Failure rate: <0.5%; (4) Resource utilization rate increased by 25%; (5) The GPU temperature is controlled at 75-78°C.

[0077] Compared with traditional methods Fixed high-quality algorithm: average processing time 24.1ms, failure rate 8.3% under high GPU load, average temperature 81°C; Fixed fast algorithm: average processing time 2.9ms, quality score 0.758, insufficient resource utilization; The adaptive method of this invention has an average processing time of 14.6ms, an average quality score of 0.817, a failure rate of <0.5%, and a 25% improvement in resource utilization.

[0078] This embodiment combines Figures 1-9 The architecture design and data visualization verified the feasibility and superiority of the method and system of this invention. Through real-time hardware status awareness and multi-branch adaptive selection, a dynamic balance between fisheye image enhancement quality, processing efficiency, and system stability was achieved. Compared with traditional fixed-strategy methods, GPU resource utilization was improved by 25%, and system stability was increased to over 99.5%. It can be widely applied to real-time image enhancement scenarios such as fisheye monitoring and vehicle surround view.

[0079] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A hardware-state-aware multi-branch adaptive enhancement method for fisheye images, characterized in that, The steps include: S1: Real-time monitoring of GPU hardware status parameters, including at least memory usage, computing utilization, and temperature; calculating the normalized score of each item based on the hardware status parameters, and obtaining the comprehensive hardware status budget value by weighted summation; S2: Based on the hardware state budget value described in step S1, select one of at least three fisheye image processing strategies, including fast enhancement, balanced enhancement, and high-quality enhancement. The different strategies correspond to different target processing time, maximum resource consumption, and expected image quality. S3: Enhance the fisheye image using the neural network branch corresponding to the selected strategy; S4: Evaluate the quality of the enhanced fisheye image and feed back the processing time and quality score to the strategy selection module.

2. The method according to claim 1, characterized in that, The calculation process of the hardware state budget value in step S1 includes the following steps: S1.1: Calculate the memory score, which is negatively correlated with memory usage; S1.2: Calculate the utilization score that is negatively correlated with GPU utilization, with reference to the safe utilization threshold of 80%-95%; S1.3: Calculate the temperature score that is negatively correlated with the current temperature, with reference to the temperature safety threshold of 80°C-90°C; S1.4: The initial budget value is calculated using a weighted summation method, with a total weight of 1; S1.5: Normalize the initial budget value to a preset range.

3. The method according to claim 2, characterized in that, The specific calculation formulas for each step are as follows: The formula for memory score is: ,in, Indicates the memory score. Indicates the current memory usage. Indicates the total amount of available memory; Utilization score formula: ,in, This represents the GPU utilization score. This indicates the current GPU utilization rate; Temperature score formula: ,in, Indicates the temperature score. Indicates the current temperature; Comprehensive budget formula: ,in, Indicates the budget value. Indicates the memory score. This represents the GPU utilization score. Indicates the temperature score; The budget value is normalized to the range [0.1, 1.0].

4. The method according to claim 1, characterized in that, The selection rule for the fisheye image enhancement strategy in step S2 is as follows: Select the rapid enhancement branch when the budget value is below the first threshold; Select the balanced enhancement branch when the budget value is between the first and second thresholds; Select the high-quality enhancement branch when the budget value is higher than the second threshold; When time constraints exist, the strategy selection is adjusted according to the time limit.

5. The method according to claim 4, characterized in that, The first threshold is 0.2-0.4; The second threshold is 0.6-0.8; The target processing time for the fast enhancement branch is 2-5ms, and the maximum memory usage is 50-150MB. The target processing time for the balanced enhancement branch is 8-15ms, and the maximum memory usage is 300-700MB. The target processing time for the high-quality enhancement branch is 20-30ms, and the maximum memory usage is 1000-2000MB.

6. The method according to claim 4, characterized in that, When the time constraint is not greater than the first time threshold, the fast enhancement branch is forced to be selected; When the time constraint is between the first time threshold and the second time threshold, the highest limit is the balanced enhancement branch; When the time constraint is greater than the second time threshold, maintain the strategy selection based on the budget value.

7. The method according to claim 1, characterized in that, The three fisheye image enhancement network branches include: fast enhancement branch, balanced enhancement branch, and high-quality enhancement branch; The fast enhancement branch employs a shallow convolutional network with depthwise separable convolutions, selectively reducing processing resolution, introducing residual connections, 3 convolutional layers, 32 channels, depthwise separable convolutions, 0.5x resolution reduction, and residual weights of 0.2-0.

4. The balanced enhancement branch adopts an encoder-decoder architecture, integrates an attention mechanism, performs multi-scale feature fusion, has 6 layers of encoder-decoder, has [64, 128, 128] channels, channel and spatial attention mechanisms, and residual weights of 0.4-0.

6. The high-quality enhancement branch adopts a deep complex network, integrates multiple enhancement modules, uses a dense connection or multi-path architecture, has a 12-layer network, has a channel number of [64, 128, 256], multi-head attention and feature refinement modules, and residual weights of 0.6-0.

8.

8. The method according to claim 1, characterized in that, The fisheye image quality assessment in step S4 is specifically implemented as follows: S4.1: Variance Assessment = ， in, Indicates the variance score. Represents the variance of a grayscale image; S4.2: Sharpness Assessment: ， in, Indicates the sharpness score. This represents the variance of the Laplace operator; S4.3: When an original reference image is available, calculate the PSNR similarity index; ， in Peak signal-to-noise ratio; S4.4: Obtain the overall quality score through weighted combination. in Represents the overall quality score. , Represents the sharpness score. This represents the PSNR score.

9. A fisheye image enhancement system for implementing the method of any one of claims 1-8, characterized in that, include: The system includes a GPU status monitoring module, a strategy selection module, a multi-branch enhanced network module, a quality assessment module, and a feedback optimization module.

10. The system according to claim 9, characterized in that: The GPU status monitoring module is implemented based on the GPUtil library or the nvidia-smi interface, with a monitoring cycle of 50-200ms. The monitored parameters include GPU memory usage, computing utilization, and temperature. The parameter update frequency is synchronized with the image acquisition frame rate. The system supports real-time processing of video streams at frame rates of 30fps and above.

Images