Software development and visual detection method and system based on de-noising enhancement recognition

Abstract

The present application relates to a software development and visual detection method and system based on denoising enhancement recognition, belonging to the technical field of computer vision and image processing. The method comprises: inputting a to-be-detected image into an end-to-end joint optimization neural network model composed of a denoising sub-network, an enhancement sub-network and a recognition sub-network, and synchronously optimizing the parameters of the three sub-networks through joint training; in the inference stage, a two-way closed-loop interaction of feeding back the recognition result to the enhancement processing and feeding back the enhancement effect to the denoising processing is realized through a linkage control unit, the targeted secondary enhancement is triggered according to the detection confidence, the differential enhancement is guided according to the noise distribution, and the denoising parameters are corrected according to the signal-to-noise ratio after enhancement. The present application also reuses historical effective strategies through a strategy memory library, accumulates difficult samples through a difficult sample library and triggers incremental training. The present application can significantly improve the visual detection accuracy and robustness of noisy low-quality images, reduce the computational overhead and realize the continuous evolution of detection performance.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, specifically to software development and visual inspection methods and systems based on denoising and enhanced recognition. Background Technology

[0002] In practical applications such as industrial visual inspection, autonomous driving environmental perception, security monitoring, and medical image analysis, the image acquisition process is inevitably affected by various noise sources, including inherent sensor noise (Gaussian noise, salt-and-pepper noise), low contrast degradation caused by changes in ambient lighting, and motion blur introduced by target movement. These noise and degradation factors severely degrade image quality, posing a significant challenge to the accuracy and reliability of subsequent target detection and recognition tasks. To address this issue, existing technologies typically employ a step-by-step processing approach: first, the acquired raw image undergoes denoising preprocessing to suppress noise; then, image enhancement algorithms are used to improve contrast and detail visibility; finally, the enhanced image is input into the target detection or recognition model for inference. However, this serialized, step-by-step processing paradigm has inherent flaws: the denoising process inevitably loses high-frequency details such as edges and textures in the image while suppressing noise, and these details are crucial for subsequent recognition tasks; the enhancement process may synchronously amplify the noise components remaining after denoising, leading to artifacts or noise prominence in the enhanced image; the optimization objectives of each step are independent of each other, with the denoising module aiming to minimize reconstruction error, the enhancement module aiming to improve subjective visual effects, and the recognition module aiming to maximize detection accuracy. The three lack global coordination, making it difficult to achieve optimal performance for the overall task. Existing research has shown that jointly training the denoising model and the object detection model can utilize the rich feature information in the object detection model to guide the training of the denoising model, making the denoised image better suited for subsequent detection tasks. For example, related technologies have proposed a scheme where a noisy image is input into the denoising model to obtain a denoised image, and then the denoised image is input into the object detection model, and the two models are jointly trained based on the denoised image, the detection results, and the corresponding noise-free image and labels. In addition, some studies have combined image quality enhancement with object detection tasks, using reinforcement learning to restore the input image in order to improve detector performance.

[0003] However, the existing technologies still have the following shortcomings. First, existing joint training schemes mainly focus on the two-level joint optimization of the denoising model and the object detection model, lacking an independent enhancement sub-network. Image enhancement processing is still treated as an independent preprocessing step outside of model training, failing to achieve end-to-end joint optimization of the three stages of denoising, enhancement, and recognition. This results in the enhancement stage being unable to perceive the needs of the recognition task, and the denoising stage being unable to adaptively adjust based on the enhancement effect. Second, existing schemes use a fixed forward propagation path in the inference stage, with denoising, enhancement (if any), and recognition forming a strictly sequential pipeline. The recognition results cannot be fed back to the previous modules for targeted optimization and adjustment. When the detection confidence is low or there are difficult samples in the image, the system lacks proactive secondary processing and dynamic adjustment capabilities. Third, denoising processing in existing technologies typically uses a globally uniform filtering strategy, failing to perform differentiated adaptive processing based on the noise type and intensity differences in different regions of the image. It also lacks a feedback correction mechanism for the enhancement effect on denoising parameters, making it difficult to achieve a dynamic balance between noise suppression and detail preservation. Fourth, the existing system lacks a mechanism for learning and reusing historical processing experience. When faced with similar scenarios, the same computational processing needs to be repeated, and the inference efficiency needs to be improved. At the same time, for low-confidence difficult samples encountered in actual deployment, there is a lack of systematic accumulation and targeted incremental training schemes, making it difficult for the model to continuously adapt to changes in data distribution in real-world scenarios. Summary of the Invention

[0004] To address the problems of existing technologies, this invention provides a software development and visual inspection method and system based on denoising and enhanced recognition.

[0005] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution: Firstly, a software development and visual inspection method based on denoising and enhanced recognition, comprising: Obtain the original image to be detected; The original image is input into a pre-trained end-to-end joint optimization neural network model, which includes a cascaded denoising subnetwork, an enhancement subnetwork, and a recognition subnetwork. The original image is subjected to noise suppression processing through the denoising sub-network, and the noise-suppressed image and noise distribution information are output. The noise distribution information includes noise type, noise intensity level and noise spatial distribution. The image enhancement process is performed on the noise-suppressed image through the enhancement sub-network, and the enhanced image is output. The enhanced image is processed by the recognition sub-network to perform target recognition processing, and the visual detection results and the confidence level of each detected target are output. Obtain a first target region with a confidence level lower than a preset confidence threshold, wherein the preset confidence threshold is selected from any value between 0.5 and 0.75; Based on the corresponding position of the first target region in the original image, the enhancement sub-network is controlled to perform targeted secondary enhancement processing on the corresponding position to generate a secondary enhanced image; The enhanced image is then input into the recognition subnetwork for secondary recognition, generating a secondary recognition result and a secondary confidence level. The confidence level and the secondary confidence level are fused for determination: when the increase in the secondary confidence level relative to the confidence level exceeds a preset increase threshold, the secondary recognition result is used as the final detection result of the first target region; when the increase does not exceed the preset increase threshold, the weighted fusion result of the recognition result corresponding to the confidence level and the secondary recognition result is used as the final detection result of the first target region, wherein the weighting coefficient of the weighted fusion is directly proportional to the increase, and the value range of the weighting coefficient is 0.5 to 1.0; The denoising subnetwork, the enhancement subnetwork, and the recognition subnetwork are jointly trained in an end-to-end manner, and the loss function of the joint training includes a weighted combination of denoising loss components, enhancement loss components, and recognition loss components.

[0006] In one specific implementation of the first aspect, the loss function for joint training is constructed in the following manner: Obtain annotation data corresponding to the original image, the annotation data including a noise-free reference image, an enhanced reference image, and recognition annotation results; Based on the mean square error between the output of the denoising sub-network and the noise-free reference image, the denoising loss component L_d is calculated. The enhancement loss component L_e is calculated based on a weighted combination of the structural similarity index and mean square error between the output of the enhancement subnetwork and the enhancement reference image. The recognition loss component L_r is calculated based on a weighted combination of the cross-entropy loss and bounding box regression loss between the output of the recognition subnetwork and the recognition annotation result. The overall loss function for the joint training is L_total=α(t)·L_d+β(t)·L_e+γ(t)·L_r, where α(t), β(t), and γ(t) are the dynamic weight coefficients of the denoising loss component, the enhancement loss component, and the recognition loss component, respectively, and t is the number of training iterations. The dynamic weight coefficients are adjusted according to the following strategy: when the number of training steps t is less than a first preset step threshold, α(t) > β(t) and α(t) > γ(t); when the number of training steps t is greater than or equal to the first preset step threshold and less than a second preset step threshold, β(t) gradually increases to β(t) > α(t) and β(t) > γ(t); when the number of training steps t is greater than or equal to the second preset step threshold, γ(t) gradually increases to γ(t) > α(t) and γ(t) > β(t); the first preset step threshold is 30% to 40% of the total number of training steps, and the second preset step threshold is 65% to 75% of the total number of training steps.

[0007] In one specific implementation of the first aspect, the training of the end-to-end jointly optimized neural network model employs a strategy combining staged pre-training and end-to-end global fine-tuning: In the first stage, the denoising sub-network is pre-trained using a denoising training dataset. The denoising training dataset contains multiple pairs of noise-clean images with different noise types and different noise intensity levels. The noise types include Gaussian noise, salt-and-pepper noise, and Poisson noise. The noise intensity level is measured by the noise standard deviation σ, and the value of σ ranges from 5 to 50. In the second stage, the augmentation subnetwork is pre-trained using an augmentation training dataset. The augmentation training dataset contains multiple sets of image pairs of low-quality images and corresponding high-quality reference images. The low-quality images are images acquired under illumination conditions of less than 100 lux or images with a contrast of less than 0.3. In the third stage, the recognition sub-network is pre-trained using a recognition training dataset, which contains detection images with labeled target categories and bounding boxes. In the fourth stage, the pre-trained denoising sub-network, the enhancement sub-network, and the recognition sub-network are cascaded in sequence, and the whole network is fine-tuned in an end-to-end manner using a joint training dataset. Each training image in the joint training dataset simultaneously contains the corresponding noiseless reference image, the enhancement reference image, and the recognition annotation result.

[0008] In one specific implementation of the first aspect, controlling the enhancement subnetwork to perform targeted secondary enhancement processing on the corresponding position of the first target region in the original image specifically includes: Extract the position coordinates and size information of the first target region in the original image; Crop the feature regions corresponding to the position coordinates and size information from the feature map of the enhanced subnetwork; The feature region is subjected to a secondary enhancement process using enhancement intensity parameters higher than those in the initial enhancement process. These higher enhancement intensity parameters include: increasing the contrast stretching ratio from the first contrast coefficient of the initial enhancement to a second contrast coefficient, where the first contrast coefficient ranges from 1.1 to 1.5 and the second contrast coefficient ranges from 1.6 to 2.5; and / or increasing the high-frequency component enhancement gain from the first gain value of the initial enhancement to a second gain value, where the first gain value ranges from 1.0 to 1.8 and the second gain value ranges from 2.0 to 4.0. The feature regions after secondary enhancement are fused back to their corresponding positions in the feature map of the enhanced sub-network to generate the secondary enhanced image.

[0009] In one specific embodiment of the first aspect, the denoising sub-network generates a noise intensity distribution map of each pixel position in the original image when performing noise suppression processing on the original image; The noise intensity distribution map is transmitted to the enhancement subnetwork; The enhancement sub-network determines the enhancement intensity coefficient for each pixel location based on the noise intensity distribution map. For locations where the noise intensity level is higher than a first preset noise level threshold, a first enhancement intensity coefficient is used, with a value ranging from 0.1 to 0.3. For locations where the noise intensity level is lower than a second preset noise level threshold, a second enhancement intensity coefficient is used, with a value ranging from 0.5 to 0.9, and the second enhancement intensity coefficient is greater than the first enhancement intensity coefficient. For locations where the noise intensity level is between the first preset noise level threshold and the second preset noise level threshold, the enhancement intensity coefficient is determined using a linear interpolation method. The enhancement subnetwork performs pixel-by-pixel differential enhancement processing on the noise-suppressed image based on the enhancement intensity coefficient.

[0010] In one specific implementation of the first aspect, after the enhancement subnetwork completes the enhancement processing of the noise-suppressed image, it calculates the signal-to-noise ratio and contrast of the enhanced image; The signal-to-noise ratio of the enhanced image is compared with a preset signal-to-noise ratio threshold, wherein the preset signal-to-noise ratio threshold ranges from 20dB to 30dB. When the signal-to-noise ratio of the target region in the enhanced image is lower than the preset signal-to-noise ratio threshold, the target region is marked as a noise amplification region; Obtain the noise intensity level corresponding to the noise amplification region in the noise intensity distribution map. If the noise intensity level is lower than the third preset noise level threshold, it is determined that the noise amplification in the noise amplification region is introduced by the enhancement process, and a noise reduction parameter correction instruction is generated. The denoising sub-network re-denoises the positions in the original image corresponding to the noise amplification region according to the denoising parameter correction instructions. The re-denoising process uses stronger smoothing parameters than the initial denoising process. The stronger smoothing parameters include: increasing the filter kernel size from a first size to a second size, where the first size ranges from 3×3 to 5×5 and the second size ranges from 7×7 to 11×11; and / or increasing the regularization coefficient from a first regularization value to a second regularization value, where the first regularization value ranges from 0.001 to 0.01 and the second regularization value ranges from 0.02 to 0.1. The image after re-denoising is input again into the enhancement subnetwork and the recognition subnetwork for enhancement and recognition.

[0011] In one specific implementation of the first aspect, a strategy memory and a hard sample library are also included; The strategy memory is used to store enhancement strategies for different scene features in historical processing and their corresponding effect scores. The scene features include at least two of the following: average image brightness, image contrast, main noise type, and detection target category. When the confidence level output by the recognition sub-network is lower than the preset confidence threshold, the scene features of the current image are extracted, and historical records with a similarity exceeding a preset similarity threshold are matched in the policy memory. The preset similarity threshold ranges from 0.75 to 0.90. If the match is successful, the enhancement strategy corresponding to the historical record is directly used to perform secondary enhancement processing on the first target region. If the match fails, an enhancement strategy is generated online according to the method described in claim 4 for secondary enhancement processing. After the enhancement processing is completed, the current scene features, the enhancement strategy adopted, and the increase in confidence level relative to the confidence level are stored as a new record in the policy memory. The effect score of the new record is the ratio of the increase in confidence level to the processing time. When the confidence score output by the recognition subnetwork is lower than a preset difficulty threshold, the corresponding target region is marked as a difficult sample region. The preset difficulty threshold ranges from 0.3 to 0.5. The location coordinates of the difficult sample region are extracted, and multiple rounds of iterative processing are performed on the difficult sample region. Each round of iterative processing includes: the denoising subnetwork and the enhancement subnetwork respectively use different parameter configurations to denoise and enhance the difficult sample region, and then the recognition subnetwork performs recognition to obtain the recognition result and confidence score of each round of iteration. Different parameters are used in different rounds of iteration. The configuration is as follows: the first round uses a parameter combination of low denoising intensity and high enhancement intensity; the second round uses a parameter combination of high denoising intensity and low enhancement intensity; and the third round uses a parameter combination of medium denoising intensity and medium enhancement intensity. The low denoising intensity corresponds to a filter kernel size of 3×3 to 5×5, the medium denoising intensity corresponds to a filter kernel size of 7×7 to 9×9, and the high denoising intensity corresponds to a filter kernel size of 11×11 to 15×15. The recognition results of each round of iteration are fused according to confidence level, with higher confidence levels assigned higher fusion weights, and the final detection result of the difficult sample region is output. The original image blocks of the difficult sample region, the parameter configuration of each round, and the recognition results of each round are stored in the difficult sample library. When the number of difficult samples of the same category in the difficult sample library exceeds a preset number threshold, the incremental training of the end-to-end joint optimization neural network model is triggered. The preset number threshold ranges from 500 to 2000 images.

[0012] Secondly, a visual inspection system based on end-to-end joint optimization includes: The image acquisition module is used to acquire the original image to be detected; An end-to-end jointly optimized neural network model is stored in a computer-readable storage medium, including cascaded denoising subnetworks, enhancement subnetworks, and recognition subnetworks; A processor that implements a software development and visual inspection method based on denoising enhancement recognition when executing the end-to-end joint optimization neural network model; The output module is used to output the visual inspection results; The strategy memory is used to store the mapping relationship between scene features and enhancement strategies, as well as the corresponding effect scores. The difficult sample library storage is used to store image blocks of difficult sample regions and processing records for each round.

[0013] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements a software development and visual inspection method based on denoising enhancement recognition.

[0014] Fourthly, a computer-readable storage medium stores a computer program that, when executed by a processor, implements a software development and visual inspection method based on denoising enhancement recognition.

[0015] The beneficial effects of this invention are as follows: 1. This invention constructs cascaded denoising, enhancement, and recognition sub-networks and employs an end-to-end joint training using a weighted overall loss function that includes denoising, enhancement, and recognition loss components. This ensures that the parameters of the three sub-networks are updated synchronously under a unified optimization objective, avoiding the cumulative error and feature mismatch problems caused by the independent optimization objectives of each step in traditional step-by-step processing. Furthermore, this invention uses a strategy combining phased pre-training with end-to-end overall fine-tuning during training, dynamically adjusting the weight coefficients of each loss component based on the number of training iterations. This ensures that the denoising, enhancement, and recognition sub-networks converge sequentially and preferentially, effectively solving the technical challenges of gradient competition and uneven convergence in multi-task joint training. This significantly improves the overall detection accuracy and robustness of the model on noisy, low-quality images. In addition, the denoising subnetwork synchronously outputs noise distribution information containing noise type, noise intensity level and noise spatial distribution during the noise suppression process. This information is passed to the enhancement subnetwork to guide pixel-by-pixel differential enhancement, enabling the enhancement process to adaptively adjust the enhancement intensity according to the noise level of each region. This achieves a dynamic balance between noise suppression and detail preservation, avoiding the technical defects of traditional uniform enhancement methods that over-amplify noise in noisy regions or under-enhance in low-noise regions. 2. In the inference phase, this invention establishes a two-way closed-loop interactive mechanism by setting up linkage control units between each sub-network, which feeds back the recognition result to the enhancement processing and the enhancement effect to the denoising processing. When the detection confidence output by the recognition sub-network is lower than a preset threshold, the linkage control unit triggers targeted secondary enhancement processing, performs local secondary enhancement on the low-confidence target area, and re-identifies it, effectively improving the detection recall rate of difficult samples. At the same time, by monitoring the signal-to-noise ratio of the enhanced image and marking areas with a signal-to-noise ratio lower than the threshold as noise amplification areas, the linkage control unit generates denoising parameter correction instructions and feeds them back to the denoising sub-network for re-denoising processing, effectively suppressing the amplification effect of the enhancement processing on residual noise, forming a virtuous cycle of mutual constraint and optimization between denoising and enhancement. Furthermore, this invention stores the mapping relationship between scene features and enhancement strategies in historical processing, along with corresponding effect scores, through a strategy memory. When encountering similar scenes in subsequent inference, historically effective strategies can be directly reused, significantly reducing computational overhead and improving inference efficiency while maintaining processing effectiveness. It also accumulates low-confidence samples and their multi-round iterative processing records through a difficult sample library. Incremental training is automatically triggered when the number of similar difficult samples reaches a preset threshold, enabling the model to continuously adapt to changes in data distribution in real-world scenarios and achieving self-evolution of detection performance. This invention further enhances the detection reliability of extremely difficult samples through parallel exploration with different parameter configuration combinations and confidence-weighted fusion decision-making in multi-round iterative processing, synergistically improving the adaptability, accuracy, and robustness of the visual detection system in complex noisy environments from multiple levels. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the end-to-end joint optimization neural network model structure of the present invention.

[0017] Figure 2 This is a schematic diagram of the dynamic linkage processing flow during the reasoning stage of this invention.

[0018] Figure 3 This is a schematic diagram of the visual inspection system module architecture of the present invention. Detailed Implementation

[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] like Figures 1 to 3 The software development and visual inspection method and system based on denoising and enhanced recognition are shown.

[0021] (a) Structure of end-to-end joint optimization neural network model like Figure 1 As shown, the end-to-end joint optimization neural network model provided by this invention consists of cascaded denoising subnetworks, enhancement subnetworks, and recognition subnetworks. The specific structure of each subnetwork is as follows: 1. Noise reduction subnetwork The denoising sub-network employs a deep convolutional neural network based on the U-Net architecture. Specifically: The denoising subnetwork consists of an encoder and a decoder. The encoder contains 4 to 6 downsampled convolutional modules, each consisting of a convolutional layer, a batch normalization layer, and a ReLU activation function layer connected in sequence. Each downsampled convolutional module has a 3×3 kernel size, a stride of 2, and the number of channels increases layer by layer, from 32 or 64 channels in the first layer to 256 or 512 channels in the last layer. Each downsampled convolutional module halves the feature map size and doubles the number of channels through a convolution operation with a stride of 2.

[0022] The decoder section contains the same number of upsampling convolutional modules as the encoder section. Each upsampling convolutional module consists of an upsampling layer, a convolutional layer, a batch normalization layer, and a ReLU activation function layer connected in sequence. The upsampling layer is implemented using bilinear interpolation or transposed convolution, restoring the feature map size to half of its original size while halving the number of channels.

[0023] Skip connections are established between corresponding layers of the encoder and decoder. These skip connections concatenate the feature maps output from each layer of the encoder with the upsampled feature maps from the corresponding layers of the decoder along the channel dimension. This allows the decoder to utilize both shallow detail information and deep semantic information simultaneously. The size and number of channels of the feature maps processed by corresponding units in the encoder and decoder are kept identical through these skip connections.

[0024] The denoising subnetwork also includes a noise estimation branch and a noise type identification branch: Noise estimation branch: Connected after the last layer of the encoder, it consists of two parallel convolutional layers, used to estimate the noise intensity (characterized by the noise standard deviation σ) at each pixel location in the image and the spatial distribution of the noise, respectively. The noise estimation branch outputs a noise intensity distribution map of the same size as the input image, where the value at each pixel location represents the estimated standard deviation of the noise at that location, ranging from 0 to 50.

[0025] Noise type identification branch: Connected after the intermediate layer of the encoder, this branch consists of a global average pooling layer and a fully connected layer, and is used to identify the noise type of the input image. Noise types include Gaussian noise, salt-and-pepper noise, Poisson noise, motion blur noise, and combinations thereof. The fully connected layer outputs a 4-dimensional or 5-dimensional probability vector, where each dimension corresponds to the probability of a noise type, and the sum of the probability values ​​is 1.

[0026] 2. Enhanced Subnetwork The enhanced subnetwork adopts an enhanced architecture based on generative adversarial networks. Specifically: The augmented subnetwork includes a generator and a discriminator.

[0027] The generator employs a U-Net architecture to map the denoised image to the enhanced image. The encoder portion of the generator contains 5 to 7 downsampling convolutional modules, and the decoder portion contains the same number of upsampling convolutional modules. Skip connections are established between corresponding layers in the encoder and decoder. The generator's input is the denoised image (H×W×3 in size, where H is the image height and W is the image width) output from the denoising subnetwork, and the output is the enhanced image (with the same size as the input).

[0028] The generator also includes a contrast enhancement module, a detail enhancement module, and a color correction module, which are embedded in parallel or series at the end of the generator's decoder. Contrast enhancement module: Employs adaptive histogram equalization or contrast stretching algorithms. The mapping function for contrast stretching is: ,in and These are the minimum and maximum gray values ​​of the input image, respectively. and These are the target lowest and highest grayscale values ​​of the output image, respectively. The contrast stretching ratio is determined by the contrast coefficient. control, The value range is from 1.1 to 2.5.

[0029] Detail enhancement module: Employs a high-frequency component enhancement method. First, the image is decomposed into low-frequency and high-frequency components using a low-pass filter (such as a Gaussian filter). Then, the high-frequency component is multiplied by a gain coefficient. It is then resynthesized with the low-frequency components. Gain coefficient. The value range is from 1.0 to 4.0.

[0030] Color correction module: Employs a color balance algorithm to independently adjust the gain of each of the three RGB channels, ensuring that the mean values ​​of each channel are consistent and eliminating color cast. The color correction coefficient is determined based on the ratio of the mean value of each channel to the target mean value.

[0031] The discriminator employs a PatchGAN architecture, consisting of 5 to 7 stacked convolutional layers, each followed by a batch normalization layer and a LeakyReLU activation function. The input to the discriminator is either the enhanced image output by the generator or a real, high-quality reference image. The output is an N×N discrimination matrix (N ranging from 16 to 70), where each element represents the discrimination probability of the corresponding image patch (ranging from 0 to 1, with values ​​closer to 1 indicating greater realism), used to determine whether the input image is a real image or a generated enhanced image. The generator's goal is to minimize the discriminator's accuracy in classifying its generated images, thus ensuring that the discriminator classifies its generated enhanced images as real images as much as possible.

[0032] 3. Identify subnetworks The recognition sub-network employs a deep learning-based object detection network. Preferably, the recognition sub-network is a Faster R-CNN object detection network. Specifically: Faster R-CNN consists of a feature extraction backbone network, a region proposal network (RPN), region of interest pooling layers, a classification branch, and a regression branch.

[0033] The feature extraction backbone network uses ResNet-50 or ResNet-101 to extract high-level semantic features from the enhanced image. The input to the backbone network is the enhanced image (H×W×3) output from the enhancement sub-network, and the output is a multi-scale feature map. ResNet-50 contains 50 convolutional layers divided into 5 stages, with the output feature map sizes of each stage being 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image, respectively.

[0034] The Region Proposal Network (RPN), connected after the fourth stage output of the backbone network, is used to generate candidate target regions. The RPN pre-defines multiple anchor boxes of different scales and aspect ratios at each location in the feature map. The anchor box scales are 8, 16, 32, 64, and 128 pixels, and the aspect ratios are 0.5, 1.0, and 2.0. The RPN outputs the objectness score and bounding box offset for each anchor box through two parallel 1×1 convolutional layers. Based on the objectness score, the RPN selects the top-N candidate regions (N ranges from 300 to 2000, preferably 1000).

[0035] Region of Interest (ROI) pooling layers are used to pool candidate regions of different sizes into feature maps of a fixed size (7×7 or 14×14), which facilitates subsequent classification and regression.

[0036] The classification branch is a fully connected layer (containing 1024 or 2048 neurons), which outputs the probability of the candidate region belonging to each category (a softmax normalized probability vector, with the sum of each dimension being 1).

[0037] The regression branch is a fully connected layer (containing 1024 or 2048 neurons), which outputs the fine adjustment of the bounding box of the candidate region (a 4-dimensional vector, corresponding to the x-coordinate offset, y-coordinate offset, width scaling and height scaling of the center point, respectively).

[0038] The visual detection results output by the recognition subnetwork include: category information for each detected target (such as defect categories such as "scratches", "dents", "bubbles", etc.), location information (represented by the coordinates of the upper left and lower right corners of the bounding box), and confidence (values ​​ranging from 0 to 1).

[0039] As an alternative, the recognition subnetwork can also use a single-stage object detection network such as the YOLO series (e.g., YOLOv5, YOLOv8) or SSD.

[0040] (ii) End-to-end joint training method 1. Training dataset construction Joint training dataset ,in The input is a noisy, low-quality image. The corresponding noise-free reference image (clean image). For the corresponding enhanced reference image (high-quality reference image). To identify the annotation results (including target category labels and bounding box coordinates), N is the total number of training samples, and the value of N ranges from 10,000 to 100,000.

[0041] The joint training dataset is constructed in the following way: (1) Basic image acquisition: High-quality original images are acquired under various lighting conditions (illuminance range of 10 lux to 10000 lux) and various shooting conditions, with an image resolution of not less than 640×480 pixels.

[0042] (2) Noise Addition: Different types and intensities of noise are added to the acquired high-quality original images to generate noisy images. Noise types include Gaussian noise (standard deviation σ = 5 to 50), salt-and-pepper noise (noise density 0.01 to 0.3), Poisson noise (λ = 10 to 100), and combinations thereof. Each noise type generates at least 5 noisy images with different intensities.

[0043] (3) Quality degradation: Further quality degradation processing is performed on some noisy images, including reducing contrast (contrast reduced to below 0.3) and reducing brightness (brightness reduced to below 30% of the original brightness), to simulate low-quality images.

[0044] (4) Labeling: All targets in the images are labeled by professionals. The labeling content includes target category labels and bounding box coordinates.

[0045] 2. Phased pre-training A training strategy combining phased pre-training and end-to-end global fine-tuning is adopted: Phase 1: Pre-training of the noise reduction network Using denoised training dataset The denoising sub-network is pre-trained. The denoising training dataset contains multiple pairs of noise-clean images with different noise types and noise intensity levels.

[0046] The pre-training loss function of the denoising sub-network is:

[0047] in This represents the forward mapping function of the denoising subnetwork. The number of samples in the denoising training dataset. This represents the square of the L2 norm (mean square error).

[0048] Pre-training uses the Adam optimizer with an initial learning rate of to Batch size ranges from 8 to 32, and training epochs range from 50 to 200. The learning rate employs a cosine annealing scheduling strategy, decaying every 10 epochs with a decay coefficient of 0.9 to 0.95.

[0049] Phase 2: Enhanced Subnetwork Pre-training Using augmented training datasets

[0050] The enhancement subnetwork is pre-trained. The enhancement training dataset contains multiple pairs of low-quality images and corresponding high-quality reference images. Low-quality images are those acquired under illumination conditions below 100 lux or with a contrast below 0.3.

[0051] The pre-training loss function for the augmented subnetwork is a weighted combination of generative adversarial loss and content loss:

[0052] Among them, generating adversarial loss Content loss To enhance the weighted combination of L1 loss and Structural Similarity Index (SSIM) loss between the image and the reference image.

[0053] The value range is from 0.1 to 1.0. The value range is from 1.0 to 10.0. Preferably, Pre-training uses the Adam optimizer with an initial learning rate of... to The batch size is 4 to 16, and the number of training rounds is 100 to 300.

[0054] Phase 3: Subnetwork Pre-training Using the training dataset for identification The recognition subnetwork is pre-trained. The recognition training dataset contains detection images with labeled target categories and bounding boxes, with no fewer than 10,000 images, and each image contains at least one labeled target.

[0055] The pre-training loss function for the recognition subnetwork is:

[0056] Classification loss Cross-entropy loss, regression loss This is the Smooth L1 loss.

[0057] Pre-training uses a momentum-driven stochastic gradient descent (SGD) optimizer with an initial learning rate of... to The momentum is 0.9, and the weight decay is... The batch size is 8 to 32, and the number of training rounds is 50 to 150.

[0058] Phase 4: End-to-end overall fine-tuning The pre-trained denoising subnetwork, enhancement subnetwork, and recognition subnetwork are cascaded sequentially, and the overall fine-tuning is performed in an end-to-end manner using the joint training dataset.

[0059] The overall loss function for joint training is:

[0060] in: For the denoising loss component, based on the output of the denoising sub-network With noise-free reference image Calculation of mean square error between them:

[0061] To enhance the loss component, the output of the enhancement subnetwork is used. With enhanced reference image The weighted combination of the structural similarity index (SSIM) and mean squared error between the two is calculated as follows:

[0062] in The value ranges from 0.5 to 2.0. The value ranges from 0.5 to 2.0. Preferably, .

[0063] To identify the loss components, the output of the identification subnetwork is used. Compared with the identification and annotation results The weighted combination of cross-entropy loss (classification) and Smooth L1 loss (regression) is calculated.

[0064] , respectively, are the dynamic weight coefficients of the denoising loss component, the enhancement loss component, and the recognition loss component, and t is the number of training iterations.

[0065] The dynamic weighting coefficients are adjusted according to the following three-stage strategy: Phase 1 (Initial Training) ):set up This allows the denoising subnetwork to converge preferentially. Preferably, .

[0066] Phase Two (Mid-Training) Gradually increase This causes the enhanced subnetwork to gradually converge. It increases linearly from 0.3 to 1.0. It decreases linearly from 1.0 to 0.5. It increases linearly from 0.1 to 0.5.

[0067] Phase Three (Late Training) Gradually increase This causes the recognition subnetwork to gradually converge. It increases linearly from 0.5 to 1.0. It decreases linearly from 0.5 to 0.2. It decreases linearly from 1.0 to 0.5.

[0068] in 30% to 40% of the total training steps This represents 65% to 75% of the total training steps. The total training steps are determined based on the size of the training dataset, typically ranging from 50,000 to 200,000 steps.

[0069] End-to-end fine-tuning uses the Adam optimizer with an initial learning rate of The batch size ranges from 4 to 16, and the number of training epochs ranges from 20 to 100. The learning rate employs a phased decay strategy, decreasing to 0.8 to 0.9 times the current rate every 10 epochs.

[0070] 3. Data augmentation during training During training, online data augmentation is performed on the input images: Geometric transformations: random cropping (cropping ratio from 0.8 to 1.0), random horizontal flipping (probability of 0.5), random rotation (angle range from -15° to +15°), and random scaling (scaling ratio from 0.8 to 1.2).

[0071] Color transformation: Random brightness adjustment (adjustment range ±20%), random contrast adjustment (adjustment range ±20%), random saturation adjustment (adjustment range ±20%).

[0072] Noise injection: Inject additional small noise (Gaussian noise, σ=1 to 5) into the training images with a probability of 0.1 to 0.3 to improve the robustness of the model.

[0073] (III) Dynamic Linkage Processing in the Reasoning Stage like Figure 2 As shown, during the inference phase, the denoising subnetwork, the enhancement subnetwork, and the recognition subnetwork achieve dynamic bidirectional interaction through a linkage control unit.

[0074] 1. Initial forward reasoning The original image to be detected (Dimensions: H×W×3, RGB three-channel) Input end-to-end joint optimization neural network model: (1) The denoising sub-network performs noise suppression processing on the original image and outputs a denoised image. and noise intensity distribution map . Size and Same (H×W×3). The size is H×W, and the value at each pixel position is... This represents the estimated standard deviation of the noise at that location.

[0075] (2) The enhancement subnetwork enhances the denoised image and outputs the enhanced image. . The dimensions are H×W×3.

[0076] (3) The recognition sub-network performs target recognition processing on the enhanced image and outputs the visual detection results. ,in For the first The category label of each detected target, These are the coordinates of the bounding box (top left and bottom right corners). Here, K represents the confidence level, and K represents the number of targets detected.

[0077] 2. Confidence-driven secondary enhancement The linkage control unit acquires the confidence level of each detected target output by the recognition subnetwork. When it exists hour( To set a pre-defined confidence threshold (selected from any value between 0.5 and 0.75), the corresponding target region is marked as the first target region.

[0078] For each primary target region, perform the following steps: Step 1: Extract the bounding box coordinates of the first target region Calculate the coordinates and dimensions of the region's center:

[0079] Step 2: Crop bounding boxes from the feature maps of the enhanced subnetwork Corresponding feature regions The feature map size is ,in R is the downsampling factor (r can be 4, 8, or 16), and C is the number of channels (C can be 256 or 512). The coordinates of the cropped area are: , , , ,

[0080] Step 3: Perform secondary enhancement processing on the trimmed feature areas using higher enhancement strength parameters: Contrast stretching ratio from the first contrast coefficient (Value range is 1.1 to 1.5) Increased to the second contrast coefficient (Values ​​range from 1.6 to 2.5); High-frequency component enhancement gain from the first gain value (Value range is 1.0 to 1.8) Increase to the second gain value (Value range is 2.0 to 4.0).

[0081] Feature regions after secondary enhancement processing The pixel value is calculated as follows:

[0082] in Low-frequency components, These are high-frequency components.

[0083] Step 4: Enhance the feature regions after secondary enhancement. The corresponding positions of the feature maps from the augmented subnetwork are fused back to generate a secondary augmented image. .

[0084] Step 5: Enhance the image in two stages. The input is then fed back into the recognition subnetwork for secondary recognition, generating a secondary recognition result. .

[0085] Step Six: Fusion Determination. When hour( If a preset threshold is set (ranging from 0.05 to 0.15), the result of the secondary recognition is used as the final detection result; otherwise, the weighted fusion result of the primary and secondary recognition results is used as the final detection result.

[0086] Among them, the weighting coefficient and the magnitude of improvement They are directly proportional:

[0087] The value range is from 0.5 to 1.0.

[0088] 3. Noise distribution-driven differential enhancement Noise intensity distribution of the output of the denoising sub-network It is passed to the reinforcement subnetwork. The reinforcement subnetwork is based on... Determine the enhancement intensity coefficient at each pixel location : for The location (high noise area) uses the first enhancement intensity coefficient. That is, low enhancement intensity, to avoid amplifying noise; for The location (low noise area) adopts the second enhancement intensity coefficient. This means high enhancement intensity, which significantly improves image quality; for The location and strength coefficient are determined by linear interpolation.

[0089] in and To preset the noise level threshold, The value range is from 5 to 15. The value range is from 25 to 40.

[0090] The reinforcement subnetwork is based on the reinforcement strength coefficient. Perform pixel-by-pixel differential enhancement on the denoised image:

[0091] in and These represent the high-frequency and low-frequency components of the denoised image, respectively.

[0092] 4. Enhanced feedback correction and noise reduction After the enhancement subnetwork completes the enhancement process, the linkage control unit calculates the signal-to-noise ratio (SNR) and contrast of the enhanced image.

[0093] The signal-to-noise ratio is calculated as follows:

[0094] in The pixel mean of the image signal region. This represents the pixel standard deviation of the noisy region in the image. For the entire image, Take the average of all pixels in the image. Calculate the standard deviation of all pixels in the image.

[0095] The contrast ratio is calculated as follows:

[0096] in and These are the maximum and minimum gray values ​​of the image, respectively.

[0097] When a target region (bounding box) is in the image after enhancement processing The signal-to-noise ratio (SNR) of the corresponding region is lower than the preset SNR threshold. When the value ranges from 20dB to 30dB, the target area is marked as a noise amplification area.

[0098] The linkage control unit obtains the noise intensity distribution map of the noise amplification area. The corresponding noise level. If the noise level in this area is lower than the third preset noise level threshold. If the value ranges from 10 to 20, it is determined that the noise amplification is introduced by the enhancement process (rather than the high noise of the original image itself), and a noise reduction parameter correction instruction is generated.

[0099] The denoising sub-network, based on the denoising parameter correction instructions, re-denoises the locations in the original image corresponding to the amplified noise areas. This re-denoising process uses stronger smoothing parameters than the initial denoising process. Filter kernel size from the first size ( The value ranges from 3 to 5) and increases to the second size. ( The value range is 7 to 11). The regularization coefficient starts from the first regularization value. (Value range is 0.001 to 0.01) Increase to the second regularization value (The value ranges from 0.02 to 0.1).

[0100] The image, after being re-denoised, is then fed back into the enhancement and recognition subnetworks for enhancement and recognition.

[0101] 5. Strategic memorization and reuse The system has a strategy memory. It is used to store enhancement strategies for different scene features and corresponding effect scores in historical processing.

[0102] Scene feature vector Includes at least two of the following five characteristics: Average image brightness The value ranges from 0 to 255; Image contrast The value ranges from 0 to 1; Main noise types (Gaussian, salt and pepper, Poisson, motion blur). Detection target categories (such as "scratches", "dents", "parts" etc.); Image resolution (width × height).

[0103] Each record in the strategy memory Includes: Scene feature vectors Enhance strategy parameters (including contrast ratio) and gain coefficient ), confidence level increase Processing time .

[0104] Recorded effect rating (Unit: confidence increase / second). The policy memory sorts the records in descending order according to their performance scores.

[0105] When the confidence level of the identification subnetwork output is lower than the preset confidence threshold At that time, the linkage control unit extracts the scene feature vector of the current image. And match historical records in the policy memory:

[0106] when hour( If a match is found to be successful, the corresponding enhancement strategy from the historical records will be directly used for secondary enhancement. (A preset similarity threshold is used, with a value ranging from 0.75 to 0.90.)

[0107] If a match fails, an enhancement strategy is generated online using a confidence-driven secondary enhancement method for secondary enhancement processing. After processing, the current scene features, the enhancement strategy adopted, and the confidence increase are stored as a new record in the strategy memory.

[0108] When the number of records in the policy memory exceeds the preset storage limit (e.g., 1,000 to 5,000 records), the record with the lowest performance score is deleted.

[0109] 6. Iterative processing of difficult samples When the confidence level of the identification subnetwork output Below the preset difficulty threshold When the value ranges from 0.3 to 0.5, the corresponding target region is marked as a difficult sample region.

[0110] The linkage control unit extracts the location coordinates of the difficult sample region and performs multiple rounds of iterative processing on the difficult sample region. Each round of iterative processing includes: the denoising subnetwork and the enhancement subnetwork use different parameter configurations to denoise and enhance the difficult sample region, and then the recognition subnetwork performs recognition to obtain the recognition result and confidence level of each round of iteration.

[0111] The three rounds of iterations used different combinations of parameter configurations: ; After three rounds of iteration, the identification results from each round are weighted and fused according to confidence level:

[0112] That is, the higher the confidence level of the recognition result, the higher the fusion weight is assigned (using the confidence level itself as the weight).

[0113] The original image patches of the difficult sample regions, the parameter configurations for each round, and the recognition results for each round are stored in the difficult sample database. When the number of difficult samples of the same category in the difficult sample database exceeds a preset threshold... When the number of samples is between 500 and 2000, incremental training of the end-to-end joint optimization neural network model is triggered, and the model is optimized in a targeted manner using difficult samples.

[0114] (iv) System Architecture like Figure 3 As shown, the visual inspection system based on end-to-end joint optimization provided by the present invention includes: Image acquisition module: Used to acquire the raw image to be inspected. The image acquisition module can be an image acquisition device such as an industrial camera, surveillance camera, or mobile phone camera, and is connected to the system via wired or wireless means.

[0115] The end-to-end jointly optimized neural network model is stored in a computer-readable storage medium and includes cascaded denoising subnetworks, enhancement subnetworks, and recognition subnetworks. The specific structure of each subnetwork is as described above.

[0116] Linkage Control Unit: Located between the denoising subnetwork, enhancement subnetwork, and recognition subnetwork, it enables dynamic bidirectional interaction during the inference phase. The linkage control unit includes: Trigger Judgment Module: Used to determine whether to trigger the linkage processing based on detection confidence, noise distribution information, and / or image quality parameters. Trigger conditions include: the presence of a confidence level lower than [a certain value]. The detection target has a noise intensity level higher than The region and the overall signal-to-noise ratio of the image are lower than wait.

[0117] Strategy generation module: After triggering the linkage process, it matches historically valid strategies from the strategy memory based on the current scene characteristics; if the match is successful, the historically valid strategy is applied directly; if the match fails, a new linkage strategy is generated online.

[0118] Effect evaluation module: Used to evaluate the effect of the linkage processing (such as the increase in confidence and the degree of improvement in signal-to-noise ratio); if the effect meets expectations, the strategy memory is updated; if the effect does not meet expectations, the linkage strategy is adjusted and re-executed.

[0119] Termination Judgment Module: This module terminates the linkage process when the number of times the linkage process is executed reaches a preset upper limit (e.g., 3 times), or the detection confidence reaches a preset high threshold (e.g., 0.85), or the confidence increase of two consecutive linkage processes is less than a preset minimum (e.g., 0.02).

[0120] Output module: Used to output visual inspection results. Output formats include image display, data reports, alarm signals, etc.

[0121] Strategy memory: Used to store the mapping relationship between scene features and enhancement strategies, as well as the corresponding effect scores. The strategy memory is a non-volatile memory (such as a solid-state drive or flash memory).

[0122] Hard sample library storage: Used to store image blocks of hard sample regions and processing records for each round. The hard sample library storage is non-volatile memory (such as solid-state drive or flash memory).

[0123] Processor: Used to execute various functions of the end-to-end joint optimization neural network model and linkage control unit. The processor can be a central processing unit (CPU), graphics processing unit (GPU), tensor processor (TPU), or application-specific integrated circuit (ASIC).

[0124] Example 1: Surface Defect Detection of Industrial Products 1. Application Scenarios This embodiment is applied to the automated visual inspection of surface defects on mobile phone casings. The product to be inspected is a metal frame of a mobile phone, and the types of surface defects to be inspected include: scratches (width ≤ 0.1 mm, length ≥ 1 mm), pits (diameter ≥ 0.2 mm, depth ≥ 0.05 mm), bubbles (diameter ≥ 0.3 mm), and foreign objects (size ≥ 0.2 mm).

[0125] 2. System Configuration Image acquisition equipment: Industrial area scan camera (resolution 2448×2048 pixels, frame rate 30fps), equipped with ring LED light source (color temperature 5000K, illuminance 2000 lux).

[0126] Computing equipment: Industrial computer (Intel Core i7 processor, NVIDIA GeForce RTX 3080 GPU, 32GB memory).

[0127] Model configuration: Denoising sub-network: U-Net architecture, 4 layers of encoder, 4 layers of decoder, initial number of channels 64, maximum number of channels 512.

[0128] Enhanced sub-network: GAN architecture, generator U-Net (5-layer downsampling + 5-layer upsampling), discriminator PatchGAN (5-layer convolution).

[0129] Recognition subnetwork: Faster R-CNN, backbone network ResNet-50, RPN anchor box scale [8,16,32,64,128], NMS threshold 0.5.

[0130] 3. Training process Training dataset: 5000 flawless mobile phone frame images were collected as base images. Simulated defects (500 scratches, dents, bubbles, and foreign objects each) were manually added to the images to generate clean images with defects. Different types and intensities of noise (Gaussian noise σ=5~30, salt-and-pepper noise density 0.01~0.15) were added to the defective images to generate noisy and defective images. Finally, 15000 training samples were obtained (noisy and defective images + noise-free reference images + enhanced reference images + defect annotations).

[0131] Training parameters: Phase 1 (Noise Reduction Pre-training): Learning rate 1×10 -3 Batch size 16, number of rounds 100.

[0132] Phase 2 (Enhanced Pre-training): Learning rate 2×10 -4 Batch size 8, number of rounds 150.

[0133] Phase 3 (Identification and Pre-training): Learning rate 1×10 -2 Batch size 8, number of rounds 80.

[0134] Phase 4 (End-to-End Fine-Tuning): Learning Rate 5×10 -5 Batch size 4, number of rounds 50.

[0135] Dynamic weights: Total steps Total steps.

[0136] 4. Reasoning process Input: An image of the phone frame to be tested (resolution 2448×2048 pixels).

[0137] Step 1 (Initial Forward Reasoning): The denoising subnetwork outputs a denoised image and a noise intensity distribution map. Detection revealed strong Gaussian noise in the lower left corner of the image. .

[0138] The enhancement subnetwork performs differentiated enhancement based on the noise intensity distribution map: the high-noise region in the lower left corner is enhanced with a low intensity. Other areas use high-strength reinforcement. .

[0139] The detection results from the identification subnetwork are as follows: A total of 5 targets were detected, of which 4 had a confidence score > 0.85, and 1 (a small scratch located in the high-noise area in the lower left corner) had a confidence score of 0.55, which is lower than the preset confidence threshold. .

[0140] Step Two (Confidence-Driven Secondary Reinforcement): The linkage control unit marks the area with the small scratch as the first target area.

[0141] The feature map of this region is cropped from the feature map of the enhanced subnetwork (corresponding to the original image position approximately (1800, 200, 100, 50)).

[0142] The region was enhanced a second time with high enhancement intensity: the contrast ratio was increased from 1.3 to 2.2, and the gain ratio was increased from 1.5 to 3.5.

[0143] After the second enhancement and subsequent identification, the confidence level increased to 0.78, representing an improvement of [percentage missing]. The results of the secondary identification were adopted.

[0144] Step 3 (Strategic Memory): Record the current scene features (average image brightness 142, contrast 0.45, main noise type Gaussian, target category scratch) and enhancement strategy (c=2.2, g=3.5), effect score = 0.23 / 0.12s=1.92.

[0145] If a similar scenario is encountered in subsequent processing, this strategy can be reused directly.

[0146] Output: Five defects were detected, including four high-confidence defects (confidence > 0.85) and one minor scratch confirmed after secondary enhancement (confidence 0.78). The category, location, and confidence level of all defects are displayed on the detection interface through the output module.

[0147] 5. Effect Comparison ; This embodiment demonstrates that, through end-to-end joint optimization and dynamic linkage mechanism in the inference stage, the present invention significantly improves the recall rate (from 92.1% to 96.8%) and precision rate (from 91.5% to 94.2%) of defect detection with only a small increase in inference time (0.03s), especially for the detection capability of low-confidence difficult samples.

[0148] Example 2: Visual inspection of highways under adverse weather conditions such as rain, fog, and snow.

[0149] 1. Application Scenarios This embodiment is applied to autonomous driving environmental perception and road target detection on highways and urban expressways under adverse weather conditions such as rain, fog, and snow. The road targets to be detected include: vehicles (cars, trucks, buses, motorcycles), pedestrians, traffic signs (speed limit signs, indicator signs, warning signs), lane markings, and road obstacles (debris, cones, etc.). The detection system must operate stably under the following adverse weather conditions: Rainy weather: Moderate to heavy rain (rainfall intensity ≥4mm / h), raindrops cause local blurring of images, reduced contrast, and severe road surface reflection; Foggy weather: Moderate to dense fog (visibility ≤150m), atmospheric scattering causes severe reduction in image contrast, blurred edges, and color distortion; Snowy weather: Moderate to heavy snow (snowfall intensity ≥3mm / h), snowflakes obscure and snow cover the road surface, making lane lines and road markings invisible; Mixed severe weather: complex degradation scenarios such as sleet, rain and fog.

[0150] This embodiment aims to address the problems of low target detection efficiency, high false negative and false positive rates caused by image degradation in adverse weather conditions, and to provide reliable visual perception capabilities for advanced driver assistance systems (ADAS) and autonomous driving systems.

[0151] 2. System Configuration Image acquisition equipment: vehicle-mounted forward-facing binocular camera (resolution 1920×1080 pixels, frame rate 30fps, dynamic range ≥120dB), equipped with windshield wipers and lens heating and defogging device; optional roadside fixed camera (resolution 3840×2160 pixels, frame rate 25fps) for highway roadside perception.

[0152] Computing equipment: vehicle-mounted computing platform (NVIDIA DRIVE Orin or equivalent computing platform, INT8 computing power ≥200TOPS), equipped with vehicle-mounted 12V power supply system and liquid cooling.

[0153] Model configuration: Denoising sub-network: U-Net architecture, 5 layers in encoder, 5 layers in decoder, initial number of channels 64, maximum number of channels 512. For different degradation types of rain, fog and snow, a weather type classification branch is added at the end of the encoder (output dimension 3, corresponding to the probability of rain, fog and snow respectively).

[0154] The enhancement subnetwork uses a GAN architecture, with a generator U-Net (6-layer downsampling + 6-layer upsampling) and a discriminator PatchGAN (7-layer convolution). The enhancement subnetwork receives the weather type probability vector from the denoising subnetwork and adaptively adjusts the enhancement strategy based on the weather type—focusing on contrast restoration and defogging for foggy days, descaling and detail restoration for rainy days, and removing snowflake occlusion and achieving brightness equalization for snowy days.

[0155] Recognition Sub-Network: RT-DETR (Real-Time Detection Transformer) target detection network, with ResNet-50 as the backbone network. The detection head adopts a Transformer decoder architecture and directly outputs the target category, bounding box, and confidence score.

[0156] 3. Training process (a) Construction of training dataset (specific to this embodiment) The joint training dataset contains noisy, low-quality road images, sharp reference images, enhanced reference images, detection annotations, and weather type labels. The data is constructed as follows: Basic image acquisition: Collect high-quality images of scenes such as highways and urban expressways under clear weather conditions, covering daytime, dusk and nighttime, with a total of no less than 20,000 images.

[0157] Physics simulation of weather degradation: Rainy day: Add raindrop stripes (size 2~10 pixels, density 50~500 stripes / frame) based on the physical rain pattern model, while reducing contrast and adding Gaussian noise (σ=5~20). Foggy weather: Based on atmospheric scattering model The scattering coefficient β is taken as 0.01~0.05 (corresponding to visibility of 50~500m), and the atmospheric light A is taken as 0.6~0.9; Snowy weather: Add snowflake occlusion based on the physical snowflake model (size 3~15 pixels, density 100~800 pieces / frame), reduce brightness and add Gaussian noise (σ=10~30).

[0158] Real data supplementation: Collect no fewer than 5,000 real images in actual rain, fog, and snow weather to enhance generalization ability.

[0159] Labeling: Boundary boxes are labeled for 10 types of targets, including vehicles, pedestrians, and traffic signs, and weather type labels are added.

[0160] (II) Brief Description of Training Strategies The same phased pre-training + end-to-end overall fine-tuning strategy as in Example 1 is adopted, with the learning rate, batch size and dynamic weight coefficients at each stage ( All are consistent with Example 1. The difference is that in the pre-training stage, the denoising subnetwork learns noise suppression and weather type classification simultaneously, and the enhancement subnetwork learns weather adaptive differential enhancement mapping.

[0161] 4. Reasoning process Input: A highway image (1920×1080) with visibility of approximately 100m in foggy weather.

[0162] (a) Weather type identification and initial forward reasoning (a unique step) The denoising subnetwork outputs a denoised image. Noise intensity distribution map and weather type probability The scene is identified as foggy. The enhancement subnetwork then selects a fog enhancement strategy (contrast coefficient) accordingly. (Improved defogging strength), and combined with Perform pixel-by-pixel differential enhancement: Low enhancement intensity is used for distant, high-fog areas. ), high enhancement intensity is used in the near-field low-fog area. The identification subnetwork outputs 8 targets, with a confidence score of 0.58 for pedestrians and 0.45 for speed limit signs, both below the threshold. .

[0163] (II) Dynamic Linkage Processing (General Process Overview) The basic process of this step—confidence-driven secondary enhancement, enhancement feedback correction denoising, policy memory and reuse, and multiple rounds of iteration on difficult samples—is exactly the same as in Example 1, and will not be repeated here. Only the application objects and effects in this scenario will be explained: Secondary enhancement targets: Local high-intensity enhancement is performed on pedestrian and speed limit sign areas with low confidence (contrast coefficient increased to 2.8, high-frequency gain increased to 4.0), pedestrian confidence is increased to 0.79, and speed limit signs are enhanced by a weighted fusion of the confidence of the primary and secondary results due to insufficient original information. ); Feedback correction target: The signal-to-noise ratio of the enhanced distant area was only 18.5dB (below the threshold of 22dB), and the original noise level of the area was low (12). It was determined that the enhancement introduced noise amplification, and the denoising sub-network was triggered to denoise again with a larger filter kernel (9×9) and a higher regularization coefficient (0.05). The overall confidence of distant detection was improved by about 8%. Strategy Memory: Combine the characteristics of this foggy scene (average brightness 98, contrast ratio 0.32) with effective enhancement strategies. Stored in a memory bank, it can be directly reused in subsequent similar foggy scenes, saving about 40% of the time spent on strategy generation; Hard sample accumulation: The rate limit flag fusion confidence is 0.51 (below the hard threshold of 0.50), triggering three rounds of iterative processing (low denoising and high enhancement, high denoising and low enhancement, medium denoising and medium enhancement). After fusion, the final confidence is 0.48. The image patch and processing record are stored in the hard sample library. Incremental training is triggered when the cumulative number of similar samples exceeds 1000.

[0164] Output: The final test results are output through the vehicle-mounted display screen and uploaded to the cloud-based traffic management system.

[0165] 5. Effect Comparison ; This embodiment demonstrates that, through differentiated enhancement of weather type perception and dynamic linkage mechanism, the present invention significantly improves the accuracy of visual detection for autonomous driving in adverse weather conditions (mAP in foggy weather is improved by 7.4 percentage points) with only a small increase in inference time (0.04s), providing reliable technical support for all-weather intelligent transportation systems.

[0166] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A software development and visual inspection method and system based on denoising and enhanced recognition, characterized in that, include: Obtain the original image to be detected; The original image is input into a pre-trained end-to-end joint optimization neural network model, which includes a cascaded denoising subnetwork, an enhancement subnetwork, and a recognition subnetwork. The original image is subjected to noise suppression processing through the denoising sub-network, and the noise-suppressed image and noise distribution information are output. The noise distribution information includes noise type, noise intensity level and noise spatial distribution. The image enhancement process is performed on the noise-suppressed image through the enhancement sub-network, and the enhanced image is output. The enhanced image is processed by the recognition sub-network to perform target recognition processing, and the visual detection results and the confidence level of each detected target are output. Obtain a first target region with a confidence level lower than a preset confidence threshold, wherein the preset confidence threshold is selected from any value between 0.5 and 0.75; Based on the corresponding position of the first target region in the original image, the enhancement sub-network is controlled to perform targeted secondary enhancement processing on the corresponding position to generate a secondary enhanced image; The enhanced image is then input into the recognition subnetwork for secondary recognition, generating a secondary recognition result and a secondary confidence level. The confidence level and the secondary confidence level are fused together for determination: when the increase in the secondary confidence level relative to the primary confidence level exceeds a preset increase threshold, the secondary recognition result is used as the final detection result of the first target region; When the improvement does not exceed the preset improvement threshold, the weighted fusion result of the recognition result corresponding to the confidence level and the secondary recognition result is used as the final detection result of the first target region, wherein the weighting coefficient of the weighted fusion is directly proportional to the improvement, and the value range of the weighting coefficient is 0.5 to 1.

0. The denoising subnetwork, the enhancement subnetwork, and the recognition subnetwork are jointly trained in an end-to-end manner, and the loss function of the joint training includes a weighted combination of denoising loss components, enhancement loss components, and recognition loss components.

2. The method according to claim 1, characterized in that, The loss function for the joint training is constructed in the following way: Obtain annotation data corresponding to the original image, the annotation data including a noise-free reference image, an enhanced reference image, and recognition annotation results; Based on the mean square error between the output of the denoising sub-network and the noise-free reference image, the denoising loss component L_d is calculated. The enhancement loss component L_e is calculated based on a weighted combination of the structural similarity index and mean square error between the output of the enhancement subnetwork and the enhancement reference image. The recognition loss component L_r is calculated based on a weighted combination of the cross-entropy loss and bounding box regression loss between the output of the recognition subnetwork and the recognition annotation result. The overall loss function for the joint training is L_total=α(t)·L_d+β(t)·L_e+γ(t)·L_r, where α(t), β(t), and γ(t) are the dynamic weight coefficients of the denoising loss component, the enhancement loss component, and the recognition loss component, respectively, and t is the number of training iterations. The dynamic weight coefficients are adjusted according to the following strategy: when the number of training steps t is less than a first preset step threshold, α(t) > β(t) and α(t) > γ(t); when the number of training steps t is greater than or equal to the first preset step threshold and less than a second preset step threshold, β(t) gradually increases to β(t) > α(t) and β(t) > γ(t); when the number of training steps t is greater than or equal to the second preset step threshold, γ(t) gradually increases to γ(t) > α(t) and γ(t) > β(t); the first preset step threshold is 30% to 40% of the total number of training steps, and the second preset step threshold is 65% to 75% of the total number of training steps.

3. The method according to claim 1, characterized in that, The training of the end-to-end jointly optimized neural network model adopts a strategy that combines staged pre-training with end-to-end global fine-tuning: In the first stage, the denoising sub-network is pre-trained using a denoising training dataset. The denoising training dataset contains multiple pairs of noise-clean images with different noise types and different noise intensity levels. The noise types include Gaussian noise, salt-and-pepper noise, and Poisson noise. The noise intensity level is measured by the noise standard deviation σ, and the value of σ ranges from 5 to 50. In the second stage, the augmentation subnetwork is pre-trained using an augmentation training dataset. The augmentation training dataset contains multiple sets of image pairs of low-quality images and corresponding high-quality reference images. The low-quality images are images acquired under illumination conditions of less than 100 lux or images with a contrast of less than 0.

3. In the third stage, the recognition sub-network is pre-trained using a recognition training dataset, which contains detection images with labeled target categories and bounding boxes. In the fourth stage, the pre-trained denoising sub-network, the enhancement sub-network, and the recognition sub-network are cascaded in sequence, and the whole network is fine-tuned in an end-to-end manner using a joint training dataset. Each training image in the joint training dataset simultaneously contains the corresponding noiseless reference image, the enhancement reference image, and the recognition annotation result.

4. The method according to claim 1, characterized in that, The step of controlling the enhancement sub-network to perform targeted secondary enhancement processing on the corresponding position of the first target region in the original image specifically includes: Extract the position coordinates and size information of the first target region in the original image; Crop the feature regions corresponding to the position coordinates and size information from the feature map of the enhanced subnetwork; The feature region is subjected to a secondary enhancement process using enhancement intensity parameters higher than those in the initial enhancement process. These higher enhancement intensity parameters include: increasing the contrast stretching ratio from the first contrast coefficient of the initial enhancement to a second contrast coefficient, where the first contrast coefficient ranges from 1.1 to 1.5 and the second contrast coefficient ranges from 1.6 to 2.5; and / or increasing the high-frequency component enhancement gain from the first gain value of the initial enhancement to a second gain value, where the first gain value ranges from 1.0 to 1.8 and the second gain value ranges from 2.0 to 4.

0. The feature regions after secondary enhancement are fused back to their corresponding positions in the feature map of the enhanced sub-network to generate the secondary enhanced image.

5. The method according to claim 1, characterized in that, When performing noise suppression processing on the original image, the denoising sub-network generates a noise intensity distribution map of each pixel position in the original image. The noise intensity distribution map is transmitted to the enhancement subnetwork; The enhancement sub-network determines the enhancement intensity coefficient of each pixel position according to the noise intensity distribution map. For positions where the noise intensity level is higher than the first preset noise level threshold, the first enhancement intensity coefficient is used. The value range of the first enhancement intensity coefficient is 0.1 to 0.

3. For locations where the noise intensity level is lower than the second preset noise level threshold, a second enhancement intensity coefficient is used. The value of the second enhancement intensity coefficient ranges from 0.5 to 0.9, and the second enhancement intensity coefficient is greater than the first enhancement intensity coefficient. For locations where the noise intensity level is between the first preset noise level threshold and the second preset noise level threshold, the enhancement intensity coefficient is determined by linear interpolation. The enhancement subnetwork performs pixel-by-pixel differential enhancement processing on the noise-suppressed image based on the enhancement intensity coefficient.

6. The method according to claim 5, characterized in that, After the enhancement subnetwork completes the enhancement processing of the noise-suppressed image, it calculates the signal-to-noise ratio and contrast of the enhanced image. The signal-to-noise ratio of the enhanced image is compared with a preset signal-to-noise ratio threshold, wherein the preset signal-to-noise ratio threshold ranges from 20dB to 30dB. When the signal-to-noise ratio of the target region in the enhanced image is lower than the preset signal-to-noise ratio threshold, the target region is marked as a noise amplification region; Obtain the noise intensity level corresponding to the noise amplification region in the noise intensity distribution map. If the noise intensity level is lower than the third preset noise level threshold, it is determined that the noise amplification in the noise amplification region is introduced by the enhancement process, and a noise reduction parameter correction instruction is generated. The denoising sub-network re-denoises the positions in the original image corresponding to the noise amplification region according to the denoising parameter correction instructions. The re-denoising process uses stronger smoothing parameters than the initial denoising process. The stronger smoothing parameters include: increasing the filter kernel size from a first size to a second size, where the first size ranges from 3×3 to 5×5 and the second size ranges from 7×7 to 11×11; and / or increasing the regularization coefficient from a first regularization value to a second regularization value, where the first regularization value ranges from 0.001 to 0.01 and the second regularization value ranges from 0.02 to 0.

1. The image after re-denoising is input again into the enhancement subnetwork and the recognition subnetwork for enhancement and recognition.

7. The method according to claim 1, characterized in that, It also includes a strategy memory bank and a hard sample bank; The strategy memory is used to store enhancement strategies for different scene features in historical processing and their corresponding effect scores. The scene features include at least two of the following: average image brightness, image contrast, main noise type, and detection target category. When the confidence level output by the recognition sub-network is lower than the preset confidence threshold, the scene features of the current image are extracted, and historical records with a similarity to the current scene features exceeding a preset similarity threshold are matched in the policy memory. The preset similarity threshold ranges from 0.75 to 0.

90. If the match is successful, the corresponding enhancement strategy in the historical record is directly used to perform secondary enhancement processing on the first target region; If the matching fails, an enhancement strategy is generated online according to the method described in claim 4 for secondary enhancement processing. After the enhancement processing is completed, the current scene features, the enhancement strategy adopted, and the increase in confidence level relative to the confidence level of the secondary confidence level are stored as a new record in the strategy memory bank. The effect score of the new record is the ratio of the increase in confidence level to the processing time. When the confidence level output by the recognition sub-network is lower than a preset difficulty threshold, the corresponding target region is marked as a difficult sample region. The preset difficulty threshold ranges from 0.3 to 0.

5. The location coordinates of the difficult sample region are extracted, and multiple rounds of iterative processing are performed on the difficult sample region. Each round of iterative processing includes: the denoising subnetwork and the enhancement subnetwork respectively use different parameter configurations to denoise and enhance the difficult sample region, and then the recognition subnetwork performs recognition to obtain the recognition result and confidence level of each round of iteration. Among them, different parameter configuration combinations are used in different rounds of iteration. The first round uses a parameter combination of low denoising intensity and high enhancement intensity, the second round uses a parameter combination of high denoising intensity and low enhancement intensity, and the third round uses a parameter combination of medium denoising intensity and medium enhancement intensity. The low denoising intensity corresponds to a filter kernel size of 3×3 to 5×5, the medium denoising intensity corresponds to a filter kernel size of 7×7 to 9×9, and the high denoising intensity corresponds to a filter kernel size of 11×11 to 15×15. The recognition results of each round of iteration are weighted and fused according to the confidence level. The recognition results with higher confidence levels are assigned higher fusion weights, and the final detection result of the difficult sample region is output. The original image blocks of the difficult sample region, the parameter configuration of each round, and the recognition results of each round are stored in the difficult sample library. When the number of difficult samples of the same category in the difficult sample library exceeds a preset number threshold, the incremental training of the end-to-end joint optimization neural network model is triggered. The preset number threshold ranges from 500 to 2000 images.

8. A visual inspection system based on end-to-end joint optimization, characterized in that, include: The image acquisition module is used to acquire the original image to be detected; An end-to-end jointly optimized neural network model is stored in a computer-readable storage medium, including cascaded denoising subnetworks, enhancement subnetworks, and recognition subnetworks; A processor that implements the method as described in any one of claims 1 to 7 when executing the end-to-end joint optimization neural network model; The output module is used to output the visual inspection results; The strategy memory is used to store the mapping relationship between scene features and enhancement strategies, as well as the corresponding effect scores. The difficult sample library storage is used to store image blocks of difficult sample regions and processing records for each round.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images