Image processing method and device of image acquisition card

Abstract

The invention relates to the technical field of image processing, and discloses an image processing method and device for an image acquisition card, and the method comprises the steps: converting an input optical signal into original image data through a sensor array of the image acquisition card, and carrying out the preprocessing, and obtaining a preprocessed image sequence; constructing an image state space model and a color measurement noise model based on the preprocessed image sequence, and performing preliminary denoising on the preprocessed image sequence to obtain a bidirectional time sequence feature tensor; inputting the bidirectional time sequence feature tensor into an additive fusion bidirectional time convolutional neural network for feature extraction to obtain a comprehensive spatial-temporal feature and a confidence map; according to the method, gradient descent image processing parameter optimization is carried out according to the comprehensive spatial-temporal characteristics and the confidence map, optimal processing parameters of the image acquisition card are obtained, and a target output image is generated, and the method improves the image definition and the detail retention capability in a dynamic scene, and improves the robustness of image processing.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image processing method and device for an image acquisition card. Background Art

[0002] Image acquisition cards face challenges such as complex lighting conditions, dynamic scene changes, and multi-source noise interference, which lead to unstable image quality and serious impact on subsequent image analysis and recognition accuracy. In particular, color measurement noise, as a kind of noise with correlation in the time domain, is significantly different from the traditional white noise model, making it difficult for conventional image processing filtering algorithms to effectively deal with it, thus affecting the processing accuracy of the image acquisition card data fusion system.

[0003] There are three main technical bottlenecks in the traditional image acquisition card processing flow: the existing noise suppression methods are usually based on the independent and identically distributed white noise assumption, and cannot effectively process color measurement noise with time-series correlation; secondly, most image processing algorithms only use one-way time-series information or completely ignore the time-series characteristics, and fail to fully utilize the two-way time-series correlation information contained in the video stream for analysis and processing; thirdly, the selection of image processing parameters usually relies on manual experience judgment or preset fixed values, and lacks the ability to adaptively adjust for different scenes and noise environments, resulting in the quality and stability of image processing results. It is difficult to guarantee, especially in environments where lighting conditions change frequently or noise characteristics change dynamically. Summary of the invention

[0004] The invention provides an image processing method and device for an image acquisition card, which improves the image clarity and detail retention capability in dynamic scenes and improves the robustness of image processing.

[0005] In a first aspect, the present invention provides an image processing method of an image acquisition card, the image processing method of the image acquisition card comprising: The input optical signal is converted into raw image data through the sensor array of the image acquisition card, and preprocessing is performed to obtain a preprocessed image sequence; Building an image state space model and a color measurement noise model based on the preprocessed image sequence, performing preliminary denoising on the preprocessed image sequence, and obtaining a bidirectional time series feature tensor; Inputting the bidirectional time series feature tensor into an additively fused bidirectional temporal convolutional neural network for feature extraction to obtain comprehensive spatiotemporal features and a confidence map; The image processing parameters are optimized by gradient descent according to the comprehensive spatiotemporal features and the confidence map to obtain the optimal processing parameters of the image acquisition card and generate a target output image.

[0006] In a second aspect, the present invention provides an image processing device of an image acquisition card, the image processing device of the image acquisition card comprising: A preprocessing module, used to convert the input optical signal into raw image data through the sensor array of the image acquisition card, and perform preprocessing to obtain a preprocessed image sequence; A preliminary denoising module, used for constructing an image state space model and a color measurement noise model based on the preprocessed image sequence, performing preliminary denoising on the preprocessed image sequence, and obtaining a bidirectional time series feature tensor; A feature extraction module, used for inputting the bidirectional time series feature tensor into an additive fusion bidirectional temporal convolutional neural network for feature extraction to obtain comprehensive spatiotemporal features and a confidence map; A generation module is used to perform gradient descent image processing parameter optimization according to the comprehensive spatiotemporal features and the confidence map, obtain the optimal processing parameters of the image acquisition card, and generate a target output image.

[0007] In the technical solution provided by the present invention, by constructing a color measurement noise model and estimating noise statistical parameters in combination with an expectation maximization algorithm, the limitations of the traditional white noise assumption are overcome, the time domain correlation characteristics of noise in the image acquisition card can be accurately captured, and the image processing quality in a complex noise environment is significantly improved. A bidirectional temporal sliding window method is used to construct forward and backward temporal sequences, and rich spatiotemporal features are extracted through multi-scale pyramid transformation and spatial block division, so that the image processing system can simultaneously capture temporal information from the past to the present and from the future to the present, greatly improving the image clarity and detail retention ability in dynamic scenes. An image quality assessment matrix constructed based on comprehensive spatiotemporal features and confidence maps is combined with a gradient descent method to achieve automatic optimization of image processing parameters, which completely solves the subjective problem of traditional methods relying on manual experience judgment, so that the image acquisition card can intelligently adjust the processing parameters according to different scene characteristics. The additive fusion bidirectional temporal convolutional neural network greatly reduces the amount of calculation parameters through dilated convolution and attention mechanisms, while maintaining a large receptive field, so that the algorithm can run efficiently on resource-constrained image acquisition cards, and the processing delay is controlled within a frame cycle to meet real-time processing requirements. The adaptive fusion strategy based on confidence map effectively combines the advantages of parameter optimization and Kalman filtering, automatically adjusts the processing strategy according to the feature reliability of different regions, improves the robustness of image processing, and significantly improves the processing effect in areas with uneven lighting.

[0008] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0009] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A schematic diagram of an embodiment of an image processing method of an image acquisition card in an embodiment of the present invention; Figure 2 FIG. 1 is a schematic diagram of an embodiment of an image processing device of an image acquisition card in an embodiment of the present invention. DETAILED DESCRIPTION

[0011] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0012] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device end including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or device ends.

[0013] To facilitate understanding of this embodiment, firstly, an image processing method of an image acquisition card disclosed in an embodiment of the present invention is described in detail. Figure 1 As shown, the method comprises the following steps: 101. Convert the input optical signal into raw image data through the sensor array of the image acquisition card, and perform preprocessing to obtain a preprocessed image sequence; It is understandable that the execution subject of the present invention may be an image processing device of an image acquisition card, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0014] Specifically, the sensor array of the image acquisition card acquires the input optical signal at a preset sampling frequency, and ensures that the sampling frequency is between 60Hz and 240Hz, and records the acquisition timestamp of each frame of the image to ensure that subsequent processing can accurately correspond to each time point to obtain the original image data. The original image data is subjected to nonlinear response correction, and the photoelectric response characteristics of the sensor are compensated by a lookup table or polynomial approximation method, so that the output image brightness and the actual incident light intensity present a linear relationship, and a linear response image is obtained. A first threshold is set for the linear response image, and the threshold is taken as 0.1% to 0.5% of the dynamic range of the grayscale value of the original image, and then all pixels of the image are traversed, and the mean deviation of each pixel relative to its neighboring pixels is calculated. When a pixel value deviates from its neighborhood mean by more than the first threshold, the pixel is identified as a bad pixel. In order to repair the bad pixels and ensure the coherence of the image, the four-directional bilinear interpolation algorithm is used to replace the bad pixels. The grayscale values ​​of the adjacent pixels in the four directions of the bad pixel location are selected, and the weighted mean values ​​in these directions are calculated respectively, and finally the new pixel values ​​are fused to generate a bad pixel-free image. The dark field image pre-stored in the image acquisition card memory is subtracted from the bad pixel-free image to eliminate the fixed noise of the sensor in the absence of light. The dark field image is pre-collected and stored when the camera is in a dark environment, and is used to compensate for the dark current noise and fixed pattern noise in the image. Through this subtraction operation, a dark field correction image is obtained, which makes the background noise of the image more uniform and eliminates the dark current influence of the sensor. In order to compensate for the non-uniform response of the optical system, a flat field correction is performed on the dark field correction image, and the dark field correction image is divided by a pre-stored normalized flat field image. The normalized flat field image is taken under uniform illumination conditions and is normalized to characterize the uniformity of the response of the entire sensor. Through the normalized division operation, a flat-field corrected image is obtained, thereby eliminating the uneven brightness problem caused by factors such as uneven lens light transmission and uneven sensor pixel response, making the brightness distribution of the image more uniform within the entire field of view. Based on the distortion correction parameter matrix and distortion coefficient vector built into the image acquisition card, the flat-field corrected image is reprojected. The pixel coordinates in the distorted image are mapped back to the ideal geometric coordinate system using a polynomial model or a reverse mapping method to eliminate radial distortion and tangential distortion, making the geometric structure of the image more accurate. After geometric correction, a preprocessed image sequence is finally obtained, and the sequence is stored in the image buffer of the image acquisition card for subsequent timing analysis, noise estimation, and deep learning processing. The entire preprocessing process is performed by a dedicated hardware acceleration unit inside the acquisition card to ensure that the processing delay does not exceed 16.7ms, thereby meeting the requirements of real-time image processing with high frame rate and low latency.

[0015] 102. Construct an image state space model and a color measurement noise model based on the preprocessed image sequence, perform preliminary denoising on the preprocessed image sequence, and obtain a bidirectional time series feature tensor; Specifically, each frame of the preprocessed image sequence is represented as a state vector and a measurement vector, where the state vector is used to describe the real state of the image, and the measurement vector is composed of the image data actually observed by the image acquisition card. Based on the state vector and the measurement vector, the state transition relationship and the measurement relationship are established to obtain the image state space model. In this model, the state transition relationship is used to describe the way the image state changes in the time series, and the measurement relationship is used to associate the relationship between the observed image data and the real state, so that the system can infer the future state based on the current observation information and correct the deviation caused by noise or external interference. Based on the state space model, the measurement noise is modeled, and the discrete color noise model is established by the inverse Euler method. In this process, it is assumed that the measurement noise has correlation between different time points, that is, the noise at the current moment is not only affected by the current state, but also by the accumulation of noise at multiple time points in the past. Therefore, a mathematical modeling method is used to describe this characteristic. Through the inverse Euler method, the continuous time dynamic equation of the measurement noise is discretized to obtain a discrete color noise model suitable for digital signal processing, which can more accurately describe the evolution characteristics of the noise over time. The state vector and the noise in the discrete color noise model are combined into an extended state vector, and an extended state relationship and an extended measurement relationship are constructed. The extended state vector contains not only information about the image state, but also the dynamic characteristics of the measurement noise, so that the subsequent filtering and optimization processing can simultaneously model the image content and noise distribution. In this way, the system more accurately characterizes the impact of noise on the image data, and compensates and suppresses it through filtering and parameter optimization methods. After the extended state space model is constructed, in order to accurately estimate the statistical characteristics of the noise, the expectation maximization algorithm is used for optimization calculation. The conditional expectation is calculated based on the current parameter estimation value to complete the first step of the algorithm. In this process, the expected value of the system state and noise distribution under the existing observation data conditions is calculated using the currently estimated noise statistical parameters to obtain an optimization target for the statistical characteristics of the noise. In the second step, the smoothed Kalman filter is used to perform forward-backward recursive calculations to obtain state estimates and error covariances, and the noise covariance matrix is ​​updated by maximization, so that the system can more accurately describe the characteristics of the measurement noise. In the forward recursive process, the state is updated based on the current observation data, and the optimal state estimate is calculated in combination with the prediction error; in the backward recursive process, the previous state estimate is corrected using future observation data, thereby improving the accuracy of the state estimate and reducing the deviation caused by noise. After the state estimation is completed, the noise statistical parameters are updated. In the second step of the expectation maximization algorithm, a new noise covariance matrix is ​​calculated based on the filtering results, and the parameters in the color noise model are updated to ensure that the model can accurately reflect the noise characteristics of the current observation data.The process is iterated continuously until the parameter change is less than the preset second threshold, that is, the change amplitude of the noise statistical parameters in two consecutive iterations is lower than the set convergence condition, or the maximum number of iterations is reached, and the optimal noise statistical parameters are obtained. The noise statistical parameters are used to perform preliminary denoising on the preprocessed image sequence. For each frame of the image, the estimated noise covariance matrix is ​​used for filtering to reduce the influence of color measurement noise while retaining the image detail information. The denoised image data is organized into a bidirectional time series feature tensor, where the forward feature tensor represents the image feature information in the forward time window, and the backward feature tensor represents the image feature information in the reverse time window, so that the system considers both past and future image information.

[0016] The noise statistical parameters obtained in the previous stage are used to perform denoising filtering on the preprocessed image sequence to generate a denoised image sequence. The denoising filtering process adaptively adjusts the filtering strategy according to the calculated noise statistical characteristics to ensure that the color measurement noise is removed to the greatest extent while retaining the image detail information as much as possible. The denoising filter is based on the noise covariance matrix obtained in the previous stage, and adaptively filters each frame of the image to eliminate the time-related noise components, so that the processed image is smoother in time sequence and has a higher signal-to-noise ratio. For each central time point in the denoised image sequence, a forward time sequence window and a backward time sequence window are constructed, where the forward time sequence window is arranged in time order, that is, it contains the current time point and multiple time frames before it, and the backward time sequence window is arranged in time reverse order, that is, it contains the current time point and multiple time frames after it, ensuring that the system considers both past and future information when analyzing time series data, thereby improving the accuracy of time series modeling. By constructing a bidirectional sliding window, a time series data structure is formed, so that the system can effectively use the information of the time dimension to enhance the denoising effect and detail retention ability in the subsequent feature extraction process. A multi-scale pyramid transform is performed on each frame of the bidirectional sliding window sequence to extract the feature information of the image at different scales. The multi-scale pyramid transform converts the original image into multiple sub-images with different resolutions by downsampling step by step, so that the system can analyze the global structure and local details of the image at the same time. For each frame of the image, a pyramid layer of the original resolution is generated, and then multiple low-resolution versions are gradually generated through downsampling operations to form a multi-scale pyramid image sequence. In this process, images of different scales can capture information at different levels. For example, high-resolution images can retain rich detail information, while low-resolution images can better describe the overall structural features, so that the system can select a suitable scale for calculation according to different processing requirements in the subsequent analysis process. Each layer of the multi-scale pyramid image sequence is divided into non-overlapping spatial blocks to form a block-divided image. Each layer of the pyramid image is divided into spatial blocks of the same size, and the size of each spatial block depends on the requirements of the processing task, such as selecting 8×8 or 16×16 pixel blocks to ensure that the local structural information of the image is fully captured when extracting local features. For each spatial block, the mean of all pixels in the block is calculated to reflect the overall brightness level of the block; the standard deviation is calculated to measure the brightness change amplitude of the block; the maximum and minimum values ​​in the block are extracted to characterize the contrast range of the area. In order to further enrich the block features, the gradient histogram is calculated, which can reflect the edge and texture distribution in the block. In the process of calculating these features, the pixel data of each spatial block is converted into a high-dimensional feature vector and combined with the original block data to obtain the enhanced block features. The enhanced block features are reorganized into forward temporal feature tensors and backward temporal feature tensors in chronological order.The enhanced block features at each time point are stacked in the order of the time series to ensure the integrity of the time series information. The forward time series feature tensor is composed of the enhanced block features in the forward time series window, arranged in the positive time order, while the backward time series feature tensor is composed of the enhanced block features in the backward time series window, arranged in the reverse time order.

[0017] 103. Input the bidirectional time series feature tensor into the additive fusion bidirectional time convolutional neural network for feature extraction to obtain comprehensive spatiotemporal features and confidence maps; Specifically, the forward temporal feature tensor is input into the forward branch of the bidirectional temporal convolutional neural network to obtain a first initial feature map, and the backward temporal feature tensor is input into the backward branch to obtain a second initial feature map. The first initial feature map is deeply processed by multiple temporal convolution blocks to generate a first multi-layer feature map of the forward branch, and the second initial feature map is processed by the same number of temporal convolution blocks to generate a second multi-layer feature map of the backward branch. The role of the temporal convolution block is to expand the receptive field in the time dimension by dilating convolution, so that the network can capture image features within a longer time range. Each temporal convolution block contains a dilated convolution layer, a batch normalization layer, a nonlinear activation layer, and a residual connection layer. This structure ensures the gradual aggregation of information, reduces the problem of gradient disappearance, and improves the expression ability of temporal features. An attention module is applied to the first multi-layer feature map and the second multi-layer feature map to calculate the attention weight and perform weighted fusion. The core function of the attention module is to assign importance weights to different time points, so that the network can focus more on temporal feature points that have a greater impact on image quality. The feature map output by each temporal convolutional block passes through a fully connected transformation layer to calculate the attention weight of each time step, and then a normalization operation is performed to map the weight to a reasonable probability distribution range, and finally the contribution of the feature in time is adjusted by weighted fusion. After the attention weighting process, the first multi-layer feature map is converted into the first attention weighted feature, and the second multi-layer feature map is converted into the second attention weighted feature. The first attention weighted feature of the last layer of the forward branch and the second attention weighted feature of the last layer of the backward branch are subjected to additive fusion operation. Different weight coefficients are set for the forward and backward features to ensure that the fused features make full use of the bidirectional information while retaining the key information in the time series. The fusion method adopts element-by-element weighted summation, and introduces feature interaction operations to ensure the complementarity of the forward and backward information, thereby improving the expressiveness of the features. After the fusion operation, the generated fused features contain the information of the entire time series. The fused features are reduced and refined through a three-layer convolutional network to obtain comprehensive spatiotemporal features, and the confidence map is calculated at the same time. The role of the three-layer convolutional network is to gradually compress the feature dimension and extract the most representative feature components, so that the network achieves the best balance between computational efficiency and storage requirements. The first layer of convolution is used to reduce the feature dimension while maintaining spatial structural information, the second layer of convolution is used to enhance the coherence of local features, and the third layer of convolution is used for final feature refinement and generate output features with high semantic information. The confidence map is also calculated, which is used to evaluate the reliability of features at different spatial locations to ensure that in subsequent processing, high confidence areas are given higher weights, while low confidence areas are treated more robustly to reduce the impact of uncertainty on the quality of the final output image.

[0018] 104. Based on the comprehensive spatiotemporal characteristics and the confidence map, the image processing parameters are optimized by gradient descent to obtain the optimal processing parameters of the image acquisition card and generate the target output image.

[0019] Specifically, the image processing parameter space of the image acquisition card is defined, which contains key parameters such as denoising intensity, sharpening coefficient, contrast gain, brightness adjustment, color saturation, detail enhancement intensity and texture preservation factor. For example, denoising intensity determines the degree of noise suppression, while sharpening coefficient affects the edge clarity of the image, contrast gain is used to adjust the overall contrast of the image, while brightness adjustment controls the global brightness of the image, color saturation affects the vividness of the color, detail enhancement intensity determines the degree of retention of texture details in the image, and texture preservation factor is used to balance the relationship between denoising and detail loss. Based on the comprehensive spatiotemporal characteristics, the clarity index, noise level index, detail preservation index and color fidelity index are calculated to construct an image quality evaluation matrix. The sharpness index measures the sharpness of the image by calculating the variance of the image gradient amplitude, while the noise level index evaluates the noise level of the image by energy analysis of high-frequency subbands. The detail preservation index measures the image detail information retained in the denoising process by calculating the structural similarity index, and the color fidelity index evaluates the accuracy of color restoration by calculating the earth moving distance of the color histogram. These indicators evaluate multiple dimensions of image quality and ensure that the quality of other key indicators will not be sacrificed due to the improvement of a single indicator during the optimization process. The image quality assessment matrix is ​​weighted according to the confidence map as the spatial weight to obtain the weighted quality score. The confidence map reflects the reliability of different areas in the image. When calculating the weighted quality score, a higher weight is given to the high confidence area, while a smaller influence is exerted on the low confidence area, ensuring that the overall optimization direction is more robust and consistent with the actual observation data. By weighted calculation of the image quality assessment matrix, the global quality of image processing is captured, while the interference of abnormal areas on the overall optimization process is reduced. In order to optimize the image processing parameters, the energy error is calculated according to the difference between the weighted quality score and the ideal image quality index, and the objective function of parameter optimization is constructed. In this process, the ideal image quality index is determined by high-quality reference images or expert-annotated data, while the energy error is used to measure the deviation between the current image quality and the ideal state, and is used as the optimization target for parameter optimization. In order to make the optimization process more stable, the normalized error measurement method is used to ensure the weight balance between different indicators and avoid the dominant role of a single indicator in the overall optimization process. After the objective function is established, the objective function is iteratively optimized by applying the gradient descent method according to the image processing parameter space to gradually update the image processing parameters and generate a parameter update sequence. In the gradient descent process, the gradient of the objective function relative to each parameter is calculated in each iteration, and the parameters are adjusted along the direction of the gradient descent, so as to gradually converge to the optimal solution. In the specific implementation process, the adaptive learning rate method is used to improve the convergence speed and reduce the oscillation problem, and the momentum optimization method is introduced to avoid falling into the local minimum point.In order to ensure the stability of the optimization process, the parameter update sequence is iteratively calculated until the convergence condition is met. That is, when the parameter change between two adjacent iterations is less than the preset threshold, or the change amplitude of the objective function is small enough, the optimization process is considered to have converged, and the optimal processing parameters of the image acquisition card are obtained. The preprocessed image sequence is processed using the optimal processing parameters to generate the target output image. For each frame of the preprocessed image, the weight of the filtering algorithm is adjusted using the optimized denoising intensity parameter to remove noise to the greatest extent without losing details. At the same time, the optimized sharpening coefficient is applied to enhance the image edge to make the image clearer, and the contrast gain is used to adjust the light and dark contrast of the image to make the detail level richer. At the same time, the brightness parameter is adjusted to optimize the overall brightness distribution, and the optimized color saturation is used to adjust the color expression to make the image more natural and vivid. The detail enhancement parameter and texture preservation factor will also affect the final output image. The detail enhancement parameter is used to enhance the high-frequency texture information, while the texture preservation factor ensures that too much image structure information is not lost during the denoising process.

[0020] Image processing operators are applied to the preprocessed image sequence based on the optimal processing parameters, so that the image achieves the best balance in terms of denoising, sharpening, contrast enhancement, etc. The role of the image processing operator is to adaptively adjust the processing method according to different image characteristics, such as improving the contrast gain parameter for low-contrast images, improving the denoising intensity parameter for high-noise images, and optimizing the detail enhancement parameter and texture retention factor for complex texture areas, so as to ensure that the final output image can effectively suppress noise and retain rich detail information. After this step, the image with optimized parameters is obtained. An extended state Kalman filter is constructed based on the noise statistical parameters calculated in the previous stage, and the state prediction, Kalman gain calculation and state update steps are performed on the preprocessed image sequence to obtain the filtered image. In the specific implementation process, the extended state space model is used to predict the state of the image sequence, that is, the optimal state of the next frame is inferred based on the state information of the current frame and the past frame, so as to achieve reasonable compensation of the time series information. After completing the state prediction, the Kalman gain is calculated to measure the credibility between the observed data and the predicted state, and the state update weight is adjusted accordingly, so that the filtered image can retain the real image structure while minimizing the influence of measurement noise. Through the state update step, the predicted state is combined with the actual observation data to obtain the final filtered image. The confidence map is used to calculate the fusion weight, and the adaptive fusion operation is performed on the parameter optimized image and the filtered image to generate a fused image. The role of the confidence map is to measure the reliability of the features at different spatial locations and assign different fusion weights accordingly. For example, in areas with high signal-to-noise ratio, the parameter optimized image is given a higher weight, while in areas with large noise, the filtered image is given a higher weight. The fusion operation is performed in a weighted average manner. Through the pixel-level fusion strategy, the final generated fused image achieves the best balance between global consistency and local detail optimization. The comprehensive spatiotemporal features are mapped to the enhancement parameter space, and the fused image is spatially adaptively enhanced to obtain an enhanced image. The comprehensive spatiotemporal features calculated in the previous stage are converted into a set of adaptive enhancement parameters, such as local contrast enhancement coefficient, local detail enhancement intensity, color enhancement parameters, etc., and adaptively adjusted at the pixel level or block level, so that the enhancement effect of different regions can be dynamically optimized according to the temporal characteristics. For example, the detail enhancement factor is increased in texture-rich areas, while the sharpness is reduced in flat areas, ensuring that the image enhancement effect does not cause artifacts or over-enhancement of edges. Global color correction is performed on the enhanced image, and the hue, saturation, and brightness of the image are adjusted to obtain a color-corrected image.The hue of the image is adjusted through the color mapping function to eliminate the color cast problem and ensure color consistency under different lighting conditions; the color saturation adjustment algorithm is used to optimize the color expression of the image, making the color more vivid without over-saturation; the global brightness distribution of the image is optimized through the brightness adjustment function, so that it can present a more balanced brightness effect under different exposure conditions. An edge-preserving filter is applied to the color-corrected image to retain the edge information of the image while removing the remaining noise, and the final filtered image is quantized to the output format of the image acquisition card to generate the target output image. During the quantization process, the image data is converted to 8-bit, 10-bit or 12-bit precision according to the output format requirements of the image acquisition card to ensure that the final target output image meets the needs of different application scenarios.

[0021] In the embodiment of the present invention, by constructing a color measurement noise model and estimating noise statistical parameters in combination with an expectation maximization algorithm, the limitations of the traditional white noise assumption are overcome, the temporal correlation characteristics of noise in an image acquisition card can be accurately captured, and the image processing quality in a complex noise environment is significantly improved. A bidirectional temporal sliding window method is used to construct forward and backward temporal sequences, and rich spatiotemporal features are extracted through multi-scale pyramid transformation and spatial block division, so that the image processing system can simultaneously capture temporal information from the past to the present and from the future to the present, greatly improving the image clarity and detail retention capabilities in dynamic scenes. An image quality assessment matrix constructed based on comprehensive spatiotemporal features and confidence maps is combined with a gradient descent method to achieve automatic optimization of image processing parameters, and the subjective problem of traditional methods relying on manual experience judgment is completely solved, so that the image acquisition card can intelligently adjust the processing parameters according to different scene characteristics. The additive fusion bidirectional temporal convolutional neural network greatly reduces the amount of calculation parameters through dilated convolution and attention mechanisms, while maintaining a large receptive field, so that the algorithm can run efficiently on a resource-constrained image acquisition card, and the processing delay is controlled within a frame cycle to meet real-time processing requirements. The adaptive fusion strategy based on confidence map effectively combines the advantages of parameter optimization and Kalman filtering, automatically adjusts the processing strategy according to the feature reliability of different regions, improves the robustness of image processing, and significantly improves the processing effect in areas with uneven lighting.

[0022] In a specific embodiment, the process of executing step 101 may specifically include the following steps: The input optical signal is collected by the sensor array of the image acquisition card at a preset sampling frequency, and the acquisition timestamp of each frame of the image is recorded to obtain the original image data; Performing nonlinear response correction on the original image data to obtain a linear response image; A first threshold is set for the linear response image, pixel points whose pixel values ​​deviate from the neighborhood mean by more than the first threshold are identified, and the pixel points are replaced using a four-directional bilinear interpolation algorithm to obtain a defect-free image; Subtracting a dark field image pre-stored in the memory of the image acquisition card from the image without bad pixels to obtain a dark field correction image, and dividing the dark field correction image by a pre-stored normalized flat field image to obtain a flat field correction image; Based on the distortion correction parameter matrix and distortion coefficient vector built in the image acquisition card, the flat field correction image is reprojected to obtain a preprocessed image sequence, and the preprocessed image sequence is stored in an image buffer of the image acquisition card.

[0023] Specifically, the input optical signal is collected by the sensor array of the image acquisition card at a preset sampling frequency, and the acquisition timestamp of each frame of the image is recorded to ensure that the image data correctly corresponds to the changes on the time axis during subsequent processing. The image acquisition card contains one or more high-precision CMOS or CCD sensor arrays, which convert optical signals into electrical signals and digitize the analog signals through A / D converters to obtain raw image data. During the acquisition process, the sampling frequency is set. For example, in industrial inspection scenarios, a sampling rate of 120Hz or higher is selected to ensure clear imaging of high-speed moving objects. In low-light environments, the sampling rate is reduced to increase the exposure time of a single frame, thereby improving the signal-to-noise ratio. At the same time, the acquisition timestamp of each frame of the image is recorded as ,in Represents the frame number. Perform nonlinear response correction on the original image data to eliminate the nonlinear effect of the photoelectric response of the sensor. The photoelectric response of the sensor is expressed as a nonlinear function: in Represents the grayscale value of the original image, Represents the actual incident light intensity at the pixel point, is the nonlinear response function of the sensor. In order to convert it into a linear response image, solve the inverse transform: in Determined by experiment and stored in the image acquisition card in the form of a lookup table or polynomial approximation. Perform bad pixel detection and repair on linear response images. Set a first threshold , the threshold is taken as the dynamic range of the original image gray value to 0.5%, and then calculate the neighborhood mean of each pixel: And identify all points where the pixel values ​​deviate from the mean by more than the threshold, that is, satisfying: The pixel points will be judged as bad pixels. In order to repair these ring points, a four-way bilinear interpolation algorithm is used, which considers the neighboring pixel values ​​in four directions around the bad pixel. , , , , and calculate the interpolation correction value as: in is the directional weighting factor, which is set to be inversely proportional to the distance from the bad pixel to the adjacent pixel to ensure the smoothness of the interpolation result. After this step, a bad pixel-free image is obtained, which effectively improves the image quality and reduces noise interference. Dark field correction is performed on the bad pixel-free image to eliminate the fixed noise of the sensor. Dark field noise mainly comes from the dark current of the sensor, which still exists in the absence of light and increases with the increase of temperature. A dark field image is pre-stored in the memory of the image acquisition card , which is acquired in a completely dark environment, represents the inherent noise distribution of the sensor. By subtracting the dark field image from the image without bad pixels, we get the dark field corrected image: This eliminates the fixed noise of the sensor and makes the image grayscale distribution more uniform. After completing the dark field correction, flat field correction is performed to compensate for the non-uniform response of the optical system. Due to factors such as lens transmittance and uneven sensor pixel response, different areas of the image have uneven brightness. A normalized flat field image is pre-stored in the acquisition card memory. , which is acquired under uniform illumination and normalized. The flat-field corrected image is obtained by dividing the dark-field corrected image by the normalized flat-field image: This ensures that the brightness distribution of the image is consistent throughout the field of view, improving the accuracy of subsequent analysis. In order to eliminate the geometric deformation caused by lens distortion, the flat-field corrected image is reprojected based on the distortion correction parameter matrix and distortion coefficient vector built into the image acquisition card. Lens distortion usually includes radial distortion and tangential distortion, and its mathematical model is expressed as: in are the uncorrected image coordinates, is the coordinate of the corrected image, is the radial distance, is the radial distortion coefficient, is the tangential distortion coefficient. Through this distortion correction model, the image is reprojected to eliminate the geometric distortion caused by lens deformation and obtain the final preprocessed image sequence After preprocessing, all processed images will be stored in the image buffer of the image acquisition card for subsequent timing analysis, noise estimation and deep learning processing. The entire preprocessing process is performed by a dedicated hardware acceleration unit inside the acquisition card to ensure that the processing delay is controlled within the frame period.

[0024] In a specific embodiment, the process of executing step 102 may specifically include the following steps: Each frame of the preprocessed image sequence is represented as a state vector and a measurement vector, a state transfer equation and a measurement equation are established, and an image state space model is obtained; The inverse Euler method is applied to the measurement noise in the image state space model to establish a discrete color noise model; The state vector and the noise in the discrete colored noise model are combined into an extended state vector, and an extended state equation and an extended measurement equation are constructed; Calculate the conditional expectation based on the current parameter estimate and complete step E of the expectation maximization algorithm; The forward-backward recursive calculation is performed using the smoothed Kalman filter to obtain the state estimate and error covariance, and the noise covariance matrix is ​​updated by maximization to complete the M step of the expectation maximization algorithm; Iteratively executing the expectation maximization algorithm until the parameter change is less than a preset second threshold value, thereby obtaining a noise statistical parameter; The noise statistical parameters are used to perform preliminary denoising on the preprocessed image sequence to obtain a bidirectional temporal feature tensor.

[0025] Specifically, each frame of the preprocessed image sequence is represented as a state vector and a measurement vector, and the state transfer equation and the measurement equation are constructed to describe the evolution of the image in the time series. Assume that each frame of the image is represented by a state vector Indicates that Represents the frame number, Contains the main feature information of the image, such as the distribution of pixel values, edge information, texture features, etc. At the same time, the measurement vector Represents the observation data obtained by the image acquisition card, which is different from the actual state There is a certain amount of noise interference between them, so the following state transfer equation and measurement equation are established: in, is the state transfer matrix, which describes how the current frame image evolves from the previous frame. is process noise, reflecting the uncertainty of the image state in time. is the observation matrix, which defines the relationship between the measured value and the true state. To describe the error in the image acquisition process, we need to measure the noise. Since the noise of the image acquisition card has time correlation, it cannot be simply assumed to be independent Gaussian noise. Therefore, we further model the measurement noise to establish a color noise model that is more in line with the actual situation. Modeling is performed and the inverse Euler method is applied to establish a discrete color noise model. Assume that the measurement noise obeys the following dynamic equation in continuous time: in, is the variation of noise over time, is the noise attenuation factor, which determines the temporal correlation of the noise, is the system noise driving matrix, is zero-mean white noise. In order to discretize the continuous-time noise model, the inverse Euler method is used for discretization, that is, the time derivative is discretely approximated to obtain a discrete color noise model: in, is the sampling time interval, is the unit matrix. This formula describes how the measurement noise evolves over time and combines the attenuation characteristics of the noise to more accurately simulate the measurement noise in the real acquisition system. In order to accurately consider the impact of noise in the state estimation process, the state vector and the measurement noise are combined into an extended state vector, and the extended state equation and extended measurement equation are constructed. The extended state vector is defined as: The extended state equation is composed of the original state equation and the colored noise model: The extended measurement equation is expressed as: After building the extended state space model, the conditional expectation is calculated based on the current parameter estimates to complete the E step of the expectation maximization (EM) algorithm. In the E step, the goal is to calculate the expected likelihood function of the data based on the current estimated noise statistical parameters. The parameter estimates are defined as ,in is the noise covariance matrix, then the goal of step E is to calculate the following expected value: in, Represents the joint probability distribution of observed data and hidden state. This expected value is used to optimize the noise statistical parameters and ensure that the subsequent Kalman filter can accurately describe the evolution of noise. After completing step E, the smoothed Kalman filter is used to perform forward-backward recursive calculations to obtain the optimal state estimate, and the noise covariance matrix is ​​updated by maximization to complete step M. During the forward recursive process, the state prediction calculation is performed: And calculate the forecast covariance matrix: Calculate the Kalman gain to measure the confidence between the current observation data and the predicted state: Update the state estimate using the observed data: And update the error covariance matrix: Through backward recursion, the state estimate is smoothed to improve the final denoising accuracy and calculate the new noise covariance matrix , which is used to update the parameter estimates and enter the next iteration of the EM algorithm. This process is repeated until the parameter change is less than the preset convergence threshold. ,Right now: Finally, the optimal noise statistical parameters are obtained, and these parameters are used to perform preliminary denoising on the preprocessed image sequence to obtain the bidirectional time series feature tensor .

[0026] In a specific embodiment, the execution step uses the noise statistical parameters to perform preliminary denoising on the preprocessed image sequence to obtain the bidirectional time series feature tensor, which may specifically include the following steps: Perform denoising filtering on the preprocessed image sequence according to noise statistical parameters to obtain a denoised image sequence; For each central time point in the denoised image sequence, a forward time window and a backward time window are constructed. The forward time window is arranged in positive time order, and the backward time window is arranged in reverse time order, thus obtaining a bidirectional sliding window sequence. Performing a multi-scale pyramid transform on each frame image in the bidirectional sliding window sequence to obtain a multi-scale pyramid image sequence, and dividing each layer of the multi-scale pyramid image sequence into non-overlapping spatial blocks to obtain a block-divided image; For each spatial block of the block-divided image, the mean, standard deviation, maximum value, minimum value and gradient histogram of the pixels in the block are calculated, and combined with the block-divided image to obtain enhanced block features; The enhanced block features are reorganized into a forward temporal feature tensor and a backward temporal feature tensor in chronological order to obtain a bidirectional temporal feature tensor.

[0027] Specifically, the noise covariance matrix calculated in the previous stage is used to denoise the image sequence to obtain a more stable image representation. Considering that the measurement noise is correlated in the time domain, an adaptive denoising method based on temporal filtering is used to maintain image details and reduce the cumulative impact of temporal noise. Represents the first frame image, where is the pixel coordinate, then the filtered image sequence Calculated by: in, is a time series filter based on noise statistical parameters, whose calculation depends on the measurement noise covariance matrix and the time-series correlation weight matrix In order to avoid the loss of image edge information, a weighted filtering strategy is adopted to prevent the areas with large image gradient changes from being over-smoothed during the time series filtering process, so as to obtain a clearer denoised image sequence. After obtaining the denoised image sequence, a forward time series window and a backward time series window are constructed to form a bidirectional sliding window sequence. For each central time point , define the forward timing window and the backward timing window ,in: in is the half-width of the sliding window, which represents the coverage of the time series window. The images in the forward window are arranged in chronological order, while the images in the backward window are arranged in chronological reverse order. This approach ensures that the bidirectional information is fully utilized in the subsequent processing, and the system pays attention to both the past and future information in the time series to more accurately model the time series features. After constructing the bidirectional sliding window sequence, a multi-scale pyramid transform is performed on each frame of the image to extract image features at different scales. The pyramid transform converts the original image into multiple image layers of different resolutions by downsampling step by step, so that the system can analyze the image structure at different scales. For each frame of image , multi-scale pyramid transform generation Layer image representation, where the The image representation of the layer is: in For the scale reduction operation, Gaussian blur plus downsampling method is used to ensure that the image still maintains local structural information during the downscaling process. In this way, a series of image layers from the original resolution to a lower resolution are generated to obtain a multi-scale pyramid image sequence. Each layer of the multi-scale pyramid image sequence is divided into non-overlapping spatial blocks to generate a block partitioned image. Each layer of the image is divided into spatial blocks of equal size, and the size of each block is set to pixels to ensure that the statistical features inside the block can be accurately calculated. Representing images In the The pixel set in the block calculates the mean, standard deviation, maximum, minimum and gradient histogram of the block, where the gradient histogram The calculation is based on the local gradient direction distribution and is used to capture texture feature information. By calculating these statistical features and combining them with the original block partitioned image, the enhanced block features are obtained. The enhanced block features are reorganized in time order to form a forward temporal feature tensor and a backward temporal feature tensor. Forward temporal feature tensor and the backward temporal feature tensor is defined as follows: in Indicates The frame image is Enhanced block features in layer pyramid, and Represent the forward and backward time series feature sequences respectively. In this way, it is ensured that the time series feature tensor contains bidirectional time series information at the same time.

[0028] In a specific embodiment, the process of executing step 103 may specifically include the following steps: Inputting the forward time series feature tensor in the bidirectional time series feature tensor into the forward branch of the bidirectional time convolutional neural network to obtain a first initial feature map, and inputting the backward time series feature tensor in the bidirectional time series feature tensor into the backward branch of the bidirectional time convolutional neural network to obtain a second initial feature map; Processing the first initial feature map through multiple temporal convolution blocks to obtain a first multi-layer feature map of a forward branch, and processing the second initial feature map through multiple temporal convolution blocks to obtain a second multi-layer feature map of a backward branch; Applying an attention module to the first multi-layer feature map and the second multi-layer feature map after each temporal convolution block, calculating attention weights and performing weighted fusion to obtain a first attention weighted feature and a second attention weighted feature; Perform an additive fusion operation on the first attention weighted feature of the last layer of the forward branch and the second attention weighted feature of the last layer of the backward branch to obtain a fused feature; The fused features are reduced in dimension and refined through a three-layer convolutional network to obtain comprehensive spatiotemporal features and calculate the confidence map.

[0029] Specifically, the bidirectional time series feature tensor is split into a forward time series feature tensor and a backward time series feature tensor, and input into the forward branch and the backward branch of the neural network respectively to extract the time series information and perform fusion. and the backward temporal feature tensor Represent the local block-level features at different times, which contain the temporal information of the image. It is passed as input to the forward branch of the bidirectional temporal convolutional neural network to obtain the first initial feature map, and the backward temporal feature tensor It is passed as input to the backward branch of the bidirectional temporal convolutional neural network to obtain the second initial feature map. After obtaining the initial feature map, it is processed by multiple temporal convolution blocks to obtain a deeper temporal feature representation. Each temporal convolution block consists of dilated convolution, batch normalization, and nonlinear activation function, and a residual connection is used to maintain the stable propagation of the gradient. Let represents the expansion factor, and Respectively represent the forward branch and the backward branch in The convolution kernel of the layer The forward feature map of a layer is calculated as follows: in, Represents a nonlinear activation function (such as ReLU or Swish), * represents a convolution operation, and the introduction of the expansion factor enables the convolution to capture contextual information over a longer time range. Similarly, the backward feature map is calculated as: In each temporal convolutional block, an attention module is applied to the forward and backward multi-layer feature maps to calculate the attention weights and perform weighted fusion. The role of the attention module is to enhance important temporal features while reducing the influence of irrelevant information. and Represent the attention weights of the forward and backward branches respectively, and the attention calculation is expressed as: in, is the attention mapping matrix, which is used to calculate the attention score. The calculated attention weights are used to weight the forward and backward features to obtain the first attention weighted features and the second attention weighted features. The first attention weighted features of the last layer of the forward branch and the second attention weighted features of the last layer of the backward branch are additively fused to integrate the bidirectional temporal information. The additive fusion method is: in, To fuse the weights, the adaptive optimization is used to adjust them to ensure that the fused features can achieve the best balance between global structure and local details. After the feature fusion is completed, the fused features are reduced in dimension and refined through a three-layer convolutional network to obtain comprehensive spatiotemporal features. The goal of dimensionality reduction is to remove redundant information while enhancing the spatiotemporal consistency of the image, and finally generate a high-dimensional feature representation that can characterize local structural information and global temporal relationships. In order to evaluate the confidence of different regions, the confidence map is calculated, which can reflect the feature credibility of the image at different spatial positions and serve as the weight parameter for subsequent image fusion. The calculation of confidence is based on temporal consistency and the stability of feature distribution, ensuring that high confidence regions are given higher weights in the subsequent processing, while low confidence regions are subjected to stronger denoising and enhancement processing, thereby ensuring the quality and stability of the final output image.

[0030] In a specific embodiment, the process of executing step 104 may specifically include the following steps: Define the image processing parameter space of the image acquisition card, which includes denoising intensity, sharpening coefficient, contrast gain, brightness adjustment, color saturation, detail enhancement intensity and texture preservation factor; The clarity index, noise level index, detail preservation index and color fidelity index are calculated based on the comprehensive spatiotemporal characteristics to construct an image quality evaluation matrix; The confidence map is used as a spatial weight to perform weighted calculation on the image quality assessment matrix to obtain a weighted quality score; Based on the difference between the weighted quality score and the ideal image quality index, the energy error is calculated to obtain the objective function of parameter optimization; According to the image processing parameter space, the objective function is iteratively optimized by applying the gradient descent method to obtain a parameter update sequence, and the parameter update sequence is iteratively calculated until the convergence condition is met to obtain the optimal processing parameters of the image acquisition card; The preprocessed image sequence is processed using the optimal processing parameters to generate the target output image.

[0031] Specifically, the image processing parameter space of the image acquisition card is defined, which includes denoising intensity, sharpening coefficient, contrast gain, brightness adjustment, color saturation, detail enhancement intensity and texture preservation factor. Let the parameter vector be expressed as: in, Represents the denoising intensity, which is used to control the smoothness of the filter operator. Represents the sharpening factor, which determines the strength of edge enhancement. is the contrast gain, which is used to adjust the difference between dark and bright areas. Represents brightness adjustment, so that the overall brightness distribution of the image is balanced. Represents color saturation, which affects the vividness of the color. Represents detail enhancement strength, used to enhance local texture information. Represents the texture preservation factor, which is used to balance the relationship between denoising and detail preservation. These parameters together constitute the image processing parameter space and are dynamically adjusted in the subsequent optimization process to ensure the best quality of the final output image. After determining the parameter space, the clarity index, noise level index, detail preservation index and color fidelity index are calculated based on the comprehensive spatiotemporal characteristics to construct an image quality evaluation matrix. Let is the image quality assessment matrix, which contains four core quality assessment indicators, defined as follows: in, It is a clarity index, which measures the sharpness of the image by calculating the image gradient energy. is a noise level indicator, reflecting the random noise intensity in the image. As a detail preservation index, the image detail information retained during the denoising process is measured by calculating the structural similarity index. is the color fidelity index, which indicates the degree of color restoration of the image. Based on these indicators, the quality performance of the image under different optimization parameters is evaluated. In order to improve the evaluation accuracy, the image quality evaluation matrix is ​​weighted according to the confidence map as the spatial weight to obtain the weighted quality score. Let the confidence map be , indicating the importance of different regions in the image, the weighted quality score The calculation is as follows: in, It reflects the contribution weight of the pixel in the overall image quality assessment. This process ensures that the high confidence area has a greater impact on the final score, while the low confidence area has a smaller impact on the optimization result, thereby improving the stability and robustness of the optimization. After calculating the weighted quality score, the optimization objective function is constructed to make the image quality score as close as possible to the ideal image quality index. Let the ideal quality vector be , then the energy error of the optimization objective function is defined as follows: The objective function represents the gap between the current image quality and the ideal image quality. The optimization goal is to adjust the parameter vector make Minimize, thereby improving image quality. In order to minimize the objective function , it is necessary to perform iterative optimization based on the gradient descent method. Indicates The parameter vector of the iteration is optimized and updated as follows: in, is the learning rate, which controls the step size of parameter updates, is the gradient of the objective function with respect to the parameter vector, which is calculated by numerical differentiation or back propagation. During the optimization process, the learning rate Dynamic adjustments are made based on the stability of parameter updates to avoid oscillation or slow convergence. The optimization process continues to iterate until the convergence condition is met, that is, when the parameter change amplitude of two consecutive iterations is less than the preset threshold When , the optimization is stopped and the optimal processing parameters of the image acquisition card are obtained. Let the final optimized parameter vector be , then the optimal parameters meet the following conditions: After optimization is completed, the optimal processing parameters are used Process the preprocessed image sequence to generate the final target output image. For each frame of preprocessed image , based on the optimal parameters, calculate the image after denoising, sharpening, contrast enhancement, etc., and obtain the final output image : in, It is an image processing operator, including optimal denoising filtering, contrast enhancement, detail sharpening and other operations to ensure that the quality of the final image reaches the optimal state.

[0032] In a specific embodiment, the process of executing the step of processing the pre-processed image sequence using the optimal processing parameters to generate the target output image may specifically include the following steps: Applying image processing operators to the preprocessed image sequence based on the optimal processing parameters to obtain images with optimized parameters; An extended state Kalman filter is constructed based on noise statistical parameters, and state prediction, Kalman gain calculation and state update steps are performed on the preprocessed image sequence to obtain a filtered image; The confidence map is used to calculate the fusion weight, and an adaptive fusion operation is performed on the parameter-optimized image and the filtered image to obtain a fused image; The comprehensive spatiotemporal features are mapped to the enhancement parameter space, and the fused image is spatially adaptively enhanced to obtain an enhanced image; Performing global color correction on the enhanced image, adjusting the hue, saturation and brightness of the image, and obtaining a color-corrected image; Apply an edge-preserving filter to the color-corrected image and quantize the filtered image to the frame grabber output format to obtain the target output image.

[0033] Specifically, the preprocessed image sequence is calculated based on the optimal processing parameters to generate an image with optimized parameters. ,in Represents the frame number, represents the pixel coordinates, and the optimal processing parameter vector is , then the optimized image is expressed by the parameter adaptive adjustment model as: in, Represents image processing operators, including denoising, sharpening, contrast enhancement, and other operations, and optimal parameters The weights of different operators are controlled so that the image can maintain the best quality under different shooting environments. An extended state Kalman filter is constructed based on noise statistical parameters to perform state prediction, Kalman gain calculation and state update steps on the preprocessed image sequence to obtain the filtered image. In the specific implementation process, let the image state vector be , then the state prediction process can be expressed as in, is the state transfer matrix, is the process noise, which is determined by the temporal correlation of the image sequence. The core of the filter is to calculate the Kalman gain , the gain is used to adjust the filtering weights to ensure that the image is optimized under the constraints of the timing information, so that the noise is suppressed and the details are preserved. After obtaining the parameter-optimized image and the filtered image, the confidence map is used to calculate the fusion weights, and an adaptive fusion operation is performed to generate the final fused image. Confidence map To reflect the reliability of different pixels, the fusion calculation adopts a weighted method, which is defined as follows: Among them, a higher confidence value means that the pixels in this area are more reliable, and the image after parameter optimization is given a higher weight in the fusion process, while the lower confidence area relies more on the filtering result to ensure the global consistency and detail retention of the final image. Based on the fused image, spatial adaptive enhancement is performed to optimize the image quality. Map to the enhanced parameter space and calculate the enhanced parameters Finally, a nonlinear enhancement transformation is performed on the fused image to obtain an enhanced image. In order to ensure that the enhancement process conforms to the human visual perception model, the contrast perception enhancement model is adopted: in, Controlled by spatiotemporal characteristics, To enhance the strength factor, Make the enhancement process smoother, adaptively optimizing contrast and texture clarity in different areas. Perform global color correction on the enhanced image to adjust hue, saturation, and brightness, and apply an edge-preserving filter to preserve structural information while denoising. Color correction uses a global transformation matrix , while the edge preserving filter It provides smoothing in flat areas and maintains high-frequency information in edge areas, so that the overall visual quality of the image reaches the optimal level. The quantized target output image meets the needs of different application scenarios.

[0034] The above describes the image processing method of the image acquisition card in the embodiment of the present invention. The following describes the image processing device of the image acquisition card in the embodiment of the present invention. Figure 2 , an embodiment of the image processing device of the image acquisition card in the embodiment of the present invention includes: The preprocessing module 201 is used to convert the input optical signal into raw image data through the sensor array of the image acquisition card, and perform preprocessing to obtain a preprocessed image sequence; A preliminary denoising module 202 is used to construct an image state space model and a color measurement noise model based on the preprocessed image sequence, perform preliminary denoising on the preprocessed image sequence, and obtain a bidirectional time series feature tensor; A feature extraction module 203 is used to input the bidirectional time series feature tensor into the additive fusion bidirectional time convolutional neural network for feature extraction to obtain comprehensive spatiotemporal features and confidence maps; The generation module 204 is used to perform gradient descent image processing parameter optimization based on the comprehensive spatiotemporal features and the confidence map, obtain the optimal processing parameters of the image acquisition card, and generate a target output image.

[0035] Through the synergy of the above components, by constructing a color measurement noise model and estimating noise statistical parameters in combination with the expectation maximization algorithm, the limitations of the traditional white noise assumption are overcome, the temporal correlation characteristics of noise in the image acquisition card can be accurately captured, and the image processing quality in complex noise environments is significantly improved. The forward and backward time series are constructed by the bidirectional temporal sliding window method. Rich spatiotemporal features are extracted through multi-scale pyramid transformation and spatial block partitioning, so that the image processing system can simultaneously capture the time series information from the past to the present and from the future to the present, greatly improving the image clarity and detail retention ability in dynamic scenes. The image quality assessment matrix constructed based on the comprehensive spatiotemporal features and confidence maps is combined with the gradient descent method to achieve automatic optimization of image processing parameters, which completely solves the subjective problem of traditional methods relying on manual experience judgment, so that the image acquisition card can intelligently adjust the processing parameters according to the characteristics of different scenes. The additive fusion bidirectional temporal convolutional neural network greatly reduces the amount of calculation parameters through dilated convolution and attention mechanism, while maintaining a large receptive field, so that the algorithm can run efficiently on resource-constrained image acquisition cards, and the processing delay is controlled within one frame cycle to meet real-time processing requirements. The adaptive fusion strategy based on confidence map effectively combines the advantages of parameter optimization and Kalman filtering, automatically adjusts the processing strategy according to the feature reliability of different regions, improves the robustness of image processing, and significantly improves the processing effect in areas with uneven lighting.

[0036] above Figure 2 The image processing device of the image acquisition card in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The image processing device of the image acquisition card in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0037] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0038] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0039] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing method for an image acquisition card, characterized in that: The method comprises: The input optical signal is converted into raw image data through the sensor array of the image acquisition card, and preprocessing is performed to obtain a preprocessed image sequence; Building an image state space model and a color measurement noise model based on the preprocessed image sequence, performing preliminary denoising on the preprocessed image sequence, and obtaining a bidirectional time series feature tensor; Inputting the bidirectional time series feature tensor into an additively fused bidirectional temporal convolutional neural network for feature extraction to obtain comprehensive spatiotemporal features and a confidence map; The image processing parameters are optimized by gradient descent according to the comprehensive spatiotemporal features and the confidence map to obtain the optimal processing parameters of the image acquisition card and generate a target output image.

2. The image processing method of the image acquisition card according to claim 1, characterized in that: The sensor array of the image acquisition card converts the input optical signal into raw image data and performs preprocessing to obtain a preprocessed image sequence, including: The input optical signal is collected by the sensor array of the image acquisition card at a preset sampling frequency, and the acquisition timestamp of each frame of the image is recorded to obtain the original image data; Performing nonlinear response correction on the original image data to obtain a linear response image; Setting a first threshold for the linear response image, identifying pixel points whose pixel values ​​deviate from the neighborhood mean by more than the first threshold, and replacing the pixel points using a four-directional bilinear interpolation algorithm to obtain a defect-free image; Subtracting a dark field image pre-stored in the memory of the image acquisition card from the image without bad pixels to obtain a dark field correction image, and dividing the dark field correction image by a pre-stored normalized flat field image to obtain a flat field correction image; Based on the distortion correction parameter matrix and distortion coefficient vector built into the image acquisition card, the flat field corrected image is reprojected to obtain a preprocessed image sequence, and the preprocessed image sequence is stored in an image buffer of the image acquisition card.

3. The image processing method of the image acquisition card according to claim 2, characterized in that: The step of constructing an image state space model and a color measurement noise model based on the preprocessed image sequence, performing preliminary denoising on the preprocessed image sequence, and obtaining a bidirectional time series feature tensor includes: Representing each frame of the preprocessed image sequence as a state vector and a measurement vector, establishing a state transfer equation and a measurement equation, and obtaining an image state space model; Applying the inverse Euler method to the measurement noise in the image state space model to establish a discrete color noise model; Combining the state vector and the noise in the discrete color noise model into an extended state vector, and constructing an extended state equation and an extended measurement equation; Calculate the conditional expectation based on the current parameter estimate and complete step E of the expectation maximization algorithm; The forward-backward recursive calculation is performed using the smoothed Kalman filter to obtain the state estimate and error covariance, and the noise covariance matrix is ​​updated by maximization to complete the M step of the expectation maximization algorithm; Iteratively executing the expectation maximization algorithm until the parameter change is less than a preset second threshold, thereby obtaining a noise statistical parameter; The noise statistical parameters are used to perform preliminary denoising on the preprocessed image sequence to obtain a bidirectional temporal feature tensor.

4. The image processing method of the image acquisition card according to claim 3, characterized in that: The method of performing preliminary denoising on the preprocessed image sequence by using the noise statistical parameters to obtain a bidirectional time series feature tensor includes: Performing denoising filtering on the preprocessed image sequence according to the noise statistical parameters to obtain a denoised image sequence; Constructing a forward time sequence window and a backward time sequence window for each central time point in the denoised image sequence, wherein the forward time sequence window is arranged in a positive time sequence and the backward time sequence window is arranged in a reverse time sequence, to obtain a bidirectional sliding window sequence; Performing a multiscale pyramid transform on each frame of the image in the bidirectional sliding window sequence to obtain a multiscale pyramid image sequence, and dividing each layer of the image in the multiscale pyramid image sequence into non-overlapping spatial blocks to obtain a block-divided image; For each spatial block of the block-divided image, a mean value, a standard deviation, a maximum value, a minimum value and a gradient histogram of pixels in the block are calculated, and the enhanced block features are obtained by combining the block-divided image with the block-divided image; The enhanced block features are reorganized into a forward temporal feature tensor and a backward temporal feature tensor in time sequence to obtain a bidirectional temporal feature tensor.

5. The image processing method of the image acquisition card according to claim 4, characterized in that: The bidirectional time series feature tensor is input into the additive fusion bidirectional time convolutional neural network for feature extraction to obtain comprehensive spatiotemporal features and confidence maps, including: Inputting the forward time series feature tensor in the bidirectional time series feature tensor into the forward branch of the bidirectional time convolutional neural network to obtain a first initial feature map, and inputting the backward time series feature tensor in the bidirectional time series feature tensor into the backward branch of the bidirectional time convolutional neural network to obtain a second initial feature map; Processing the first initial feature map through multiple temporal convolution blocks to obtain a first multi-layer feature map of a forward branch, and processing the second initial feature map through multiple temporal convolution blocks to obtain a second multi-layer feature map of a backward branch; Applying an attention module to the first multi-layer feature map and the second multi-layer feature map after each temporal convolution block, calculating attention weights and performing weighted fusion to obtain a first attention weighted feature and a second attention weighted feature; Performing an additive fusion operation on the first attention weighted feature of the last layer of the forward branch and the second attention weighted feature of the last layer of the backward branch to obtain a fused feature; The fusion features are reduced in dimension and refined through a three-layer convolutional network to obtain comprehensive spatiotemporal features, and the confidence map is calculated.

6. The image processing method of the image acquisition card according to claim 5, characterized in that: The image processing parameter optimization by gradient descent according to the comprehensive spatiotemporal features and the confidence map, obtaining the optimal processing parameters of the image acquisition card, and generating a target output image, comprises: Defining an image processing parameter space of the image acquisition card, wherein the image processing parameter space includes denoising intensity, sharpening coefficient, contrast gain, brightness adjustment, color saturation, detail enhancement intensity and texture preservation factor; Calculating a clarity index, a noise level index, a detail preservation index, and a color fidelity index based on the comprehensive spatiotemporal features, and constructing an image quality assessment matrix; Performing weighted calculation on the image quality assessment matrix according to the confidence map as a spatial weight to obtain a weighted quality score; Based on the difference between the weighted quality score and the ideal image quality index, an energy error is calculated to obtain an objective function for parameter optimization; According to the image processing parameter space, the objective function is iteratively optimized by applying a gradient descent method to obtain a parameter update sequence, and the parameter update sequence is iteratively calculated until a convergence condition is met to obtain the optimal processing parameters of the image acquisition card; The pre-processed image sequence is processed using the optimal processing parameters to generate a target output image.

7. The image processing method of the image acquisition card according to claim 6, characterized in that: The step of processing the pre-processed image sequence using the optimal processing parameters to generate a target output image includes: Applying an image processing operator to the preprocessed image sequence based on the optimal processing parameters to obtain an image after parameter optimization; Constructing an extended state Kalman filter based on the noise statistical parameters, performing state prediction, Kalman gain calculation and state update steps on the preprocessed image sequence to obtain a filtered image; Calculating fusion weights using the confidence map, performing adaptive fusion operations on the parameter-optimized image and the filtered image to obtain a fused image; Mapping the comprehensive spatiotemporal features to an enhanced parameter space, and performing spatial adaptive enhancement on the fused image to obtain an enhanced image; Performing global color correction on the enhanced image to adjust the hue, saturation and brightness of the image to obtain a color-corrected image; An edge-preserving filter is applied to the color-corrected image, and the filtered image is quantized into an image acquisition card output format to obtain a target output image.

8. An image processing device for an image acquisition card, characterized in that: Used to execute the image processing method of the image acquisition card according to any one of claims 1 to 7, the image processing device of the image acquisition card comprises: A preprocessing module, used to convert the input optical signal into raw image data through the sensor array of the image acquisition card, and perform preprocessing to obtain a preprocessed image sequence; A preliminary denoising module, used for constructing an image state space model and a color measurement noise model based on the preprocessed image sequence, performing preliminary denoising on the preprocessed image sequence, and obtaining a bidirectional time series feature tensor; A feature extraction module, used for inputting the bidirectional time series feature tensor into an additive fusion bidirectional temporal convolutional neural network for feature extraction to obtain comprehensive spatiotemporal features and a confidence map; A generation module is used to perform gradient descent image processing parameter optimization according to the comprehensive spatiotemporal features and the confidence map, obtain the optimal processing parameters of the image acquisition card, and generate a target output image.

Images