Image detail recovery method based on attention mechanism in vehicle-mounted night vision system

Abstract

This invention discloses an image detail restoration method based on an attention mechanism in a vehicle-mounted night vision system, comprising: acquiring an image tensor of a low-light night vision image; inputting the image tensor into a pre-trained image generation model and performing feature mapping on it; generating a detail-restored image at the output layer according to the mapping parameters of the feature mapping; wherein, the mapping parameters of the feature mapping represent the color matrix and gamma value of the sRGB standard image learned by the image generation model in pre-training; this invention overcomes the shortcomings of existing image enhancement methods by combining local features extracted by convolutional layers with global dependency features captured by self-attention layers; when processing low-light images, convolutional layers focus on detail restoration, while self-attention layers can capture the relationship between distant pixels in the image, thereby effectively restoring global information, so that the restored image is enhanced in both detail and global structure.

Description

Technical Field

[0001] This invention relates to the field of image enhancement, specifically to an image detail recovery method based on an attention mechanism in vehicle-mounted night vision systems. Background Technology

[0002] With the development of vehicle-mounted night vision systems, image processing in low-light environments has become an important research direction. Traditional image enhancement methods mostly rely on simple global processing techniques, such as histogram equalization, contrast enhancement, and color adjustment. However, these methods often fail to effectively recover image details under low-light conditions. Especially for complex lighting changes in nighttime driving scenarios, existing technologies are prone to causing the loss of local image details or excessive global brightness adjustment, thereby affecting the driver's visual experience and road safety. Patent document CN119130876A discloses a high-quality image reconstruction method, which uses PSNR (Peak Signal-to-Noise Ratio) and Structural Similarity (SSIM) to evaluate the quality of the restored image, thereby obtaining an optimal convolutional neural network model. The obtained optimal convolutional neural network model is then applied to perform high-quality image reconstruction processing on CIS image data to obtain a high-quality image, which is then transmitted to a display device.

[0003] However, while convolutional neural network (CNN)-based image restoration methods can extract local features, they typically lack effective modeling of the global image structure and dependencies between distant pixels. This prevents both global consistency and detail restoration from achieving high levels simultaneously. Although self-attention-based models can effectively capture global dependencies, they still suffer from insufficient accuracy in detail restoration under low-light conditions and cannot adequately handle the enhancement of local details. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an image detail recovery method based on an attention mechanism in vehicle-mounted night vision systems. This method solves the technical problems mentioned in the background by constructing mutually independent convolutional layers and feature fusion after self-attention layers.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] Image detail recovery methods based on attention mechanisms in vehicle-mounted night vision systems include:

[0007] S1. Obtain the image tensor of the low-light night view image;

[0008] S2. Input the image tensor into a pre-trained image generation model and perform feature mapping on it;

[0009] S3. Generate a detail restoration image in the output layer according to the mapping parameters of the feature mapping; wherein, the mapping parameters of the feature mapping are characterized as the color matrix and gamma value of the sRGB standard image learned by the image generation model in pre-training.

[0010] In some embodiments, the input layer of the image generation model is configured with a dual-channel architecture that independently connects convolutional layers and self-attention layers.

[0011] In some embodiments, the image tensor is input into a pre-trained image generation model, and feature mapping is performed on it, including:

[0012] S2-1. Receive the image tensor as input to the convolutional layer and the self-attention layer, and extract its local features and global dependency features respectively;

[0013] S2-2. Perform pixel-wise multiplication on local features using the pre-trained multiplication graph to generate contrast-optimized local features; where the multiplication graph is represented as the set of contrasts of the sRGB standard image learned by the image generation model during pre-training.

[0014] S2-3. The global dependent features are added pixel by pixel using the pre-trained additive graph to generate global features optimized for brightness; where the additive graph represents the set of brightness values ​​of the sRGB standard image learned by the image generation model during pre-training.

[0015] S2-4. The local features optimized for contrast and the global features optimized for brightness are fused to generate optimized global features;

[0016] S2-5. The optimized global features are input into the fully connected layer, and a detail restoration image is generated in the output layer through forward propagation.

[0017] In some embodiments, the feature fusion is to concatenate contrast-optimized local features and brightness-optimized global features into vectors.

[0018] In some embodiments, the pre-training step of the image generation model includes:

[0019] A1. Obtain an image pair sample set; wherein, the image pair sample consists of raw-RGB images and sRGB standard images;

[0020] A2. Construct the initial network architecture for the image generation model;

[0021] A3. Define the overall loss function for the image generation model;

[0022] A4. Use the images to perform iterative supervised learning on the sample set within the initial network architecture until the defined comprehensive loss function minimizes the comprehensive loss;

[0023] A5. If the overall loss is minimized, then the initial network architecture is defined as the image generation model.

[0024] In some embodiments, obtaining an image pair sample set includes:

[0025] A1-1. Acquire the raw-RGB image from the image sensor before image signal processing;

[0026] A1-2. Perform image signal processing on the raw-RGB image to generate a corresponding sRGB standard image; wherein, the image signal processing includes: a predefined optimal parameter set for contrast, brightness, color matrix and gamma value;

[0027] A1-3. Using the raw-RGB image as input features and the corresponding sRGB standard image as target labels, construct image pair samples;

[0028] A1-4. Obtain several image pair samples to obtain the image pair sample set.

[0029] In some of these embodiments, the initial network architecture for constructing the image generation model includes:

[0030] A2-1. Select the initial model;

[0031] A2-2. Construct independent convolutional layers and self-attention layers after the input layer of the initial model;

[0032] A2-3. A first activation layer is connected after the convolutional layer; wherein, the first activation layer is used to learn the optimal contrast of the sRGB standard image, and the optimal contrast of the pixel set is defined as the multiplication graph;

[0033] A2-4. A second activation layer is connected after the self-attention layer; wherein, the second activation layer is used to activate the learning of the optimal global brightness of the sRGB standard image, and the optimal global brightness of the pixel set is defined as an additive graph;

[0034] A2-5. Connect the first activation layer and the second activation layer to the feature fusion layer, and then connect the fully connected layer and the output layer in sequence after the feature fusion layer to construct the initial network architecture.

[0035] In some embodiments, a comprehensive loss function for the image generation model is defined, including:

[0036] A3-1. Define four types of sub-loss terms for the comprehensive loss function; the four types of sub-loss terms include: contrast loss, brightness loss, gamma value loss, and color matrix loss;

[0037] A3-2. Use mean square error as the loss function for contrast loss, brightness loss, and gamma value loss;

[0038] The contrast loss function is:

[0039] ;

[0040] in, Indicates contrast loss. This represents the contrast of the detail-recovery image at pixel (i, j). This represents the contrast of an sRGB standard image at pixel (i, j);

[0041] The brightness loss function is:

[0042] ;

[0043] in, Indicates brightness loss. This represents the brightness of the detail-recovery image at pixel (i, j). This represents the brightness of a pixel (i, j) in an sRGB standard image.

[0044] The gamma-value loss function is:

[0045] ;

[0046] in, Indicates gamma value loss. This represents the gamma value of the detail-recovered image at pixel (i, j). This represents the gamma value of a pixel (i, j) in an sRGB standard image.

[0047] A3-3. Use matrix angle loss as the loss function for color matrix loss;

[0048] The color matrix loss function is:

[0049] ;

[0050] in, Indicates the color matrix loss. M represents the color matrix of the detail-restored image, and M represents the color matrix of the sRGB standard image. The Frobenius norm of the color matrix of the image with detailed reconstruction is represented. The Frobenius norm represents the color matrix of an sRGB standard image. This represents the matrix dot product of the detail-restored image and the sRGB standard image;

[0051] A3-4. Construct the comprehensive loss function by combining the contrast loss, brightness loss, gamma value loss, and color matrix loss.

[0052] The expression for the comprehensive loss function is:

[0053] ;

[0054] in, Indicates the overall loss. , These represent the weighting coefficients for contrast loss, brightness loss, gamma value loss, and color matrix loss, respectively.

[0055] In some embodiments, the images are used to perform iterative supervised learning on the sample set within an initial network architecture until the defined comprehensive loss function minimizes the comprehensive loss, including:

[0056] A4-1. Extract the first round of raw-RGB images from the image sample set, propagate them forward, and generate the first round of detail recovery images in the output layer;

[0057] A4-2. Calculate the combined loss between the detail-restored image from the first round and the sRGB standard image from the first round;

[0058] A4-3. If the overall loss in the first round is not minimized, calculate the gradient of the overall loss in the first round with respect to the current model parameters.

[0059] A4-4. Based on the gradient of the current model parameters, update the model parameters in the opposite direction of the gradient with a preset learning rate to obtain the updated parameters.

[0060] A4-5. Extract the next round of raw-RGB images from the image sample set, propagate them forward using the updated parameters, and generate the next round of detail recovery images in the output layer.

[0061] A4-6. Calculate the combined loss between the detail-restored image in the next round and the sRGB standard image in the next round, until the combined loss function in the next round minimizes the combined loss.

[0062] In some embodiments, the model parameters are updated in the reverse direction of the gradient with a preset learning rate based on the gradient of the current model parameters to obtain the updated parameters, including:

[0063] A4-4-1. Randomly initialize model parameters;

[0064] A4-4-2. Calculate the gradient of contrast loss relative to the initialized model parameters in the first activation layer;

[0065] A4-4-3. Calculate the gradient of the brightness loss relative to the initialized model parameters in the second activation layer;

[0066] A4-4-4 Calculate the gradient of the color matrix and gamma values ​​relative to the initialized model parameters in the fully connected layer;

[0067] A4-4-5. Update the model parameters along the opposite direction of the gradient and the predefined learning rate;

[0068] A4-4-6. Repeat A4-4-2 to A4-4-5 until the overall loss is minimized;

[0069] A4-4-7. Define the model parameters corresponding to minimizing the comprehensive loss as the generation parameters of the image generation model;

[0070] The generation parameters of the first activation layer are represented by a matrix structure as a multiplication graph, and the generation parameters of the second activation layer are represented by a matrix structure as an addition graph.

[0071] This invention provides an image detail recovery method based on an attention mechanism in a vehicle-mounted night vision system, which has the following beneficial effects:

[0072] This invention overcomes the shortcomings of existing image enhancement methods by combining local features extracted by convolutional layers with global dependency features captured by self-attention layers. When processing low-light images, convolutional layers focus on restoring details such as edges and textures, while self-attention layers can capture the relationships between distant pixels in the image, thereby effectively restoring global information and enhancing both the details and global structure of the restored image.

[0073] Furthermore, this invention defines a comprehensive loss function that combines contrast, brightness, gamma value, and color matrix losses to precisely optimize various parameters of low-light images. By weighted optimization of different loss terms, it ensures a balanced performance of the detail-restored image in terms of contrast, brightness, color, and gamma value, thereby avoiding the problems of single optimization direction or over-enhancement that may occur in traditional methods. Attached Figure Description

[0074] Figure 1 This is a flowchart illustrating the image detail recovery method based on the attention mechanism in the vehicle-mounted night vision system of the present invention.

[0075] Figure 2 This is a network architecture diagram of the image generation model of the present invention;

[0076] Figure 3 This is a schematic diagram of the pre-training process of the image generation model of the present invention. Detailed Implementation

[0077] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0078] First, the prior art and related concepts involved in the embodiments of the present invention will be described:

[0079] Raw-RGB images: Raw RGB images are unprocessed data directly acquired from the image sensor. They contain the raw values ​​of the red (R), green (G), and blue (B) channels for each pixel, and are typically saved as RAW format files. This data has not undergone any ISP (Image Signal Processor) processing steps, such as white balance, color correction, gamma correction, etc.

[0080] sRGB standard images: sRGB standard images are images that have undergone a complete ISP processing flow and conform to the sRGB standard color space. These images can be directly displayed on most monitors and exhibit good color consistency across different devices. sRGB images undergo processing steps such as white balance correction, color correction, and gamma correction to ensure accurate colors and good visual effects.

[0081] Example 1: Please refer to Figures 1 to 2 This invention provides an image detail recovery method based on an attention mechanism in a vehicle-mounted night vision system, comprising the following steps:

[0082] S1. Obtain the image tensor of the low-light night view image;

[0083] S2. Input the image tensor into a pre-trained image generation model and perform feature mapping on it;

[0084] In this embodiment, the input layer of the image generation model is independently connected to the convolutional layer and the self-attention layer in a dual-channel architecture.

[0085] Step S2 further includes:

[0086] S2-1. Receive the image tensor as input to the convolutional layer and the self-attention layer, and extract its local features and global dependency features respectively;

[0087] S2-2. Perform pixel-wise multiplication on local features using the pre-trained multiplication graph to generate contrast-optimized local features; where the multiplication graph is represented as the set of contrasts of the sRGB standard image learned by the image generation model during pre-training.

[0088] S2-3. The global dependent features are added pixel by pixel using the pre-trained additive graph to generate global features optimized for brightness; where the additive graph represents the set of brightness values ​​of the sRGB standard image learned by the image generation model during pre-training.

[0089] S2-4. The local features optimized for contrast and the global features optimized for brightness are fused to generate optimized global features;

[0090] Furthermore, the feature fusion involves concatenating the contrast-optimized local features and the brightness-optimized global features into vectors.

[0091] The feature fusion method in this embodiment preserves information from both local and global features through vector concatenation, enabling the model to fully leverage the advantages of both, thereby improving the detail recovery capability and image quality of low-light images.

[0092] S2-5. The optimized global features are input into the fully connected layer, and a detail restoration image is generated in the output layer through forward propagation.

[0093] In this embodiment, the input layer of the image generation model adopts a dual-channel architecture, which is independently connected through convolutional layers and self-attention layers to simultaneously extract local features and global dependency features of the image. The convolutional layers are mainly responsible for extracting local detail information in the image, such as edges and textures, while the self-attention layers capture global dependencies in the image, especially the mutual influence between distant pixels.

[0094] Furthermore, after the image tensor is input into the pre-trained model, the local features extracted in the convolutional layers are first multiplied pixel-by-pixel through the pre-trained multiplication map to optimize the image contrast, thereby generating contrast-optimized local features. The multiplication map has already learned the contrast information of the sRGB standard image during pre-training and is applied to this operation. Next, the globally dependent features of the image are added pixel-by-pixel through the additive map to optimize the image brightness, where the additive map represents the brightness set of the sRGB standard image learned by the image generation model during pre-training.

[0095] After these two optimization steps, the local and global features are combined by the feature fusion layer to generate an optimized global feature. Finally, the optimized global feature is input into a fully connected layer, and after forward propagation, a detail-restored image is generated in the output layer, ultimately achieving detail restoration and brightness enhancement of the image.

[0096] This embodiment uses a dual-channel architecture to process local details and global information separately, which can effectively restore details in the image and optimize the image contrast and brightness, thereby improving the image quality and visibility in low-light environments in the vehicle night vision system and improving the visual experience of night driving.

[0097] S3. Generate a detail restoration image in the output layer according to the mapping parameters of the feature mapping; wherein, the mapping parameters of the feature mapping are characterized as the color matrix and gamma value of the sRGB standard image learned by the image generation model in pre-training.

[0098] In this embodiment, firstly, a low-light night vision image is acquired and converted into an image tensor, which serves as input data. Next, the image tensor is fed into a pre-trained image generation model trained on a large number of standard sRGB images, enabling it to effectively extract key features from low-light images. Within the image generation model, based on the mapping parameters (including the color matrix and gamma values) learned during pre-training, brightness and color are optimized in the low-light image, thereby generating a high-quality image with restored details, approaching the quality of a standard sRGB image.

[0099] This embodiment can restore the details and colors of an image by adjusting the mapping parameters of the image generation model, thereby providing a clearer, more natural and realistic image output in the vehicle night vision system, which greatly improves the visual experience and safety when driving at night.

[0100] Example 2: See Figure 3 The technical solution of this embodiment 2 differs from that of embodiment 1 in that it discloses the pre-training steps of the image generation model in embodiment 1, wherein the pre-training steps include:

[0101] A1. Obtain an image pair sample set; wherein, the image pair sample consists of raw-RGB images and sRGB standard images;

[0102] A2. Construct the initial network architecture for the image generation model;

[0103] A3. Define the overall loss function for the image generation model;

[0104] A4. Use the images to perform iterative supervised learning on the sample set within the initial network architecture until the defined comprehensive loss function minimizes the comprehensive loss;

[0105] A5. If the overall loss is minimized, then the initial network architecture is defined as the image generation model.

[0106] In this embodiment, the pre-training steps of the image generation model include multiple stages. First, the A1 stage acquires a set of image pairs, where each pair consists of a raw-RGB image and a corresponding sRGB standard image. These image pairs provide a mapping between low-light images (raw-RGB) and standard images (sRGB), allowing the model to learn how to recover low-light images to images under standard lighting.

[0107] Next, in stage A2, the initial network architecture for the image generation model, such as the Transformer, is constructed. This initial architecture provides the basic framework for subsequent training to handle image restoration tasks.

[0108] In the A3 stage, a comprehensive loss function is defined, which takes into account multiple factors such as image contrast, brightness, color matrix, and gamma value. The loss function is designed to evaluate the difference between the restored image and the target image and guide network optimization during training.

[0109] In stage A4, iterative supervised learning is performed on the sample set using images within the initial network architecture. The network is trained according to the loss function until the overall loss is minimized. During this process, the network gradually adjusts the weights to recover images that are as close as possible to the target sRGB standard image.

[0110] Finally, in the A5 stage, if the model training achieves the goal of minimizing the overall loss, the initial network architecture is defined as a convergent image generation model, which means that the model has learned how to effectively generate restored images from low-light images.

[0111] Through this pre-training step, this embodiment can provide an effective learning process for the image generation model, enabling it to recover low-light raw-RGB images into high-quality sRGB standard images.

[0112] For example, in step A1, acquiring the image pair sample set includes:

[0113] A1-1. Acquire the raw-RGB image from the image sensor before image signal processing;

[0114] A1-2. Perform image signal processing on the raw-RGB image to generate a corresponding sRGB standard image; wherein, the image signal processing includes: a predefined optimal parameter set for contrast, brightness, color matrix and gamma value;

[0115] A1-3. Using the raw-RGB image as input features and the corresponding sRGB standard image as target labels, construct image pair samples;

[0116] A1-4. Obtain several image pair samples to obtain the image pair sample set.

[0117] In this embodiment, the image sample set acquisition step includes several key stages. First, the A1-1 stage acquires a raw-RGB image from the image sensor without image signal processing. This image represents data directly acquired from the sensor under low-light conditions and typically contains raw image information. However, due to insufficient lighting, the image may have blurred details or be too dark.

[0118] Next, in stage A1-2, image signal processing is performed on the raw-RGB image to generate the corresponding sRGB standard image. The image signal processing involves adjusting parameters such as contrast, brightness, color matrix, and gamma value to ensure the image has normal brightness and color distribution that conforms to visual perception. The goal of this stage is to optimize the image using a predefined set of optimal parameters, making it closer to a standard sRGB image and improving visual quality.

[0119] In stages A1-3, the acquired raw-RGB images are used as input features, and the sRGB standard images are used as target labels to construct image pair samples. The construction of the image pair sample set is the foundation of iterative supervised learning. These image pairs will be used to train the image generation model, enabling it to learn how to generate corresponding high-quality sRGB images from raw-RGB images.

[0120] Finally, stages A1-4 acquire several image pairs to form an image pair sample set, providing ample data support for the subsequent training of the image generation model. This image pair sample set will help the network learn how to recover image details under low-light conditions. This embodiment can provide accurate input and target labels for the image generation model, thereby achieving effective conversion from low-light images to standard images.

[0121] Further, in step A2, the initial network architecture for constructing the image generation model includes:

[0122] A2-1. Select an initial model; the selected model needs to have good feature extraction capabilities and be able to recover details in low-light images. Depending on the task requirements, you can choose from classic models like UNet and ResNet to more modern Transformer-based models (such as Restormer).

[0123] A2-2. Construct independent convolutional layers and self-attention layers after the input layer of the initial model;

[0124] In this architecture, convolutional layers are responsible for extracting local features, such as edges and textures, helping to restore image details. Self-attention layers are responsible for capturing global dependencies in the image, such as the correlation between distant pixels. The independent architecture of convolutional and self-attention layers means that they process different features: convolutional layers focus on extracting local information, while self-attention layers focus on understanding global information.

[0125] A2-3. A first activation layer is connected after the convolutional layer; wherein, the first activation layer is used to learn the optimal contrast of the sRGB standard image, and the optimal contrast of the pixel set is defined as the multiplication graph;

[0126] A2-4. A second activation layer is connected after the self-attention layer; wherein, the second activation layer is used to activate the learning of the optimal global brightness of the sRGB standard image, and the optimal global brightness of the pixel set is defined as an additive graph;

[0127] A2-5. Connect the first activation layer and the second activation layer to the feature fusion layer, and then connect the fully connected layer and the output layer in sequence after the feature fusion layer to construct the initial network architecture.

[0128] In this embodiment, local and global features are extracted through an independent architecture design of convolutional layers and self-attention layers, respectively. The contrast and brightness of the image are optimized through multiplication and addition maps, ultimately achieving the generation of a detail-restored image.

[0129] For example, in step A3, the comprehensive loss function of the image generation model is defined as follows:

[0130] A3-1. Define four types of sub-loss terms for the comprehensive loss function; the four types of sub-loss terms include: contrast loss, brightness loss, gamma value loss, and color matrix loss;

[0131] A3-2. Use mean square error as the loss function for contrast loss, brightness loss, and gamma value loss;

[0132] The contrast loss function is:

[0133] ;

[0134] in, Indicates contrast loss. This represents the contrast of the detail-recovery image at pixel (i, j). This represents the contrast of an sRGB standard image at pixel (i, j);

[0135] Specifically, contrast loss measures the difference in contrast between a detail-restored image and an sRGB standard image. Contrast is typically a difference in brightness between local areas, so it's used to optimize local details by calculating the contrast difference for each pixel. Contrast loss enhances image detail, making bright and dark areas of the image more distinct, thereby improving image visibility.

[0136] The brightness loss function is:

[0137] ;

[0138] in, Indicates brightness loss. This represents the brightness of the detail-recovery image at pixel (i, j). This represents the brightness of a pixel (i, j) in an sRGB standard image.

[0139] Specifically, luminance loss measures the overall luminance difference between the detail-restored image and the sRGB standard image. Luminance is the overall brightness of an image, and the purpose of luminance loss is to make the luminance distribution of the detail-restored image closer to that of the sRGB standard image, avoiding an image that is too dark or too bright.

[0140] The gamma-value loss function is:

[0141] ;

[0142] in, Indicates gamma value loss. This represents the gamma value of the detail-recovered image at pixel (i, j). This represents the gamma value of a pixel (i, j) in an sRGB standard image.

[0143] Specifically, gamma loss measures the difference in brightness between the detail-restored image and the sRGB standard image after gamma correction, making it more consistent with the perception under normal lighting conditions.

[0144] A3-3. Use matrix angle loss as the loss function for color matrix loss;

[0145] The color matrix loss function is:

[0146] ;

[0147] in, Indicates the color matrix loss. M represents the color matrix of the detail-restored image, and M represents the color matrix of the sRGB standard image. The Frobenius norm of the color matrix of the image with detailed reconstruction is represented. The Frobenius norm represents the color matrix of an sRGB standard image. This represents the matrix dot product of the detail-restored image and the sRGB standard image;

[0148] Specifically, a color matrix is ​​a transformation matrix typically used for color space conversion. The color information of an image is adjusted by linearly transforming the color channels (such as RGB) of each pixel using the color matrix. In image processing, the color matrix is ​​commonly used to convert an image from one color space to another, for example, from RGB to HSV or Lab color spaces. The color matrix is ​​usually pre-defined in image signal processing and optimized according to specific objectives (such as adjusting the hue, saturation, and brightness of an image).

[0149] The Frobenius norm, which represents the square root of the sum of the squares of all elements of a matrix, is expressed as:

[0150] ;

[0151] in, Let (i, j) be the element in the i-th row and j-th column of matrix M (i.e., pixel (i, j)). The Frobenius norm is used to measure the size of the matrix; a larger norm indicates greater differences between the elements of the matrix. In color matrix loss, the numerical differences can be determined by comparing the Frobenius norms of the color matrix of the detail-recovered image and the color matrix of the sRGB standard image. Minimizing the difference in norms helps optimize the color reproduction of the detail-recovered image.

[0152] The matrix dot product measures the product of two vectors or matrices. In the case of matrices, the dot product calculates the sum of the element-wise products of the two matrices. Specifically, the matrix dot product measures the similarity between the color matrices of the detail-restored image and the color matrix of the sRGB standard image by calculating the sum of the element-wise products of corresponding elements. In other words, the dot product measures the similarity of two matrices at corresponding positions; a larger dot product indicates a more similar structure between the two matrices.

[0153] A3-4. Construct the comprehensive loss function by combining the contrast loss, brightness loss, gamma value loss, and color matrix loss.

[0154] The expression for the comprehensive loss function is:

[0155] ;

[0156] in, Indicates the overall loss. , , , These represent the weighting coefficients for contrast loss, brightness loss, gamma value loss, and color matrix loss, respectively.

[0157] In this embodiment, the comprehensive loss function of the image generation model includes four sub-loss terms: contrast loss, brightness loss, gamma loss, and color matrix loss. Contrast loss measures the contrast difference between the restored image and the standard image, optimizing local details and enhancing the bright and dark areas of the image. Brightness loss optimizes the brightness distribution to ensure the restored image's brightness is close to the standard image, avoiding overly dark or bright conditions. Gamma loss focuses on gamma correction of image brightness, ensuring the image conforms to visual perception under normal lighting conditions. Color matrix loss uses a matrix angle loss function to optimize the image's color performance by minimizing the difference in color matrices between the restored image and the standard image. The comprehensive loss function combines these loss terms and uses a weighted sum to adjust the model training process, ensuring the final restored image closely approximates the sRGB standard image in terms of detail, brightness, contrast, and color.

[0158] For example, in step A4, the specific steps of iterative supervised learning include:

[0159] A4-1. Extract the first round of raw-RGB images from the image sample set, propagate them forward, and generate the first round of detail recovery images in the output layer;

[0160] A4-2. Calculate the combined loss between the detail-restored image from the first round and the sRGB standard image from the first round;

[0161] A4-3. If the overall loss in the first round is not minimized, calculate the gradient of the overall loss in the first round with respect to the current model parameters.

[0162] A4-4. Based on the gradient of the current model parameters, update the model parameters in the opposite direction of the gradient with a preset learning rate to obtain the updated parameters.

[0163] A4-5. Extract the next round of raw-RGB images from the image sample set, propagate them forward using the updated parameters, and generate the next round of detail recovery images in the output layer.

[0164] A4-6. Calculate the combined loss between the detail-restored image in the next round and the sRGB standard image in the next round, until the combined loss function in the next round minimizes the combined loss.

[0165] In this embodiment, the image generation model is trained through iterative supervised learning. First, raw-RGB images are extracted from the image pair sample set and forward propagated to generate detail-restored images. Then, the combined loss between the detail-restored images and the sRGB standard images is calculated, and the gradient is calculated based on the current model parameters. If the loss is not minimized, the model parameters are updated based on the gradient, and the updated parameters are used for forward propagation of the next round of raw-RGB images. This process is repeated until the loss is minimized, thereby optimizing the image detail restoration effect.

[0166] Furthermore, the specific steps of step A4-4 also include:

[0167] A4-4-1. Randomly initialize model parameters;

[0168] A4-4-2. Calculate the gradient of contrast loss relative to the initialized model parameters in the first activation layer;

[0169] A4-4-3. Calculate the gradient of the brightness loss relative to the initialized model parameters in the second activation layer;

[0170] A4-4-4 Calculate the gradient of the color matrix and gamma values ​​relative to the initialized model parameters in the fully connected layer;

[0171] A4-4-5. Update the model parameters along the opposite direction of the gradient and the predefined learning rate;

[0172] A4-4-6. Repeat A4-4-2 to A4-4-5 until the overall loss is minimized;

[0173] A4-4-7. Define the model parameters corresponding to minimizing the comprehensive loss as the generation parameters of the image generation model;

[0174] The generation parameters of the first activation layer are represented by a matrix structure as a multiplication graph, and the generation parameters of the second activation layer are represented by a matrix structure as an addition graph.

[0175] In summary, in this embodiment, the model parameter update process is optimized using gradient descent. First, in stage A4-4-1, the model parameters are randomly initialized. Then, in stages A4-4-2 to A4-4-4, the gradients of the corresponding losses (contrast loss, brightness loss, color matrix loss, and gamma value loss) relative to the current model parameters are calculated for the first activation layer, the second activation layer, and the fully connected layer, respectively. In stage A4-4-5, the model parameters are updated along the reverse direction of the gradient using a preset learning rate. This process is repeated until the overall loss is minimized. In stage A4-4-6, the parameters corresponding to the minimized loss are finally defined as the optimal parameters of the image generation model, where the generation parameters of the first activation layer are represented as a multiplication graph, and the generation parameters of the second activation layer are represented as an addition graph. In conclusion, this invention effectively improves the detail recovery performance of low-light images by optimizing model parameters through gradient descent, ensuring that the image's contrast, brightness, and color optimization achieve the predefined visual effect.

[0176] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means.

[0177] The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g.,...), etc. DVD ( ), or semiconductor media. Semiconductor media can be solid-state drives (SSDs).

[0178] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division of a waterway underwater topography change analysis system and method. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0179] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An image detail recovery method based on an attention mechanism in a vehicle-mounted night vision system, characterized in that, include: S1. Obtain the image tensor of the low-light night view image; S2. Input the image tensor into a pre-trained image generation model and perform feature mapping on it; S3. Generate a detail restoration image in the output layer according to the mapping parameters of the feature mapping; wherein, the mapping parameters of the feature mapping are characterized as the color matrix and gamma value of the sRGB standard image learned by the image generation model in pre-training; The input layer of the image generation model uses a dual-channel architecture to independently connect convolutional layers and self-attention layers; The image tensor is input into a pre-trained image generation model, and feature mapping is performed on it, including: S2-1. Receive the image tensor as input to the convolutional layer and the self-attention layer, and extract its local features and global dependency features respectively; S2-2. Perform pixel-wise multiplication on local features using the pre-trained multiplication graph to generate contrast-optimized local features; where the multiplication graph is represented as the set of contrasts of the sRGB standard image learned by the image generation model during pre-training. S2-3. The global dependent features are added pixel by pixel using the pre-trained additive graph to generate global features optimized for brightness; where the additive graph represents the set of brightness values ​​of the sRGB standard image learned by the image generation model during pre-training. S2-4. The local features optimized for contrast and the global features optimized for brightness are fused to generate optimized global features; S2-5. The optimized global features are input into the fully connected layer, and a detail restoration image is generated in the output layer through forward propagation.

2. The image detail restoration method based on attention mechanism in the vehicle-mounted night vision system according to claim 1, characterized in that, The feature fusion involves concatenating local features optimized for contrast and global features optimized for brightness into vectors.

3. The image detail restoration method based on attention mechanism in the vehicle-mounted night vision system according to claim 1, characterized in that, The pre-training steps of the image generation model include: A1. Obtain an image pair sample set; wherein, the image pair sample consists of raw-RGB images and sRGB standard images; A2. Construct the initial network architecture for the image generation model; A3. Define the overall loss function for the image generation model; A4. Use the images to perform iterative supervised learning on the sample set within the initial network architecture until the defined comprehensive loss function minimizes the comprehensive loss; A5. If the overall loss is minimized, then the initial network architecture is defined as the image generation model.

4. The image detail restoration method based on an attention mechanism in a vehicle-mounted night vision system according to claim 3, characterized in that, Obtain the image pair sample set, including: A1-1. Acquire the raw-RGB image from the image sensor before image signal processing; A1-2. Perform image signal processing on the raw-RGB image to generate a corresponding sRGB standard image; wherein, the image signal processing includes: a predefined optimal parameter set for contrast, brightness, color matrix and gamma value; A1-3. Using the raw-RGB image as input features and the corresponding sRGB standard image as target labels, construct image pair samples; A1-4. Obtain several image pair samples to obtain the image pair sample set.

5. The image detail restoration method based on an attention mechanism in a vehicle-mounted night vision system according to claim 4, characterized in that, The initial network architecture for constructing the image generation model includes: A2-1. Select the initial model; A2-2. Construct independent convolutional layers and self-attention layers after the input layer of the initial model; A2-3. A first activation layer is connected after the convolutional layer; wherein, the first activation layer is used to learn the optimal contrast of the sRGB standard image, and the optimal contrast of the pixel set is defined as the multiplication graph; A2-4. A second activation layer is connected after the self-attention layer; wherein, the second activation layer is used to activate the learning of the optimal global brightness of the sRGB standard image, and the optimal global brightness of the pixel set is defined as an additive graph; A2-5. Connect the first activation layer and the second activation layer to the feature fusion layer, and then connect the fully connected layer and the output layer in sequence after the feature fusion layer to construct the initial network architecture.

6. The image detail restoration method based on an attention mechanism in a vehicle-mounted night vision system according to claim 5, characterized in that, Define the overall loss function for the image generation model, including: A3-1. Define four types of sub-loss terms for the comprehensive loss function; the four types of sub-loss terms include: contrast loss, brightness loss, gamma value loss, and color matrix loss; A3-2. Use mean square error as the loss function for contrast loss, brightness loss, and gamma value loss; The contrast loss function is: ； in, Indicates contrast loss. This represents the contrast of the detail-recovery image at pixel (i, j). This represents the contrast of an sRGB standard image at pixel (i, j); The brightness loss function is: ； in, Indicates brightness loss. This represents the brightness of the detail-recovery image at pixel (i, j). This represents the brightness of a pixel (i, j) in an sRGB standard image. The gamma-value loss function is: ； in, Indicates gamma value loss. This represents the gamma value of the detail-recovered image at pixel (i, j). This represents the gamma value of a pixel (i, j) in an sRGB standard image. A3-3. Use matrix angle loss as the loss function for color matrix loss; The color matrix loss function is: ； in, Indicates the color matrix loss. M represents the color matrix of the detail-restored image, and M represents the color matrix of the sRGB standard image. The Frobenius norm of the color matrix of the image with detailed reconstruction is represented. The Frobenius norm represents the color matrix of an sRGB standard image. This represents the matrix dot product of the detail-restored image and the sRGB standard image; A3-4. Construct the comprehensive loss function by combining the contrast loss, brightness loss, gamma value loss, and color matrix loss. The expression for the comprehensive loss function is: ； in, Indicates the overall loss. , , , These represent the weighting coefficients for contrast loss, brightness loss, gamma value loss, and color matrix loss, respectively.

7. The image detail restoration method based on an attention mechanism in a vehicle-mounted night vision system according to claim 6, characterized in that, The images are used to perform iterative supervised learning on the sample set within the initial network architecture until the defined comprehensive loss function is minimized, including: A4-1. Extract the first round of raw-RGB images from the image sample set, propagate them forward, and generate the first round of detail recovery images in the output layer; A4-2. Calculate the combined loss between the detail-restored image from the first round and the sRGB standard image from the first round; A4-3. If the overall loss in the first round is not minimized, calculate the gradient of the overall loss in the first round with respect to the current model parameters. A4-4. Based on the gradient of the current model parameters, update the model parameters in the opposite direction of the gradient with a preset learning rate to obtain the updated parameters. A4-5. Extract the next round of raw-RGB images from the image sample set, propagate them forward using the updated parameters, and generate the next round of detail recovery images in the output layer. A4-6. Calculate the combined loss between the detail-restored image in the next round and the sRGB standard image in the next round, until the combined loss function in the next round minimizes the combined loss.

8. The image detail restoration method based on attention mechanism in the vehicle-mounted night vision system according to claim 7, characterized in that, Based on the gradient of the current model parameters, update the model parameters in the reverse direction of the gradient with a preset learning rate to obtain the updated parameters, including: A4-4-1. Randomly initialize model parameters; A4-4-2. Calculate the gradient of contrast loss relative to the initialized model parameters in the first activation layer; A4-4-3. Calculate the gradient of the brightness loss relative to the initialized model parameters in the second activation layer; A4-4-4 Calculate the gradient of the color matrix and gamma values ​​relative to the initialized model parameters in the fully connected layer; A4-4-5. Update the model parameters along the opposite direction of the gradient and the predefined learning rate; A4-4-6. Repeat A4-4-2 to A4-4-5 until the overall loss is minimized; A4-4-7. Define the model parameters corresponding to minimizing the comprehensive loss as the generation parameters of the image generation model; The generation parameters of the first activation layer are represented by a matrix structure as a multiplication graph, and the generation parameters of the second activation layer are represented by a matrix structure as an addition graph.

Images