Image analysis method fusing physical prior

Abstract

The present application relates to the field of image data processing, and particularly relates to an image analysis method fusing physical priori, which comprises obtaining image pairs composed of multiple clear images and final underwater degraded images as training data; inputting a current image and a previous image in a generated underwater image sequence into an image analysis model to calculate a predicted semantic result of the current image; calculating a total loss function based on cross-entropy loss, Dice loss and time consistency loss; obtaining a trained image analysis model; inputting a current image and a previous image of a new underwater image sequence into the trained image analysis model to obtain a predicted semantic result of the current image. The existing underwater image analysis method has the problem of poor generalization performance in unknown environment, and the image analysis method fusing physical priori provided by the present application has good generalization performance in unknown environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image data processing, in particular to an image analysis method fusing physical priori. BACKGROUND

[0002] Image analysis refers to dividing an image into multiple regions through a deep learning algorithm and classifying pixels in each region, aiming to assign semantic labels to each pixel in the image, so as to realize accurate understanding of the image content; analyzing underwater images faces difficulties that natural images (i.e. clear images) do not have, such as low brightness, low contrast, color distortion and detail blur of underwater images caused by light attenuation, scattering and color deviation, etc., so that the analysis accuracy is poor only when a single underwater image is analyzed, therefore people usually collect an underwater image sequence containing multiple continuous underwater images, and combine the consistency of objects and the continuity of object motion between the multiple continuous underwater images to enhance the stability of image analysis.

[0003] The steps of the existing underwater image analysis method are usually as follows: extracting optical flow or feature mapping to depict the continuity of the target in time; using a deep convolutional neural network or a time sequence model to simultaneously learn the spatial structure and the time dynamics; in the training process, multi-frame joint supervision and time sequence consistency constraints are used to reduce segmentation jitter and boundary misjudgment. In the inference stage, the image analysis model can use the context information of multiple continuous underwater images to improve the target recognition effect in weak texture areas. However, the existing method is limited by the scarcity of training samples, and the performance is unstable in unknown underwater environments.

[0004] Therefore, the existing underwater image analysis method has the problem of poor generalization performance in unknown environments. SUMMARY

[0005] The technical problem to be solved by the present application is to overcome the shortcomings of the prior art and provide an image analysis method fusing physical priori with good generalization performance in unknown environments.

[0006] To solve the above technical problems, the present application provides an image analysis method fusing physical priori, which comprises the following steps:

[0007] S1. The image sequence in the natural light environment comprises multiple clear images, the underwater imaging model is used to convert the multiple clear images into multiple final underwater degraded images, the image pairs composed of the multiple clear images and the final underwater degraded images thereof are obtained as training data, and the multiple final underwater degraded images constitute a generated underwater image sequence;

[0008] S2. inputting the current image and the previous image in the generated underwater image sequence into the image analysis model to calculate the predicted semantic result of the current image;

[0009] S3. Calculate the total loss function based on cross-entropy loss, Dice loss, and temporal consistency loss;

[0010] S4. Repeat S2~S3 until the preset stopping iteration condition is met, then stop training and obtain the trained image analysis model;

[0011] S5. Input the current image and the previous image of the new underwater image sequence into the trained image analysis model to obtain the predicted semantic result of the current image.

[0012] As a further improvement of the present invention: the underwater imaging model in S1 is the Jaffe-McGlamery underwater optical imaging model.

[0013] As a further improvement of the present invention: step S1 includes:

[0014] S101. The underwater imaging model randomly generates underwater environment simulation parameters for each clear image;

[0015] Underwater environment simulation parameters include generated background light, generated attenuation coefficient, and generated depth value;

[0016] S102. Calculate the generated transmittance based on the generated attenuation coefficient and the generated depth value;

[0017] S103. Based on each clear image, the generated background light calculated in S101 and the generated transmittance calculated in S102, the underwater degraded image of each clear image is synthesized.

[0018] S104. Add noise to the underwater degraded images of each clear image to obtain the final underwater degraded image.

[0019] Preferably, S101 includes:

[0020] S1011. Calculate and generate background light;

[0021] S1012. Calculate the attenuation coefficient;

[0022] S1013. Calculate the generated depth value.

[0023] Preferably, step S2 includes:

[0024] S201. Input the current image and the previous image in the generated underwater image sequence;

[0025] S202. Use the ConvNeXt network to extract high-level features of the current image and the previous image;

[0026] S203. Calculate high-level features of the aligned previous image based on optical flow estimates;

[0027] S204. After concatenating the high-level features of the previous image with the features of the current image in the channel dimension, the fused features are then fused through a convolutional attention module (CBAM) to calculate the fused features.

[0028] S205. The fused features are fed into the feature fusion decoder of the image analysis model to obtain the predicted semantic results of the current image.

[0029] Preferably, step S202 includes:

[0030] The shared-weight encoder in the S2021.ConvNeXt network consists of multiple... Composed of, calculate each The output characteristics;

[0031] S2022. Based on each The output features are used to calculate the high-level features of the current image and the high-level features of the previous image.

[0032] Preferably, step S203 includes:

[0033] S2031. Calculate the estimated optical flow;

[0034] S2032. Use bilinear sampling to align the high-level features of the previous image to the current image coordinate system according to optical flow, and obtain the aligned high-level features of the previous image.

[0035] As a further improvement of the present invention: step S3 includes:

[0036] S301. Calculate the cross-entropy loss;

[0037] S302. Calculate Dice loss;

[0038] S303. Calculate the timing consistency loss;

[0039] S304. Calculate the total loss function based on cross-entropy loss, Dice loss, and temporal consistency loss.

[0040] The beneficial effects of the present invention are as follows: The image analysis method that integrates physical priors provided by the present invention has good generalization performance in unknown environments.

[0041] First, underwater environments vary greatly due to differences in water quality, depth, and lighting conditions, and current training methods lack diverse training samples. S1 diversifies the training samples by generating a large number of image pairs, enabling the final trained image analysis model to perform stably in unseen underwater environments (with different water quality, depth, and lighting conditions), reducing reliance on large amounts of real-world labeled data. Meanwhile, existing methods typically employ simple color correction or conventional data augmentation, failing to deeply utilize the physical laws of underwater imaging, leading to unstable model performance in severely degraded images. S1, however, incorporates the physical laws revealed by underwater optical imaging models as known knowledge into the image analysis model, making the model more stable by fusing physical priors.

[0042] Secondly, S2 will generate the current image and the previous image in the underwater image sequence and input them into the image analysis model to calculate the predicted semantic result of the current image. The semantic information of the previous image will be used to guide the fusion through optical flow, providing a strong prior for the segmentation of the current image and significantly improving the accuracy in challenging scenarios such as occlusion and blur.

[0043] In summary, the image analysis method fused with physical priors provided by this invention exhibits good generalization performance in unknown environments, and can provide reliable visual support for tasks such as underwater exploration, robot navigation and resource monitoring, meeting the needs of marine engineering, fisheries and other fields. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the principle of the present invention;

[0045] Figure 2 This is an architecture diagram of the image analysis model in this invention. Detailed Implementation

[0046] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0047] Definitions of relevant terms used in this invention:

[0048] (1) Natural images: refers to images of the real world taken by a camera under natural lighting conditions, with the default shooting medium being air;

[0049] (2) Underwater images: refers to all images taken in underwater environments;

[0050] (3) Light attenuation: When light propagates underwater, it gradually weakens due to its interaction with the water and its substances, which affects the brightness of underwater images, etc.

[0051] (5) Scattering: Light scattering caused by suspended particles in water can blur the details of underwater images;

[0052] (6) Color distortion: Water selectively absorbs light waves, causing underwater images to appear mostly blue-green, resulting in color distortion;

[0053] (7) COCO: A large image dataset commonly used in the field of computer vision, containing a large number of labeled images, which can be used for training or testing of image-related algorithms; it is often used to provide samples of clear images to support research such as underwater image degradation simulation.

[0054] (8) PWC-Net: A small but efficient CNN model for optical flow estimation that combines classic principles such as pyramid processing, deformation and cost volume;

[0055] (9) DeepLabV3+: DeepLabv3+ is an image analysis model based on an Encoder-Decoder architecture, combining dilated convolution and multi-scale feature extraction techniques to improve segmentation accuracy without significantly increasing computational costs. It includes improved shared-weight encoder and decoder modules;

[0056] (10) ImageNet: is a dataset containing tens of millions of images (such as cats, dogs, cars, landscapes, etc.). It is one of the most commonly used datasets for image classification, detection, and localization in the field of deep learning.

[0057] like Figure 1 , Figure 2 As shown, the image analysis method fused with physical priors provided by this invention includes:

[0058] S1. Diverse Data Augmentation Based on Physical Models: Acquire image sequences under natural lighting conditions, which include multiple clear images. Use an underwater imaging model to convert these clear images into multiple final underwater degraded images. Obtain image pairs consisting of multiple clear images and their final underwater degraded images as training data. Multiple final underwater degraded images constitute the generated underwater image sequence. This approach addresses the challenges of data scarcity and domain adaptation through diverse training data.

[0059] The underwater imaging model is the Jaffe-McGlamery underwater optical imaging model. Data augmentation and optical flow-guided inter-frame feature fusion are performed using the Jaffe-McGlamery underwater optical imaging model to improve the accuracy and robustness of underwater image analysis.

[0060] Each underwater degradation image includes 1 pixel Using pixel indexes, the Jaffe-McGlamery underwater optical imaging model is used to calculate underwater degraded images. The formula is:

[0061] ,

[0062] In the formula, For pixels spatial coordinates, , Let q be the horizontal coordinate of pixel q; Let q be the vertical coordinate of pixel q; For clear images, the data source is the COCO source annotation dataset; This is the element-wise product, i.e., the Hadamard product; Transmittance; As background light; It is Gaussian noise;

[0063] The background light is an RGB three-channel vector light. Represented as:

[0064] ,

[0065] In the formula, B r For the red channel component; For green channel components; For the blue channel component;

[0066] transmittance The calculation formula is:

[0067] ,

[0068] In the formula, It is an exponential function; This is the attenuation coefficient, used to describe the attenuation rate of red, green, and blue light in water; This is a depth value used to represent the distance from each pixel in the scene to the camera;

[0069] The attenuation coefficient is a vector defined separately for each of the RGB three channels. Represented as:

[0070] ,

[0071] In the formula, This represents the attenuation coefficient for the red channel; This represents the attenuation coefficient for the green channel. This represents the attenuation coefficient of the blue channel;

[0072] Under normal circumstances, This is because in an underwater environment, red light has the longest wavelength and the fastest attenuation rate, while blue light has the shortest wavelength and the slowest attenuation rate.

[0073] S101. The underwater imaging model randomly generates underwater environment simulation parameters for each clear image;

[0074] Randomly generated underwater environment simulation parameters refer to the underwater imaging model randomly generating a set of underwater environment simulation parameters for each clear image to simulate the underwater environment. Random generation ensures that each set of underwater environment simulation parameters represents different underwater conditions.

[0075] Underwater environment simulation parameters include generated background light, generated attenuation coefficient, and generated depth value;

[0076] S1011. Calculate and generate background light;

[0077] The generated background light is a random variable that follows a uniform distribution within an interval. The calculation formula is:

[0078] ,

[0079] In the formula, b, Represents red, Represents green, Represents blue; The distribution is a uniform distribution biased towards low values; For use in defining The set of parameters for the color channel background light intensity range, usually... Set it low to simulate red light decay; for The lower limit of random sampling of color channels; for The upper limit of random sampling for color channels;

[0080] S1012. Calculate the attenuation coefficient;

[0081] Generate attenuation coefficient Similarly satisfied Constraints, attenuation coefficient The calculation formula is:

[0082] ,

[0083] In the formula, It is a uniformly distributed random sampling function; for The minimum value that the attenuation coefficient of the color channel can take; for The maximum value that the attenuation coefficient of the color channel can take;

[0084] S1013. The depth value is generated by combining a linear gradient model with random perturbation;

[0085] Generate depth values The calculation formula is:

[0086] ;

[0087] In the formula, This is a random scaling factor; Normalized vertical coordinates Incrementing from the top (0) of the image to the bottom (H); This represents the image height, i.e., the bottom of the image. This is a random noise term used to simulate local disturbances and irregularities in a real underwater environment;

[0088] S102. Calculate the generated transmittance based on the generated attenuation coefficient and the generated depth value;

[0089] Generate transmittance The calculation formula is:

[0090] ;

[0091] S103. Based on each clear image, the generated background light calculated in S101 and the generated transmittance calculated in S102, the underwater degraded image of each clear image is synthesized.

[0092] The underwater environment simulation parameters corresponding to each clear image, calculated in S101, and the transmittance calculated in S102 are substituted into the Jaffe-McGlamery underwater optical imaging model to calculate the underwater degraded images. From the formula, we get: ;

[0093] S104. Add noise to the underwater degraded images of each clear image to obtain the final underwater degraded image;

[0094] Final underwater degradation image The calculation formula is:

[0095] ;

[0096] In the formula, Spatial coordinates Additive noise at the location, , The noise level is randomly generated;

[0097] The underwater imaging model simulates the attenuation, scattering, and noise effects of light in water. The generated transmittance is calculated using the generated attenuation coefficient and the generated depth value, utilizing the generated background light and spatial coordinates. Additive noise is added to further simulate the optical characteristics of the underwater environment, and the clear image is synthesized into the final underwater degraded image, realizing diversified data augmentation based on the physical model.

[0098] S2. Input the current image and the adjacent previous image (hereinafter referred to as "previous image") in the generated underwater image sequence into the image analysis model to calculate the predicted semantic result of the current image;

[0099] The image analysis model includes a weighted encoder for extracting multi-scale features, an optical flow estimation branch, and a feature fusion decoder;

[0100] The encoder with shared weights is a ConvNeXt network pre-trained on ImageNet; the feature fusion decoder is based on the DeepLabV3+ architecture, which combines ASPP (Atrous Spatial Pyramid Pooling) and skip connections to fuse low-level features. After the ASPP output, the temporal information is enhanced by convolutional layers containing CBAM (Convolutional Block Attention Module).

[0101] S201. Input the current image and the previous image in the generated underwater image sequence;

[0102] S202. Use the ConvNeXt network to extract high-level features of the current image and the previous image;

[0103] The shared-weight encoder in the S2021.ConvNeXt network consists of multiple... Composed of, calculate each The output characteristics;

[0104] Output characteristics The calculation formula is:

[0105] ,

[0106] In the formula, This is the l-th layer of the network; The input features for the Block; For pointwise convolution weights; For the ConvNeXt network activation function; These are depthwise convolution weights; For layer normalization function;

[0107] S2022. Based on each The output features are used to calculate the high-level features of the current image and the high-level features of the previous image;

[0108] Current image high-level features The calculation formula is:

[0109] ,

[0110] In the formula, The first in the shared weight encoder Each module unit; For the index of the module unit; This refers to the (L-1)th module unit in the encoder with shared weights. This refers to the Lth module unit in the encoder with shared weights. The current image; For time indexing;

[0111] High-level features of the previous image The calculation formula is:

[0112] ,

[0113] In the formula, This is the previous image.

[0114] S203. Calculate high-level features of the aligned previous image based on optical flow estimates;

[0115] S2031. Calculate optical flow estimates based on PWC-Net;

[0116] Optical flow estimate The calculation formula is:

[0117] ,

[0118] In the formula, This is an upsampling operation; This is the second weight matrix; For activation functions; This is the first weight matrix; For the sake of volume calculation, That is, Cost Volume; For distortion operations; For optical flow estimation at the next higher level; It is a convolutional neural network; This is the first bias vector; This is the second bias vector;

[0119] S2032. Using bilinear sampling, high-level features from the previous image are aligned to the current image coordinate system according to optical flow, resulting in aligned high-level features from the previous image. This ensures that the features match the current image in space, which is suitable for feature misalignment problems caused by motion in underwater videos.

[0120] Aligned with high-level features of the previous image The calculation formula is:

[0121] ,

[0122] In the formula, High-level features of the previous image have been aligned; The optical flow field represents the pixel. Horizontal and vertical displacement from the previous image to the current image; The new coordinates are non-integer coordinates, achieved through differentiable bilinear interpolation;

[0123] Optical flow field The calculation formula is:

[0124] ,

[0125] In the formula, This represents the horizontal component of the optical flow vector. This represents the vertical component of the optical flow vector;

[0126] New coordinates The calculation formula is:

[0127] ;

[0128] S204. After concatenating the high-level features of the previous image with the features of the current image in the channel dimension, the fused features are then fused through a convolutional attention module (CBAM) to calculate the fused features.

[0129] Fusion features The calculation formula is:

[0130] ,

[0131] In the formula, CBAM stands for Convolutional Block Attention Module. This is the fourth weight matrix; This is the third weight matrix; For feature splicing operations; This is the third bias vector; This is the fourth bias vector;

[0132] S205. Feed the fused features into the feature fusion decoder of the image analysis model to obtain the predicted semantic results of the current image;

[0133] Predicted semantic results for the current image The calculation formula is:

[0134] ,

[0135] In the formula, This is an upsampling operation; This is the sixth weight matrix; This is the fifth weight matrix; For time Low-level features; This is the fifth bias vector; This is the sixth bias vector;

[0136] time Low-level features The calculation formula is:

[0137] ,

[0138] In the formula, These are the weights of the lower-level convolutional kernels; This is a convolution operation; The current image; For bias terms;

[0139] Multi-level features of the current image and the previous image are extracted by a weighted encoder. Pixel-level motion is calculated using an optical flow estimation branch and the features of the previous image are aligned. Finally, temporally consistent image analysis results are generated through feature fusion and decoding.

[0140] S3. Calculate the total loss function based on cross-entropy loss, Dice loss, and temporal consistency loss;

[0141] S301. Calculate the cross-entropy loss;

[0142] Cross-entropy loss is the standard classification loss. The calculation formula is:

[0143] ,

[0144] In the formula, The true semantic label of the current image; This represents the total number of pixels. Indexed by category; Total number of categories;

[0145] S302. Calculate Dice loss;

[0146] Dice loss is used to address class imbalance, and is particularly suitable for scenarios where the background constitutes a large proportion, such as underwater environments. The calculation formula is:

[0147] ,

[0148] In the formula, For smoothing terms;

[0149] S303. Calculate the timing consistency loss;

[0150] Temporal consistency loss is used to ensure that the prediction results change smoothly over time by utilizing optical flow distortion, thus ensuring that the segmentation results are smooth and stable in the time dimension and reducing flickering. Temporal consistency loss is calculated using the L1 norm or MSE loss. The calculation formula is:

[0151] ,

[0152] In the formula, For distortion operation, This means using optical flow to warp the prediction from the previous image to the current image, using the same optical flow. Segmentation probability map of the previous image Obtained by performing a distortion operation; This is the segmentation probability map of the previous image; To sum after taking the absolute value; Also based on optical flow Will Each pixel is mapped to the coordinates of the current image;

[0153] S304. Calculate the total loss function (i.e., the objective function for optimizing the image analysis model) based on cross-entropy loss, Dice loss, and temporal consistency loss.

[0154] Total loss function The calculation formula is:

[0155] ,

[0156] In the formula, These are the weighting coefficients for the Dice loss; This is a convolution operation; The weighting coefficient for the time consistency loss;

[0157] S3 reduces flickering by introducing a total loss function that combines cross-entropy loss, Dice loss, and temporal consistency loss, and by using optical flow to distort the prediction of the previous image to constrain the output of the current image. Together with S2, it optimizes the class accuracy and inter-frame stability of the image analysis model in underwater image segmentation.

[0158] S4. Repeat S2~S3 until the preset stopping iteration condition is met, then stop training and obtain the trained image analysis model;

[0159] The preset stopping condition is reaching the preset maximum number of iterations, which is 5000.

[0160] S5. Input the current image and the previous image of the new underwater image sequence into the trained image analysis model to obtain the predicted semantic result of the current image;

[0161] Since the first frame does not contain the previous image, the first frame is directly segmented or copied using a static model; subsequent frames are segmented by combining the features of the previous image.

Claims

1. An image analysis method incorporating physical priors, characterized in that, include: S1. The image sequence under natural lighting conditions includes multiple clear images. The underwater imaging model is used to convert the multiple clear images into multiple final underwater degraded images. The image pairs consisting of multiple clear images and their final underwater degraded images are used as training data. The multiple final underwater degraded images constitute the generated underwater image sequence. S2. Input the current image and the previous image from the generated underwater image sequence into the image analysis model to calculate the predicted semantic result of the current image; S201. Input the current image and the previous image in the generated underwater image sequence; S202. Use the ConvNeXt network to extract high-level features of the current image and the previous image; S203. Calculate high-level features of the aligned previous image based on optical flow estimates; S204. After concatenating the high-level features of the previous image with the high-level features of the current image along the channel dimension, the fused features are calculated by using a convolutional attention module CBAM. S205. Feed the fused features into the feature fusion decoder of the image analysis model to obtain the predicted semantic results of the current image; S3. Calculate the total loss function based on cross-entropy loss, Dice loss, and temporal consistency loss; S4. Repeat S2~S3 until the preset stopping iteration condition is met, then stop training and obtain the trained image analysis model; S5. Input the current image and the previous image of the new underwater image sequence into the trained image analysis model to obtain the predicted semantic result of the current image.

2. The image analysis method integrating physical priors according to claim 1, characterized in that, The underwater imaging model in S1 is the Jaffe-McGlamery underwater optical imaging model.

3. The image analysis method integrating physical priors according to claim 1, characterized in that, The steps in S1 include: S101. The underwater imaging model randomly generates underwater environment simulation parameters for each clear image; Underwater environment simulation parameters include generated background light, generated attenuation coefficient, and generated depth value; S102. Calculate the generated transmittance based on the generated attenuation coefficient and the generated depth value; S103. Based on each clear image, the generated background light calculated in S101 and the generated transmittance calculated in S102, the underwater degraded image of each clear image is synthesized. S104. Add noise to the underwater degraded images of each clear image to obtain the final underwater degraded image.

4. The image analysis method incorporating physical priors according to claim 3, characterized in that, S101 includes: S1011. Calculate and generate background light; S1012. Calculate the attenuation coefficient; S1013. Calculate the generated depth value.

5. The image analysis method incorporating physical priors according to claim 1, characterized in that, The steps in S202 include: The shared-weight encoder in the S2021.ConvNeXt network consists of multiple... Composed of, calculate each The output characteristics; S2022. Based on each The output features are used to calculate the high-level features of the current image and the high-level features of the previous image.

6. The image analysis method integrating physical priors according to claim 1, characterized in that, The steps in S203 include: S2031. Calculate the estimated optical flow; S2032. Use bilinear sampling to align the high-level features of the previous image to the current image coordinate system according to optical flow, and obtain the aligned high-level features of the previous image.

7. The image analysis method integrating physical priors according to claim 1, characterized in that, The steps in S3 include: S301. Calculate the cross-entropy loss; S302. Calculate Dice loss; S303. Calculate the timing consistency loss; S304. Calculate the total loss function based on cross-entropy loss, Dice loss, and temporal consistency loss.

Images