Underwater multi-degraded image enhancement and target identification joint optimization method based on genetic algorithm

Abstract

The invention discloses an underwater multi-degraded image enhancement and target recognition joint optimization method based on a genetic algorithm, and relates to the field of underwater image enhancement and underwater target recognition, and the method comprises the steps: building an underwater robot target recognition system, obtaining underwater event camera data, and processing the underwater event camera data to obtain a sample data set; initializing by adopting a genetic algorithm to obtain a primary population, namely an initial enhancement parameter; performing initial enhancement on the environment information by adopting the initial enhancement parameter to obtain an initial enhanced image data set; taking YOLOv5 as a basic framework, constructing an underwater target recognition algorithm model, and obtaining underwater target detection average precision mAP; the underwater target detection average precision mAP is used as a genetic algorithm fitness value, a genetic algorithm is adopted to optimize enhancement parameters of an underwater multi-degraded image enhancement module, then target enhancement parameters are obtained to enhance environment information to obtain a target enhanced image, and the purposes of image enhancement and effective recognition are achieved at the same time.

Description

Technical Field

[0001] The invention relates to the field of underwater image enhancement and underwater target recognition, in particular to a joint optimization method of underwater multi-degraded image enhancement and target recognition based on a genetic algorithm. Background Art

[0002] As a widely used unmanned submersible, underwater robots are often used for ocean exploration, intelligence reconnaissance, underwater search and rescue, etc. Thanks to these characteristics, underwater robots can replace humans in underwater work at depths beyond human limits and in highly dangerous environments, and are of great significance in scientific research and military fields. Underwater robots are usually equipped with cameras to identify underwater targets.

[0003] On the one hand, due to the complexity and changeability of the underwater environment, images taken underwater usually have multiple degradation problems, which brings great challenges to underwater robot operations and target recognition. The absorption and scattering of light by the water medium leads to reduced contrast and visibility of underwater images. At the same time, the underwater environment requires artificial fill light, which will result in uneven lighting, resulting in underexposure or overexposure of some images. Therefore, it is necessary to study an image enhancement method for underwater environments to reduce the adverse effects of underwater image quality degradation on underwater robot target recognition. Generally, there are two ways to improve underwater image quality: the first is the hardware method, which introduces new equipment to make up for the shortcomings of ordinary cameras and improve image quality; the other is the software method, which improves image quality through underwater image processing methods.

[0004] On the other hand, underwater image processing and underwater target recognition are two different tasks. Underwater target recognition improves the visibility of images in terms of human perception, while the goal of object detection algorithms is to improve the accuracy of target recognition. Directly using enhanced underwater images for target recognition may not produce good target recognition results. Therefore, it is very necessary to adjust the underwater image processing algorithm in combination with underwater target recognition. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a joint optimization algorithm for underwater multi-degraded image enhancement and target recognition based on genetic algorithm, which uses genetic algorithm to optimize the parameters of underwater multi-degraded image enhancement, takes the detection mean accuracy (mAP) of target recognition as the fitness function, and obtains the target underwater multi-degraded image enhancement target solution for underwater target recognition, thereby improving the accuracy of underwater target recognition. The present invention provides a joint optimization method for underwater multi-degraded image enhancement and target recognition based on genetic algorithm, which includes the following steps:

[0006] S1. Building an underwater robot target recognition system, wherein the underwater robot target recognition system includes an underwater robot and an underwater event camera, wherein the underwater event camera is mounted on the underwater robot;

[0007] S2, acquiring underwater event camera data, analyzing the underwater event camera data to obtain environmental information of the underwater environment, wherein the environmental information, i.e., the underwater multi-degraded image, includes an event stream E0 and an RGB image, i.e., a color frame I0 captured by the underwater event camera, wherein the event stream E0 follows the RGB image respectively, and the event stream E0 enhances the multi-degraded RGB image in the process of underwater multi-degraded image enhancement;

[0008] S3, using the target recognition data annotation tool to annotate the RGB image, and divide the environmental information according to the annotation results to obtain a sample data set;

[0009] S4, using genetic algorithm to initialize and obtain the initial enhancement parameters of the first generation population, i.e., the underwater multi-degraded image enhancement algorithm;

[0010] S5, using initial enhancement parameters to perform initial enhancement on the environmental information to obtain an initial enhanced image data set;

[0011] S6. Using YOLOv5 as the basic framework, an underwater target recognition algorithm model is constructed. The initial enhanced image data set is input into the underwater target recognition algorithm model to obtain the average precision mAP of underwater target detection.

[0012] S7. Take the target detection mean average precision mAP as the fitness value of the genetic algorithm, optimize the initial enhancement parameters through selection-crossover-mutation operations to obtain the next generation population, and then enhance the original image. The maximum evolutionary number of the genetic algorithm is preset, and S5 and S6 are repeated until the genetic algorithm reaches the maximum evolutionary number to obtain the target enhancement parameters. Combined with the target enhancement parameters, the underwater multi-degraded image enhancement algorithm is used to enhance the environmental information to obtain the target enhanced image, and the YOLOv5 model is used for target recognition to obtain the target recognition result.

[0013] Preferably, the specific contents of performing initial enhancement on the environmental information using the initial enhancement parameters to obtain the initial enhanced image data set include:

[0014] S301, environmental information enters a preprocessing stage to obtain preprocessing information, the environmental information preprocessing stage includes an event stream reconstruction unit and an RGB image color correction unit, and the preprocessing information includes an event reconstruction image and an RGB color correction image;

[0015] S302, the pre-processed information enters the defogging stage to obtain enhanced environmental information data, wherein the enhanced environmental information data includes a fog-free event image and a fog-free RGB image, and the defogging stage includes an event reconstruction image defogging unit and an RGB image defogging unit;

[0016] S303: The enhanced environment information data enters the image fusion unit to obtain a fused image, ie, an initial enhanced image.

[0017] Preferably, the specific content of the event stream reconstruction unit is:

[0018] The event stream is a data sequence E i =(x i ,y i , t i , p i ), which contains information about the pixel position (x i ,y i ), sampling time t i and change symbol p i information;

[0019] The event stream E0 is reconstructed into an intensity image I using the E2VID method E ;

[0020] The specific contents of the E2VID method include using a convolutional recurrent network to learn intensity frame reconstruction from asynchronous event data;

[0021] Building an improved recursive network based on the UNet architecture;

[0022] The event simulator is used to generate 1000 simulated event sequences from the event stream to train and test the network, of which 950 are training simulated event sequences and 50 are verification simulated event sequences.

[0023] Divide the simulation event sequence into a series of non-overlapping space-time windows and convert them into event tensors;

[0024] The simulated event sequence in each spatiotemporal window is sequentially input into the trained network, and the intensity frames are reconstructed to obtain the reconstructed event image.

[0025] Preferably, the specific content of the RGB image color correction unit is:

[0026] According to the average intensity value of the three color channels R, G, and B in the underwater RGB image;

[0027] The expression for calculating the average intensity value is as follows:

[0028]

[0029] Where H and W represent the height and width of the input image respectively.

[0030] According to the average intensity value from high to low, the three color channels of R, G, and B are divided into: high-quality color channels Medium quality color channel Low quality color channels

[0031] High-quality color channels For reference, the compensation factor J for medium quality color channel compensation and the compensation factor K for low quality color channel compensation are calculated respectively;

[0032] The formula for calculating the compensation factor J and the compensation factor K is as follows:

[0033]

[0034] Use compensation factors J and K to compensate for low- and medium-quality color channels;

[0035] The compensation expression is as follows:

[0036] I m (x,y)=I m (x,y)+(J×I hq (x,y));

[0037] I lq (x,y)=I lq (x,y)+(K×I hq (x,y));

[0038] The white balance algorithm based on the gray world hypothesis is used for further color correction to obtain the color-corrected RGB image I R That is, RGB color corrected image.

[0039] Preferably, the specific content of the event reconstruction image defogging unit is:

[0040] Event Image Reconstruction Using Multiscale Fusion Framework I E Atomization treatment;

[0041] Event Reconstruction Image I E Generate sharpened image I ES and gamma-corrected image sequence I EG ;

[0042] Event reconstruction image I E Generate sharpened image I ES The calculation expression is:

[0043] I ES =(I E +N{IE -G*I E}) / 2;

[0044] Where G*I represents the blurred version of the input image I, and N{} represents the linear normalization operator;

[0045] Event reconstruction image I E Generate gamma-corrected image sequence I EG The expression is:

[0046] I EGk =I E αk ;

[0047] Among them, α k is a positive constant, α k The value range is [1.0, 3.0], k is the number of images with different exposures, k = 3. α1, α2, α3 are three parameters to be adjusted;

[0048] Contrast weight map C i (x,y) and exposure weight map E i (x, y) is used as the index, and the Laplace pyramid fusion method is used to reconstruct four versions of the event image Fusion: A sharpened version I ES (x,y) and three gamma-corrected versions I EG (x,y), get the event image after defogging That is, the fog-free event map;

[0049] Contrast weight map C i (x,y) and exposure weight map E i The expressions for (x,y) are:

[0050]

[0051] Where L is the Laplace operator and σ is the parameter of the standard deviation, which is usually set to 0.2.

[0052] Preferably, the specific content of the RGB image defogging unit is:

[0053] Color Correction of RGB Images Using Multi-Scale Fusion Framework R Atomization treatment;

[0054] Color Correction RGB Image to Generate Sharp Image I RS and gamma-corrected image sequence I RG ;

[0055] Sharpen Image I RS The calculation expression is:

[0056] I RS =(I R +N{I R -G*I R}) / 2;

[0057] Gamma Corrected Image Sequence I RG The calculation expression is:

[0058]

[0059] Among them, β l is a positive constant, β l The value range is [1.0, 3.0], l is the number of images with different exposures, l = 3. β1, β2, β3 are three parameters to be adjusted;

[0060] Contrast weight map C j (x,y) and the exposure weight map T j (x, y) is used as the index, and the Laplacian pyramid fusion method is used to color correct the RGB image I R Four versions of Fusion: A sharpened version I RS (x,y) and 3 gamma-corrected versions I RG (x, y) is fused to obtain the RGB image after defogging That is, the fog-free RGB image;

[0061] Contrast weight map C i (x,y) and the saturation weight map T j The calculation expression of (x,y) is:

[0062]

[0063] in, is the jth version of the color-corrected RGB image; R j , G j , B j are the R, G, and B color channels of the jth version of the RGB image, It is the average of the three RGB color channel values.

[0064] Preferably, the enhanced environmental information data in S303 enters the image fusion unit to obtain a fused image, that is, an initial enhanced image, with the specific content being:

[0065] The illumination map is extracted from the enhanced RGB image using the RRDNET image decomposition method.

[0066] Use gamma correction to adjust the illumination map The expression for adjusting the illumination map using the gamma correction method is as follows:

[0067]

[0068] Among them, γ m is a positive constant, γ m The value range is [0.1, 3.0], m is the number of images with different exposures, m = 3. γ1, γ2, γ3 are three parameters to be adjusted.

[0069] Enhanced event reconstruction using gamma correction exposure, generating n multiple exposure images, and using gamma correction to enhance the event reconstruction image The expression of exposure is as follows:

[0070]

[0071] Among them, n is a positive constant, λ n The value range is [0.1, 3.0], n is the number of images with different exposures, n=3, λ1, λ2, λ3 are the three parameters to be adjusted;

[0072] The weight map of each artificially exposed image is set using three features: contrast, exposure, and average brightness;

[0073] Multi-scale fusion strategy is used to fuse multiple exposure image sequences and weight maps to generate fused images

[0074] Based on Retinex theory, the fused image Restore to RGB image and fuse the image The expression for restoring to RGB image is:

[0075]

[0076] Preferably, YOLOv5 is used as the basic framework to build an underwater target recognition algorithm model, and the initial enhanced image data set is input into the underwater target recognition algorithm model respectively to obtain the average precision mAP of multiple underwater target detections as follows:

[0077] S501, image input: The initial enhanced image is adjusted to a fixed size (640×640) and normalized to ensure that the image input to the YOLOv5 target detection model maintains a consistent scale in diverse environments.

[0078] S502, feature extraction and fusion: Use CSPDarknet53 as the backbone network to extract underlying features from the enhanced underwater images, and introduce the spatial attention mechanism to further improve the feature extraction capability of small and blurred targets in underwater environments. Use the feature pyramid FPN and path enhancement network PANet to fuse the extracted underlying features and generate multi-scale feature maps.

[0079] S504, bounding box and classification prediction: On each grid cell of the feature map, the model will simultaneously output multiple predictions, including the target box, target confidence, and target category incidence, and use maximum value suppression (NMS) to eliminate overlapping prediction boxes and retain the box with the highest confidence;

[0080] S506, model training: by calculating the gradient of the loss function, using the optimizer for back propagation, updating the weights of the target recognition network, and using loss functions including bounding box regression loss, classification loss, and confidence loss;

[0081] The formula of the total loss function L is as follows:

[0082] L=L box +L conf +L cls ;

[0083] Where: L box is the bounding box regression loss, L conf is the confidence loss, L cls is the classification loss;

[0084] S507. After the training is completed, the average precision mAP of target detection is calculated on the test set:

[0085]

[0086] Where C is the number of target categories, AP i is the average precision of the τth category.

[0087] Preferably, the target detection mean accuracy mAP is used as the fitness value of the genetic algorithm, and the initial enhancement parameters are optimized through selection-crossover-mutation operations to obtain the next generation population, and then the original image (environmental information) is enhanced. The maximum evolutionary number of the genetic algorithm is preset, and S5 and S6 are repeated until the genetic algorithm reaches the maximum evolutionary number to obtain the target enhancement parameters. The underwater multi-degraded image enhancement algorithm is used to enhance the environmental information in combination with the target enhancement parameters to obtain a target enhanced image, and the YOLOv5 model is used for target recognition. The specific content of the target recognition result can be obtained as follows:

[0088] Genetic algorithm is used to optimize the initial enhancement parameters, where the parameters involved include α1, α2, α3, β1, β2, β3, γ1, γ2, γ3, λ1, λ2, λ3. The specific steps are as follows:

[0089] S601, initialization: set the counter u = 0, the maximum evolutionary number U = 20, and the population size t = 30. Perform chromosome real number encoding on the 12 parameters to be adjusted in the multi-degraded image enhancement method to generate a sample p containing 12 genetic genes. t , combined with the range of the parameters to be adjusted, randomly initialize 30 samples {p1, p2, p3…p 30} as the initial population P0;

[0090] S602, sample fitness value calculation: Use the mean average precision (mAP) as the genetic fitness function, and calculate P one by one u The fitness value of all samples in;

[0091] S603, sample selection: based on population P u The fitness values ​​of the samples in the 30 samples are calculated, and the normalized probability of the fitness values ​​of the 30 samples is calculated. Then, the roulette method is used to select 30 samples to generate a new population P. u+1 ;

[0092] S604, crossover operation: select the seed from the population P according to the crossover probability u+1 Select two chromosomes, randomly determine the crossover position on the chromosome, and perform a crossover operation on the gene;

[0093] S605, mutation operation: according to the mutation probability, in the population P u+1 Randomly select the individuals that need to mutate, then randomly select two mutation positions on the chromosome and exchange the genes at these two positions;

[0094] S606, merging the selected parent chromosome with the offspring chromosome generated by crossover and mutation operations to form a new generation population P u+1 , calculate the new generation population P u+1 The fitness value of the current generation is determined, and whether the optimal result of the current generation meets the termination condition of the iteration. If u<U, the termination condition is not met, and S602-S606 are repeated;

[0095] If u=U, the iteration terminates and the sample with the largest fitness in the last generation of the population is output.

[0096] In summary, the present invention is a joint optimization method for underwater multi-degraded image enhancement and target recognition based on genetic algorithm. Compared with the traditional technology, the underwater target object is photographed by an underwater robot equipped with an event camera to obtain the corresponding event information and RGB image information. When applied, the underwater robot collects data in different water environments (such as clear waters, turbid waters, uneven lighting, etc.), and a diversified and comprehensive underwater target recognition data set can be constructed, which is convenient for training a more generalized target recognition model and improving the robustness of recognition. The image enhanced by the underwater multi-degraded image enhancement method based on event stream and RGB fusion can simultaneously achieve color correction, fog removal and uneven lighting detail restoration. By using the joint optimization method for underwater multi-degraded image enhancement and target recognition based on genetic algorithm, an underwater multi-degraded image enhancement result that is conducive to target recognition can be obtained, and the accuracy of underwater target recognition can be improved.

[0097] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 It is a flow chart of a joint optimization method for underwater multi-degraded image enhancement and target recognition based on genetic algorithm of the present invention;

[0099] Figure 2 A flow chart of multi-degraded image enhancement based on events and RGB fusion of a joint optimization method for underwater multi-degraded image enhancement and target recognition based on a genetic algorithm of the present invention;

[0100] Figure 3 A flow chart of a joint optimization method for underwater multi-degraded image enhancement and target recognition based on a genetic algorithm of the present invention, which uses a genetic algorithm to jointly optimize an underwater target recognition algorithm model and an underwater multi-degraded image enhancement method;

[0101] Figure 4 The present invention is a genetic algorithm flow chart of a genetic algorithm-based underwater multi-degraded image enhancement and target recognition joint optimization method. DETAILED DESCRIPTION

[0102] The technical solution of the present invention is further described below by means of the accompanying drawings and embodiments. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​described in these embodiments do not limit the scope of the present application.

[0103] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.

[0104] Technologies, systems, and devices known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, systems, and devices should be considered part of the specification.

[0105] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0106] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0107] like Figure 1 As shown, the present invention provides a joint optimization method for underwater multi-degraded image enhancement and target recognition based on genetic algorithm, S1, building an underwater robot target recognition system, the underwater robot target recognition system includes an underwater robot and an underwater event camera, and the underwater event camera is mounted on the underwater robot;

[0108] S2, acquiring underwater event camera data, analyzing the underwater event camera data to obtain environmental information of the underwater environment, wherein the environmental information, i.e., the underwater multi-degraded image, includes an event stream E0 and an RGB image, i.e., a color frame I0 captured by the underwater event camera, wherein the event stream E0 follows the RGB image and enhances the RGB image;

[0109] S3, using the target recognition data annotation tool to annotate the RGB image, and divide the environmental information according to the annotation results to obtain a sample data set;

[0110] S4, using genetic algorithm to initialize and obtain initial enhancement parameters;

[0111] S5, using initial enhancement parameters to perform initial enhancement on the environmental information to obtain an initial enhanced image data set;

[0112] like Figure 2 As shown, preferably, the specific contents of using the initial enhancement parameters to initially enhance the environmental information to obtain the initial enhanced image data set include:

[0113] S301, environmental information enters a preprocessing stage to obtain preprocessing information, the environmental information preprocessing stage includes an event stream reconstruction unit and an RGB image color correction unit, and the preprocessing information includes an event reconstruction image and an RGB color correction image;

[0114] S302, the pre-processed information enters the defogging stage to obtain enhanced environmental information data, wherein the enhanced environmental information data includes a fog-free event image and a fog-free RGB image, and the defogging stage includes an event reconstruction image defogging unit and an RGB image defogging unit;

[0115] S303: The enhanced environment information data enters the image fusion unit to obtain a fused image, ie, an initial enhanced image.

[0116] Preferably, the specific content of the event stream reconstruction unit is:

[0117] The event stream is a data sequence E i =(x i ,y i , t i , p i ), which contains information about the pixel position (x i ,y i ), sampling time t i and change symbol p i The information of the event stream and the RGB frame cannot be directly fused due to their different expressions, but should be converted into intensity images through preprocessing. ;

[0118] The event stream E0 is reconstructed into an intensity image I using the E2VID method E ;

[0119] The specific contents of the E2VID method include using a convolutional recurrent network to learn intensity frame reconstruction from asynchronous event data;

[0120] Building an improved recursive network based on the UNet architecture;

[0121] The event simulator is used to generate 1000 simulated event sequences from the event stream to train and test the network, of which 950 are training simulated event sequences and 50 are verification simulated event sequences.

[0122] Divide the simulation event sequence into a series of non-overlapping space-time windows and convert them into event tensors;

[0123] The simulated event sequence in each spatiotemporal window is sequentially input into the trained network, and the intensity frames are reconstructed to obtain the reconstructed event image.

[0124] Preferably, the specific content of the RGB image color correction unit is:

[0125] According to the average intensity value of the three color channels R, G, and B in the underwater RGB image;

[0126] The expression for calculating the average intensity value is as follows:

[0127]

[0128] Where H and W represent the height and width of the input image respectively.

[0129] According to the average intensity value from high to low, the three color channels of R, G, and B are divided into: high-quality color channels Medium quality color channel Low quality color channels

[0130] High-quality color channels For reference, the compensation factor J for medium quality color channel compensation and the compensation factor K for low quality color channel compensation are calculated respectively;

[0131] The formula for calculating the compensation factor J and the compensation factor K is as follows:

[0132]

[0133] Use gain (compensation) factors J and K to compensate for low- and medium-quality color channels;

[0134] The compensation expression is as follows:

[0135] I m (x,y)=I m (x,y)+(J×I hq (x,y));

[0136] I lq (x,y)=I lq (x,y)+(K×I hq (x,y));

[0137] The white balance algorithm based on the gray world hypothesis is used for further color correction to obtain the color-corrected RGB image I R That is, RGB color corrected image.

[0138] Preferably, the specific content of the event reconstruction image defogging unit is:

[0139] Event Image Reconstruction Using Multiscale Fusion Framework I E Atomization treatment;

[0140] Event reconstruction image I E Generate sharpened image I ES and gamma-corrected image sequence I EG ;

[0141] Event reconstruction image I E Generate sharpened image I ES The calculation expression is:

[0142] I ES =(I E +N{I E -G*I E}) / 2;

[0143] Where G*I represents the blurred version of the input image I, and N{} represents the linear normalization operator;

[0144] Event reconstruction image I E Generate gamma-corrected image sequence I EG The expression is:

[0145] I EGk =I E αk ;

[0146] Among them, α k is a positive constant, α k The value range is [1.0, 3.0], k is the number of images with different exposures, k = 3. α1, α2, α3 are three parameters to be adjusted;

[0147] Contrast weight map C i (x,y) and exposure weight map E i (x, y) is used as the index, and the Laplace pyramid fusion method is used to reconstruct four versions of the event image Fusion: A sharpened version I ES (x,y) and three gamma-corrected versions I EG (x,y), get the event image after defogging That is, the fog-free event map;

[0148] Contrast weight map C i (x,y) and exposure weight map E i The expressions for (x,y) are:

[0149]

[0150] Where L is the Laplace operator and σ is the parameter of the standard deviation, which is usually set to 0.2.

[0151] Preferably, the specific content of the RGB image defogging unit is:

[0152] Color Correction of RGB Images Using Multi-Scale Fusion Framework R Atomization treatment;

[0153] Color Correction RGB Image to Generate Sharp Image I RS and gamma-corrected image sequence I RG ;

[0154] Sharpen Image I RS The calculation expression is:

[0155] I RS =(I R+N{I R -G*I R}) / 2;

[0156] Gamma Corrected Image Sequence I RG The calculation expression is:

[0157]

[0158] Among them, β l is a positive constant, β l The value range is [1.0, 3.0], l is the number of images with different exposures, l = 3. β1, β2, β3 are three parameters to be adjusted;

[0159] Contrast weight map C j (x,y) and the exposure weight map T j (x, y) is used as the index, and the Laplacian pyramid fusion method is used to color correct the RGB image I R Four versions of Fusion: A sharpened version I RS (x,y) and 3 gamma-corrected versions I RG (x, y) is fused to obtain the RGB image after defogging That is, the fog-free RGB image;

[0160] Contrast weight map C i (x,y) and the saturation weight map T j The calculation expression of (x,y) is:

[0161]

[0162] in It is the average of the three RGB color channel values.

[0163] Preferably, the specific content of the enhanced environmental information data entering the image fusion unit in S303 to obtain the fused image, i.e., the initial enhanced image, is:

[0164] The illumination map is extracted from the enhanced RGB image using the RRDNET image decomposition method.

[0165] Use gamma correction to adjust the illumination map The expression for adjusting the illumination map using the gamma correction method is as follows:

[0166]

[0167] Among them, γ m is a positive constant, γ mThe value range is [0.1, 3.0], m is the number of images with different exposures, m = 3. γ1, γ2, γ3 are three parameters to be adjusted;

[0168] Enhanced event reconstruction using gamma correction exposure, generating n multiple exposure images, and using gamma correction to enhance the event reconstruction image The expression of exposure is as follows:

[0169]

[0170] Among them, n is a positive constant, λ n The value range is [0.1, 3.0], n is the number of images with different exposures, n=3, λ1, λ2, λ3 are the three parameters to be adjusted;

[0171] To ensure that more scene details are preserved and the brightness is consistent during the image fusion process, the weight map of each artificially exposed image is set using three features: contrast, exposure, and average brightness;

[0172] Multi-scale fusion strategy is used to fuse multiple exposure image sequences and weight maps to generate fused images

[0173] Based on Retinex theory, the fused image Restore to RGB image and fuse the image The expression for restoring to RGB image is:

[0174]

[0175] S6. Using YOLOv5 as the basic framework, an underwater target recognition algorithm model is constructed. The initial enhanced image data set is input into the underwater target recognition algorithm model to obtain the average precision mAP of multiple underwater target detections. The average precision mAP of multiple underwater target detections is input into the genetic algorithm to generate the next generation population through selection-crossover-mutation.

[0176] like Figure 3 As shown in the figure, taking YOLOv5 as the basic framework, an underwater target recognition algorithm model is constructed, and the initial enhanced image data set is input into the underwater target recognition algorithm model respectively, and the specific contents of obtaining the average precision mAP of multiple underwater target detection are as follows:

[0177] S501, image input: The initial enhanced image is adjusted to a fixed size (640×640) and normalized to ensure that the image input to the YOLOv5 target detection model maintains a consistent scale in diverse environments.

[0178] S502, feature extraction and fusion: Use CSPDarknet53 as the backbone network to extract underlying features from the enhanced underwater images, and introduce the spatial attention mechanism to further improve the feature extraction capability of small and blurred targets in underwater environments. Use the feature pyramid FPN and path enhancement network PANet to fuse the extracted underlying features and generate multi-scale feature maps.

[0179] S504, bounding box and classification prediction: On each grid cell of the feature map, the model will simultaneously output multiple predictions, including the target box, target confidence, and target category incidence, and use maximum value suppression (NMS) to eliminate overlapping prediction boxes and retain the box with the highest confidence;

[0180] S506, model training: by calculating the gradient of the loss function, using the optimizer for back propagation, updating the weights of the target recognition network, and using loss functions including bounding box regression loss, classification loss, and confidence loss;

[0181] The formula of the total loss function L is as follows:

[0182] L=L box +L conf +L cls ;

[0183] Where: L box is the bounding box regression loss, L conf is the confidence loss, L cls is the classification loss;

[0184] S507. After the training is completed, the average precision mAP of target detection is calculated on the test set:

[0185]

[0186] Where C is the number of target categories, AP i is the average precision of the τth category.

[0187] S7. Take the target detection average precision mAP as the fitness value of the genetic algorithm, optimize the initial enhancement parameters through selection-crossover-mutation operations to obtain the next generation population, and then enhance the original image (environmental information). The maximum evolutionary number of the genetic algorithm is preset, and S5 and S6 are repeated until the genetic algorithm reaches the maximum evolutionary number to obtain the target enhancement parameters. Combined with the target enhancement parameters, the underwater multi-degraded image enhancement algorithm is used to enhance the environmental information to obtain the target enhanced image, and the YOLOv5 model is used for target recognition to obtain the target recognition result.

[0188] like Figure 4As shown in the figure, the underwater optimization recognition accuracy is taken as the fitness value of the genetic algorithm, and the genetic algorithm is used to optimize the initial enhancement parameters and then enhance the initial image. The maximum evolutionary number of the genetic algorithm is preset, and S5 and S6 are repeated until the genetic algorithm reaches the maximum evolutionary number to obtain the target enhancement parameters. Combined with the target parameters, the event stream and RGB are fused to perform initial enhancement on the multi-degraded environment information to obtain the target enhanced image. The specific content is:

[0189] Genetic algorithm is used to optimize the initial enhancement parameters, where the parameters involved include α1, α2, α3, β1, β2, β3, γ1, γ2, γ3, λ1, λ2, λ3. The specific steps are as follows:

[0190] S601, initialization: set the counter u = 0, the maximum evolutionary number U = 20, and the population size t = 30. Perform chromosome real number encoding on the 12 parameters to be adjusted in the multi-degraded image enhancement method to generate a sample p containing 12 genetic genes. t , combined with the range of the parameters to be adjusted, randomly initialize 30 samples {p1, p2, p3…p 30} as the initial population P0;

[0191] S602, sample fitness value calculation: Use the mean average precision (mAP) as the genetic fitness function, and calculate P one by one u The fitness value of all samples in;

[0192] S603, sample selection: based on population P u The fitness values ​​of the samples in the 30 samples are calculated, and the normalized probability of the fitness values ​​of the 30 samples is calculated. Then, the roulette method is used to select 30 samples to generate a new population P. u+1 ;

[0193] S604, crossover operation: select the seed from the population P according to the crossover probability u+1 Select two chromosomes, randomly determine the crossover position on the chromosome, and perform a crossover operation on the gene;

[0194] S605, mutation operation: according to the mutation probability, in the population P u+1 Randomly select the individuals that need to mutate, then randomly select two mutation positions on the chromosome and exchange the genes at these two positions;

[0195] S606, merging the selected parent chromosome with the offspring chromosome generated by crossover and mutation operations to form a new generation population P u+1 , calculate the new generation population P u+1 The fitness value of the current generation is determined, and whether the optimal result of the current generation meets the termination condition of the iteration. If u<U, the termination condition is not met, and S602-S606 are repeated;

[0196] If u=U, the iteration is terminated and the sample with the largest fitness in the last generation of population is output, that is, the target enhancement parameter.

[0197] The target enhancement parameters are applied to the underwater multi-degraded image enhancement algorithm to enhance the environmental information, and the event stream is reconstructed into an intensity image through the event stream reconstruction module; the underwater image color is compensated through the RGB image color correction module; the image fogging and blur are removed through the RGB image defogging module and the event reconstruction image defogging module to improve the image clarity; the enhanced RGB and event images are fused through the RGB / event image fusion module to fuse the texture detail features of the two and better restore the details of the dark and bright areas.

[0198] The image enhanced by the underwater multi-degraded image enhancement method based on event stream and RGB fusion can simultaneously achieve color correction, haze removal and uneven illumination detail restoration to obtain the target enhanced image.

[0199] Then, the target enhanced image is input into the target recognition module, and the YOLOv5 model is used for target recognition to obtain the target recognition result.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A joint optimization method for underwater multi-degraded image enhancement and target recognition based on genetic algorithm, characterized in that: The following steps are involved: S1. Building an underwater robot target recognition system, wherein the underwater robot target recognition system includes an underwater robot and an underwater event camera, wherein the underwater event camera is mounted on the underwater robot; S2, acquiring underwater event camera data, using the event camera to record multi-degraded environment information in a complex underwater environment, wherein the multi-degraded environment information includes a multi-degraded event stream E0 and a multi-degraded RGB image, i.e., a multi-degraded color frame I0 captured by the underwater event camera, wherein the event stream E0 follows the RGB image respectively, and the event stream E0 enhances the multi-degraded RGB image in the underwater multi-degraded image enhancement process; S3, using the target recognition data annotation tool to annotate the RGB image, and divide the environmental information according to the annotation results to obtain a sample data set; S4, using genetic algorithm to initialize and obtain the initial population, i.e., the initial enhancement parameters; S5, using initial enhancement parameters to perform initial enhancement on the environmental information to obtain an initial enhanced image data set; S6. Using YOLOv5 as the basic framework, an underwater target recognition algorithm model is constructed, and the initial enhanced image data set is input into the underwater target recognition algorithm model to obtain the average precision mAP of underwater target detection; S7. Taking the mean average precision mAP of target detection as the fitness value of the genetic algorithm, the initial enhancement parameters are optimized through selection-crossover-mutation operations to obtain the next generation population, and then the original image, i.e., the multi-degraded environmental information, is enhanced. The maximum evolutionary number of the genetic algorithm is preset, and S5 and S6 are repeated until the genetic algorithm reaches the maximum evolutionary number to obtain the target enhancement parameters. Combined with the target enhancement parameters, the underwater multi-degraded image enhancement algorithm is used to enhance the environmental information to obtain the target enhanced image, and the YOLOv5 model is used for target recognition to obtain the optimized target recognition result.

2. According to the genetic algorithm-based joint optimization method for underwater multi-degraded image enhancement and target recognition according to claim 1, it is characterized in that: In S5, the initial enhancement parameters are used to initially enhance the environmental information to obtain the initial enhanced image data set. The specific contents include: S301, the environmental information enters the preprocessing stage to obtain preprocessing information, the environmental information preprocessing stage includes an event stream reconstruction unit, an RGB image color correction unit, and the preprocessing information includes an event reconstruction image I E and RGB color correction image I R ; S302, the pre-processed information enters the defogging stage to obtain enhanced environmental information data, wherein the enhanced environmental information data includes a fog-free event image and a fog-free RGB image, and the defogging stage includes an event reconstruction image defogging unit and an RGB image defogging unit; S303: The enhanced environment information data enters the image fusion unit to obtain a fused image, ie, an initial enhanced image.

3. The method for joint optimization of underwater multi-degraded image enhancement and target recognition based on genetic algorithm according to claim 2, characterized in that: The specific content of the event stream reconstruction unit in S301 is: The event stream is a data sequence E i =(x i ,y i , t i , p i ), which contains information about the pixel position (x i ,y i ), sampling time t i and change symbol p i information; The event stream E0 is reconstructed into an intensity image I using the E2VID method E ; The specific contents of the E2VID method include using a convolutional recurrent network to learn intensity frame reconstruction from asynchronous event data; Building an improved recursive network based on the UNet architecture; The event simulator is used to generate 1000 simulated event sequences from the event stream to train and test the network, of which 950 are training simulated event sequences and 50 are verification simulated event sequences. Divide the simulation event sequence into a series of non-overlapping space-time windows and convert them into event tensors; The simulated event sequence in each spatiotemporal window is sequentially input into the trained network, and the intensity frames are reconstructed to obtain the reconstructed event image.

4. The method for joint optimization of underwater multi-degraded image enhancement and target recognition based on genetic algorithm according to claim 3 is characterized in that: The specific content of the RGB image color correction unit in S301 is: According to the average intensity value of the three color channels R, G, and B in the underwater RGB image; The expression for calculating the average intensity value is as follows: Where H and W represent the height and width of the input image respectively. (x,y) is the pixel position, Represents the average intensity value of the three color channels R, G, and B of the input image. According to the average intensity value from high to low, the three color channels of R, G, and B are divided into: high-quality color channels Medium quality color channel Low quality color channels High-quality color channels For reference, the compensation factor J for medium quality color channel compensation and the compensation factor K for low quality color channel compensation are calculated respectively; The formula for calculating the compensation factor J and the compensation factor K is as follows: Use compensation factor J and compensation factor K to compensate for medium and low quality color channels; The compensation expression is as follows: I m (x,y)=I m (x,y)+(J×I hq (x,y)); I lq (x,y)=I lq (x,y)+(K×I hq (x,y)); The white balance algorithm based on the gray world hypothesis is used for further color correction to obtain the color-corrected RGB image I R That is, RGB color corrected image.

5. The method for joint optimization of underwater multi-degraded image enhancement and target recognition based on genetic algorithm according to claim 4, characterized in that: The specific contents of the event reconstruction image defogging unit in S302 are: Event Image Reconstruction Using Multiscale Fusion Framework I E Atomization treatment; Event reconstruction image I E Generate sharpened image I ES and gamma-corrected image sequence I EG ; Event reconstruction image I E Generate sharpened image I ES The calculation expression is: I ES =(I E +N{I E -G*I E }) / 2; Among them, G*I E Represents the input image I E , fuzzy version, N{} represents the linear normalization operator; Event reconstruction image I E Generate gamma-corrected image sequence I EG The expression is: I EGk =I E αk ; Among them, α k is a positive constant, α k The value range is [1.0, 3.0], k is the number of images with different exposures, k = 3. α1, α2, α3 are three parameters to be adjusted; Contrast weight map C i (x,y) and exposure weight map E i (x, y) is used as the index, and the Laplace pyramid fusion method is used to reconstruct the image I after four processed events i Fusion of (i=1,2,3,4): a sharpened version I ES (x,y) and three gamma-corrected versions I EG (x,y), get the event image after defogging That is, the fog-free event map; Contrast weight map C i (x,y) and exposure weight map E i The expressions for (x,y) are: Where (x, y) is the pixel position; in Reconstruct the image for the processed event, L is the Laplace operator, σ is the parameter of the standard deviation, usually set to 0.2, and exp is the exponential function.

6. The method for joint optimization of underwater multi-degraded image enhancement and target recognition based on genetic algorithm according to claim 5, characterized in that: The specific content of the RGB image defogging unit in S302 is: Color Correction of RGB Images Using Multi-Scale Fusion Framework R Atomization treatment; Color Correction RGB Image to Generate Sharp Image I RS and gamma-corrected image sequence I RG ; Sharpen Image I RS The calculation expression is: I RS =(I R +N{I R -G*I R }) / 2; Gamma Corrected Image Sequence I RG The calculation expression is: Among them, β l is a positive constant, β l The value range is [1.0, 3.0], l is the number of images with different exposures, l = 3. β1, β2, β3 are three parameters to be adjusted; Contrast weight map C j (x,y) and the exposure weight map T j (x, y) is used as the index, and the Laplacian pyramid fusion method is used to color correct the RGB image I R Four versions of Fusion: A sharpened version I RS (x,y) and 3 gamma-corrected versions I RG (x, y) is fused to obtain the RGB image after defogging That is, the fog-free RGB image; Contrast weight map C i (x,y) and the saturation weight map T j The calculation expression of (x,y) is: in, is the jth version of the color-corrected RGB image; R j , G j , B j are the R, G, and B color channels of the jth version of the RGB image, It is the average of the three RGB color channel values.

7. The method for joint optimization of underwater multi-degraded image enhancement and target recognition based on genetic algorithm according to claim 6, characterized in that: The specific contents of the enhanced environmental information data entering the image fusion unit in S303 to obtain the fused image, i.e. the initial enhanced image, are: The illumination map is extracted from the enhanced RGB image using the RRDNET image decomposition method. Use gamma correction to adjust the illumination map The expression for adjusting the illumination map using the gamma correction method is as follows: Among them, γ m is a positive constant, γ m The value range is [0.1, 3.0], m is the number of images with different exposures, m=3, γ1, γ2, γ3 are the three parameters to be adjusted; Enhanced event reconstruction using gamma correction exposure, generating n multiple exposure images, and using gamma correction to enhance the event reconstruction image The expression of exposure is as follows: Among them, λ n is a positive constant, λ n The value range is [0.1, 3.0], n is the number of images with different exposures, n=3, λ1, λ2, λ3 are the three parameters to be adjusted; The weight map of each artificially exposed image is set using three features: contrast, exposure, and average brightness; Multi-scale fusion strategy is used to fuse multiple exposure image sequences and weight maps to generate fused images Based on Retinex theory, the fused image Restore to RGB image F(x,y), and fuse the image The expression for restoring to RGB image is: in, is the RGB image after dehazing; To fuse the images.

8. The method for joint optimization of underwater multi-degraded image enhancement and target recognition based on genetic algorithm according to claim 7, characterized in that: Taking YOLOv5 as the basic framework, an underwater target recognition algorithm model is constructed. The initial enhanced image data set is input into the underwater target recognition algorithm model to obtain the average precision mAP of underwater target detection. The specific contents are as follows: S501, image input: The initial enhanced image is adjusted to a fixed size of 640×640 and normalized to ensure that the image input to the YOLOv5 target detection model maintains a consistent scale in diverse environments. S502, feature extraction and fusion: Use CSPDarknet53 as the backbone network to extract underlying features from the enhanced underwater images, and introduce the spatial attention mechanism to further improve the feature extraction capability of small and blurred targets in underwater environments. Use the feature pyramid FPN and path enhancement network PANet to fuse the extracted underlying features and generate multi-scale feature maps. S504, bounding box and classification prediction: On each grid cell of the feature map, the model will simultaneously output multiple predictions, including the target box, target confidence, and target category incidence, and use maximum value suppression (NMS) to eliminate overlapping prediction boxes and retain the box with the highest confidence; S506, model training: by calculating the gradient of the loss function, using the optimizer for back propagation, updating the weights of the target recognition network, and using loss functions including bounding box regression loss, classification loss, and confidence loss; The formula of the total loss function L is as follows: L=L box +L conf +L cls ; Where: L box is the bounding box regression loss, L conf is the confidence loss, L cls is the classification loss; S507. After the training is completed, the average precision mAP of target detection is calculated on the test set: Where C is the number of target categories, AP i is the average precision of the τth category.

9. The method for joint optimization of underwater multi-degraded image enhancement and target recognition based on genetic algorithm according to claim 8, characterized in that: The average precision mAP of underwater target detection is used as the fitness value of the genetic algorithm. The initial enhancement parameters are optimized through selection-crossover-mutation operations to obtain the next generation population, and then the original image (environmental information) is enhanced to obtain an enhanced image. The maximum evolutionary number of the genetic algorithm is preset. S5 and S6 are repeated until the genetic algorithm reaches the maximum evolutionary number to obtain the target enhancement parameters. Combined with the target enhancement parameters, the underwater multi-degraded image enhancement algorithm is used to enhance the environmental information to obtain the target enhanced image. The specific content is: Genetic algorithm is used to optimize the initial enhancement parameters, where the parameters involved include α1, α2, α3, β1, β2, β3, γ1, γ2, γ3, λ1, λ2, λ3. The specific steps are as follows: S601, initialization: set the counter u = 0, the maximum evolutionary number U = 20, and the population size t = 30. Perform chromosome real number encoding on the 12 parameters to be adjusted in the multi-degraded image enhancement method to generate a sample p containing 12 genetic genes. t , combined with the range of the parameters to be adjusted, randomly initialize 30 samples {p1, p2, p3…p 30 } as the initial population P0; S602, sample fitness value calculation: Use the mean average precision (mAP) as the genetic fitness function, and calculate P one by one u The fitness value of all samples in; S603, sample selection: based on population P u The fitness values ​​of the samples in the 30 samples are calculated, and the normalized probability of the fitness values ​​of the 30 samples is calculated. Then, the roulette method is used to select 30 samples to generate a new population P. u+1 ; S604, crossover operation: select the seed from the population P according to the crossover probability u+1 Select two chromosomes, randomly determine the crossover position on the chromosome, and perform a crossover operation on the gene; S605, mutation operation: according to the mutation probability, in the population P u+1 Randomly select the individuals that need to mutate, then randomly select two mutation positions on the chromosome and exchange the genes at these two positions; S606, merging the selected parent chromosome with the offspring chromosome generated by crossover and mutation operations to form a new generation population P u+1 , calculate the new generation population P u+1 The fitness value of the current generation is determined, and whether the optimal result of the current generation meets the conditions for terminating the iteration; If u<U, the termination condition is not met, then S602-S606 are repeated; If u=U, the iteration terminates and the sample with the largest fitness in the last generation of the population is output.

Images