Target detection method, system and device based on image data augmentation and image vision enhancement algorithm and medium

Abstract

The invention relates to the technical field of image data processing, and discloses a target detection method, system and device based on an image data augmentation and image vision enhancement algorithm and a medium. A parameter prediction sub-module (Pe-PP module) is adopted to carry out visual enhancement preprocessing on the image on the basis of parameters provided by the parameter prediction sub-module (Pe-PP module), then a visual enhancement sub-module (VE module) is adopted to carry out contrast, brightness and white balance adjustment, and finally a target detection result is output; the target detection system comprises a target database construction module, a joint training module and a target detection module. The equipment and the medium are used for storing a computer program, and a target detection method is realized when the program is executed; according to the method, the problem of target category imbalance can be relieved, the overlapped target detection capability of the model and the overall detection precision are improved, and the method has good performance for solving the visual quality problems of image target category imbalance, low brightness and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing, and in particular relates to a target detection method, system, device and medium based on image data augmentation and image vision enhancement algorithm. Background Art

[0002] Object detection is a very challenging research topic in the field of computer vision. Traditional marine environmental exploration mainly relies on human diving, which is not only inefficient but also requires high skills and physical fitness of divers. Therefore, using underwater detection technologies such as remotely operated underwater vehicles (ROVs) and autonomous underwater vehicles (AUVs) to detect underwater environments and develop marine ecological resources has become a more ideal solution. These devices can collect valuable data and images in deep-sea areas that are difficult for humans to reach, providing scientists and engineers with extremely valuable information resources.

[0003] As one of the key technologies and application areas for processing, analyzing and effectively utilizing these data and images, target detection is the core of realizing automated detection tasks. Through the sensor and camera data collected by ROV and AUV, underwater target detection algorithms can automatically identify targets, significantly improve data processing efficiency and reduce the need for manual intervention, thereby accelerating the progress of underwater detection tasks. The application of deep learning technology further improves the performance of these algorithms, enabling them to achieve high-precision target recognition under complex visual conditions such as lighting changes and turbid water quality, which is critical for adapting to the uncertainty and changes in the underwater environment.

[0004] Underwater target detection algorithms are mainly divided into methods based on sonar images and methods based on optical images. Sonar images have the characteristics of low resolution, slow acquisition and processing speed, and inability to achieve real-time high-resolution imaging, which limits their application in the field of target detection. In contrast, optical images have greater development potential due to their high resolution, ability to be directly acquired through optical imaging equipment, real-time high-resolution acquisition and processing, and rich color and texture information. An ideal underwater optical target detection algorithm based on deep learning should show high detection accuracy and strong robustness in complex underwater environments.

[0005] However, compared with general target detection algorithms, underwater optical target detection algorithms face many challenges: (1) Unbalanced target categories: The randomness of underwater optical image capture and the limited data collection scenarios result in some categories of underwater targets appearing less frequently in the dataset; (2) Target aggregation and mutual occlusion: Due to their unique living habits, some underwater organisms tend to cluster in space, which often leads to mutual occlusion between targets, bringing additional challenges to deep learning-based target detectors in extracting target details, thereby increasing the difficulty of detection; (3) Image visual quality issues: The scattering and absorption of light by water, the use of artificial light sources and the presence of impurities in water, result in images acquired by underwater optical imaging systems often being affected by low brightness, low contrast and color distortion. These problems make underwater optical images inferior to atmospheric optical images in terms of color and texture information, thus limiting the performance of deep learning-based target detectors.

[0006] In the prior art, the invention with publication number "CN202410154598" discloses "a small sample learning remote sensing image detection method based on the SAM model", which improves the overall generalization ability of the detection model through the generalization segmentation ability of the SAM model, but it does not add new training samples and cannot solve the problem of too few training samples.

[0007] In the prior art, the invention with publication number "CN202410292522" discloses "a low-illumination underwater target detection method based on improved YOLOv8n". By adding the Double MSRCR algorithm to the input end of the model, the input data set is enhanced to improve dim, blurred and color-distorted images, thereby improving the performance of the target detector. However, the image enhancement algorithm and the target detector are processed as a cascaded independent process, and the two cannot be regarded as a common learning process, which may have a negative impact on underwater detection. Summary of the invention

[0008] In view of the deficiencies in the above-mentioned prior art, the purpose of the present invention is to propose a target detection method, system, device and medium based on image data augmentation and image visual enhancement algorithm. The method constructs a target database by using the SAM visual large model, adopts Poisson fusion for sample reorganization, and performs visual enhancement preprocessing on the image based on the parameters provided by the parameter prediction submodule (Pe-PP module), and then adopts the visual enhancement submodule (VE module) to perform contrast, brightness and white balance adjustment. The present invention can alleviate the problem of target category imbalance, and at the same time improve the model's ability to detect overlapping targets, thereby improving the overall detection accuracy, and has good performance in solving image target category imbalance and visual quality problems such as low brightness, low contrast, color distortion, etc.

[0009] To achieve the above purpose, the technical solution adopted by the present invention is:

[0010] In a first aspect, a target detection method based on image data augmentation and image visual enhancement algorithm comprises the following steps:

[0011] S1. Input the image training set into the SAM model, segment the image training set according to the “bounding box prompt” in the SAM model, and form and output a foreground object mask after segmentation;

[0012] S2, the foreground target mask and the image training set described in step S1 are accurately cropped according to the coordinates of the "bounding box prompt" described in step S1, and mask image blocks and foreground target image blocks of the same size as the "bounding box prompt" are obtained respectively, and the mask image blocks and foreground target image blocks are filtered and screened to construct the target database D object ;

[0013] S3, from the target database D described in step S2 object Randomly select a foreground target image i, and perform mask-based recombination on the foreground target image i and the image training set described in step S1 to obtain a new training sample;

[0014] S4, using the parameter prediction submodule (Pe-PP module), the visual enhancement submodule (VE module) and the target detector, jointly training the new training examples described in step S3 and the image training set described in step S1 to obtain a trained target detection model;

[0015] S5. Use the trained target detection model described in step S4 to detect the image to be detected, and output the target detection result.

[0016] Further, step S2 is implemented according to the following steps:

[0017] S2.1, according to the coordinates of the "bounding box hint" in step S1, the foreground target mask in step S1 and the background image in the image training set are respectively cropped to obtain the target area, so as to obtain the mask image block and the foreground target image block of the same size as the "bounding box hint";

[0018] S2.2, for the foreground target image blocks and the corresponding mask image blocks obtained in step S2.1 that do not meet the use standard, the data are manually screened according to the principle that the mask image blocks contain the foreground target, contain a single foreground target, and the single foreground target image is clear;

[0019] S2.3, after the manual screening described in step S2.2, the qualified foreground target image blocks and their corresponding mask image blocks are classified and integrated to obtain the target database Dobject .

[0020] Further, step S3 is implemented according to the following steps:

[0021] S3.1. Select a background image I from the image training set t and the set L of existing target markers t , t represents the tth image in the image training set;

[0022] S3.2, determine the background image I to be used together with the background image I in step S3.1 t The target category c and target number n to be reorganized;

[0023] S3.3, from the target database D in step S2 object Randomly extract a foreground target image i of target category c from the image, and read the RGB image O of the foreground target image i i and foreground object mask M i ;

[0024] S3.4, RGB image O of the foreground target image i described in step S3.3 i and foreground object mask M i Perform random rotation or flipping, and generate the rotated image trans_O after processing. i and rotated foreground object mask trans_M i ;

[0025] S3.5, adjust the rotated image trans_O described in step S3.4 i and rotated foreground object mask trans_M i The size of the background image I t Same, generate resize_O images respectively i and size foreground object mask resize_M i ;

[0026] S3.6, determine the foreground target image i in step S3.3 in the background image I t The center point coordinates (x center ,y center ), the center point coordinate (x center ,y center ) needs to follow two conditions: the target frame of the newly added foreground target image i and the background image I t The IoU (intersection over union) of the original target box in is less than 0.5, and the target box of the newly added foreground target image i and the background image I t The original target boxes do not contain each other;

[0027] S3.7, resize_0 the image rotated to the size described in step S3.5 i 、Size foreground object mask resize_M i 、Background Image I t and the center point coordinates (x center ,y center ) is input into the Poisson fusion algorithm to perform Poisson fusion to complete the foreground target image i and the background image I t reorganization;

[0028] S3.8, rotate the image resize_0 according to the size described in step S3.5 i The height h and width w and the center point coordinates (x center ,y center ), calculate the reconstructed foreground target image i in the background image I t The bounding box coordinates (x min ,y min )、(x max ,y max ), the bounding box coordinates (x min ,y min )、(x max ,y max ) together with the target category c constitutes the label l of the foreground object i i ;

[0029] S3.9, repeat steps S3.3 to S3.8 until a new foreground target image i and label l are added. i The preset target number n is reached;

[0030] S3.10, output the foreground target image i described in step S3.9 as a new training example, and mark l i Add to the existing set of target markers L t A corresponding tag set is formed in .

[0031] Furthermore, the Poisson fusion described in step S3.7 is implemented by the following algorithm:

[0032] Let the function representing the pixels of the fused image within Ω be f, and the function representing the pixels of the fused image outside Ω be f * , the smoothing condition can be expressed as:

[0033]

[0034] In formula (4-1), v represents the gradient field of the foreground image to be synthesized;

[0035] The condition for keeping the boundary consistent can be expressed as:

[0036]

[0037] make

[0038] F=|▽fv 2 =(▽f x -v x ) 2 +(▽f y -v y ) 2 (4-8)

[0039] Substituting formula (4-3) into the Euler-Lagrange equation, we have:

[0040]

[0041] In formula (4-4), x represents the horizontal coordinate of the pixel point in the image, and y represents the vertical coordinate of the pixel point in the image;

[0042] F is a function of ▽f, Therefore, it can be obtained

[0043]

[0044] In formula (4-5), Δf=div v is the solution of Poisson's equation.

[0045] Further, step S4 is implemented according to the following steps:

[0046] The image training set described in step S1 and the new training examples described in step S3.10 are divided into two paths. One path is resized to 416×416 pixels or 640×640 pixels as the input of the visual enhancement submodule (VE module), wherein the visual enhancement submodule (VE module) is a series connection of three visual enhancement algorithms in the order of contrast adjustment, brightness adjustment and white balance operation; the other path is resized to 256×256 pixels as the input of the parameter prediction submodule (Pe-PP module), and the parameter prediction submodule (Pe-PP module) infers and outputs the visual The configuration parameters of the enhancement submodule (VE module) are output to the visual enhancement submodule (VE module). After receiving the configuration parameters, the visual enhancement submodule (VE module) can perform visual enhancement processing on the input image of 416×416 pixels or 640×640 pixels, and use the processed image as the input of the target detector, calculate the loss of the output of the target detector and the corresponding label set, and update the parameters of the parameter prediction submodule (Pe-PP module) and the target detector with the goal of minimizing the loss to obtain a trained target detection model.

[0047] Further, step S5 is implemented according to the following steps:

[0048] The image to be detected is input into the trained target detection model described in step S4. The target detection model infers the input image to be detected and outputs the target detection result, thereby completing the target detection task.

[0049] In a second aspect, a target detection system based on image data augmentation and image visual enhancement algorithm, applying the target detection method, comprises a target database construction module, a joint training module and a target detection module:

[0050] Target database construction module: input the image training set into the SAM model, segment the image training set according to the "bounding box prompt" in the SAM model, form and output the foreground target mask after segmentation; accurately crop the foreground target mask and the image training set according to the coordinates of the "bounding box prompt" to obtain mask image blocks and foreground target image blocks of the same size as the "bounding box prompt" respectively, filter and screen the mask image blocks and foreground target image blocks to construct the target database D object ;

[0051] Joint training module: From the target database D object A foreground target image i is randomly selected, and the foreground target image i and the image training set are recombined based on a mask to obtain a new training sample; the new training sample and the image training set are jointly trained using a parameter prediction submodule (Pe-PP module), a visual enhancement submodule (VE module) and a target detector to obtain a trained target detection model;

[0052] Object detection module: uses the trained object detection model to detect the image to be detected and outputs the object detection result.

[0053] In a third aspect, an electronic device includes: a memory for storing a computer program; and a processor for executing the computer program to implement the target detection method.

[0054] In a fourth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program implements the target detection method when executed by a processor.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] 1. Steps S1 and S2 of the present invention use a large visual model SAM to perform data augmentation, which can alleviate the problem of insufficient learning of a smaller number of categories caused by an imbalance in the number of target categories and improve the model's ability to detect overlapping targets.

[0057] 2. In step S3 of the present invention, the Poisson fusion algorithm is used to naturally fuse the foreground target image and the new background image, thereby ensuring the consistency of the foreground target image and the new background image in color, lighting and contrast, ensuring the naturalness and authenticity of the reconstructed new training samples, and effectively avoiding the domain shift problem that may be caused by direct pasting.

[0058] 3. In step S4 of the present invention, the image preprocessing technology of the visual enhancement algorithm (i.e., the visual enhancement processing of the visual enhancement submodule) and the joint training (i.e., the training method of the parameter prediction submodule) is adopted, which can jointly train the visual enhancement algorithm and the target detection model end-to-end, update the parameters by sharing the target detection loss, and adaptively adjust the visual enhancement parameters for various images, thereby improving the visual quality of the image.

[0059] In summary, the present invention can alleviate the problem of target category imbalance through the image data augmentation method based on the visual large model SAM, the visual enhancement algorithm and the image preprocessing technology of joint training, while improving the model's ability to detect overlapping targets, thereby improving the overall detection accuracy, and has good performance in solving image target category imbalance and visual quality problems such as low brightness, low contrast, and color distortion. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flow chart of the steps of the present invention.

[0061] Figure 2(a) to Figure 2(d) They are respectively foreground target masks of different categories of sea cucumbers, scallops, starfish, and sea urchins in the underwater image obtained by SAM model segmentation in Example 1.

[0062] Figure 3(a) to Figure 3(d) They are respectively the underwater target database D of the first embodiment object Foreground target image patches of sea cucumber, scallop, starfish, and sea urchin and their corresponding mask image patches.

[0063] Figure 4(a) to Figure 4(d) They are comparison diagrams of different underwater images before and after being processed by the VE module after joint training in Example 1.

[0064] FIG. 5( a ) and FIG. 5( b ) are comparison diagrams of target detection results before and after the VE module is processed after joint training for different images to be detected in Example 1. FIG. DETAILED DESCRIPTION

[0065] The following is combined with Figure 1 5 and the embodiments of the present invention are described in detail. Figure 1 The flowchart of the steps of the present invention is shown as a target detection method based on image data augmentation and image visual enhancement algorithm, comprising the following steps:

[0066] S1. Input the image training set into the SAM model, segment the image training set according to the “bounding box prompt” in the SAM model, and form and output a foreground object mask after segmentation;

[0067] S2, the foreground target mask and the image training set described in step S1 are accurately cropped according to the coordinates of the "bounding box prompt" described in step S1, and mask image blocks and foreground target image blocks of the same size as the "bounding box prompt" are obtained respectively, and the mask image blocks and foreground target image blocks are manually filtered and screened to construct the target database D object ;

[0068] S3, from the target database D described in step S2 object Randomly select foreground target images i from step S1, and perform mask-based recombination on these foreground target images i and the image training set described in step S1 to obtain new training samples;

[0069] S4, using the parameter prediction submodule (Pe-PP module), the visual enhancement submodule (VE module) and the target detector, jointly training the new training examples described in step S3 and the image training set described in step S1 to obtain a trained target detection model;

[0070] S5. Use the trained target detection model described in step S4 to detect the image to be detected, and output the target detection result to complete the target detection task.

[0071] Embodiment 1

[0072] This target detection method takes underwater images as the object and carries out image data augmentation and image vision enhancement processing.

[0073] The detailed process of step S1 is as follows: using the labeled underwater image as the image training set to input the SAM model, using the SAM model to segment the image in the image training set and obtain the foreground target mask, and using the manually marked bounding box as the segmentation prompt, the output of the model is the foreground target mask. In this way, the SAM model uses its "prompt segmentation" ability to achieve effective segmentation of the foreground target and background in the underwater image, and generates a foreground target mask. Figure 2(a) to Figure 2(d) They are respectively foreground target masks of different categories of sea cucumbers, scallops, starfish, and sea urchins in the underwater image obtained by SAM model segmentation in Example 1.

[0074] Step S2 is specifically implemented according to the following steps:

[0075] S2.1, Separate the foreground target from the background image in the image training set: According to the coordinates of the "bounding box prompt" in step S1, the foreground target mask (i.e., the binary Mask image) in step S1 and the corresponding background image in the image training set are respectively cropped to obtain the target area, thereby obtaining the mask image block and the foreground target image block of the same size as the "bounding box prompt". This process separates the foreground target from the background image in the image training set, which is the key to constructing the underwater target database D object basis;

[0076] S2.2, Screening and filtering of foreground target images: For the foreground target image blocks and the corresponding mask image blocks obtained in step S2.1 that do not meet the use standards (i.e., they may not have sufficient clarity or accuracy), the data are strictly screened manually based on the principle that the mask image blocks contain foreground targets, contain single foreground targets, and the single foreground target image is clear, so as to ensure that the constructed underwater target database D object The quality of the model is to provide high-quality data for model training. Figure 3(a) to Figure 3(d) They are respectively the underwater target database D of the first embodiment object Foreground target image patches of sea cucumber, scallop, starfish, and sea urchin and their corresponding mask image patches;

[0077] S2.3. Building underwater target database D object : After the manual screening described in step S2.2, the qualified foreground target image blocks and their corresponding mask image blocks are classified and integrated to obtain the target database D object Finally, we get an underwater target database D containing four types of targets: sea cucumbers, scallops, and starfish. object .

[0078] Step S3 is based on the cluster distribution simulation strategy and small sample category supplementation strategy designed in this paper, from the target database D object Randomly select foreground target images from the training set, and perform mask-based recombination on these foreground target images and training set images to obtain new training samples. Step S3 is specifically implemented according to the following steps:

[0079] S3.1. Select an underwater background image I from the image training set t and the set L of existing target markers t (ie background image I t The corresponding “bounding box hint”), t represents the tth image in the image training set;

[0080] S3.2, determine the underwater background image I described in step S3.1 tThe target category c and the target number n to be reorganized. In the underwater image training set (UPRC2022 dataset) in Example 1, the target category c is 4 categories. The 4 categories are sorted from small to large according to the number of targets: sea cucumber, scallop, starfish and sea urchin. The number of sea urchins is significantly larger than the other three categories. Therefore, sea cucumber, scallop and starfish are randomly selected with a probability of 50%, 30% and 20% respectively to supplement the number of targets in the minority sample category;

[0081] S3.3, from the target database D in step S2 object A foreground target image i of target category c is randomly selected from the cluster distribution simulation strategy, and the RGB image O of the foreground target image i is read. i and foreground object mask M i ;

[0082] S3.4, in order to increase the diversity of target morphology, the RGB image O of the foreground target image i described in step S3.3 is i and foreground object mask M i Perform random rotation or flipping, and generate the rotated image trans_O after processing. i and rotated foreground object mask trans_M i ;

[0083] S3.5, adjust the rotated image trans_O described in step S3.4 i and rotated foreground object mask trans_M i The size of the background image I t Same, after resizing, generate resize_O respectively i and size foreground object mask resize_M i ;

[0084] S3.6, determine the foreground target image i in step S3.3 in the background image I t The center point coordinates (x center ,y center ), that is, the fusion center. The selection of the fusion center needs to follow two conditions: the target frame of the newly added foreground target image i and the background image I t The IoU (intersection over union) of the original target box in is less than 0.5; the target box of the newly added foreground target image i and the background image I t The original target boxes do not contain each other; because mutual inclusion will cause complete occlusion, which makes the model unable to learn the features of the occluded target;

[0085] S3.7, resize_0 the image rotated to the size described in step S3.5 i、Size foreground object mask resize_M i 、Background Image I t and the center point coordinates (x center ,y center ) is used as input to the Poisson fusion algorithm to perform Poisson fusion and complete the foreground target image i and the background image I t reorganization;

[0086] S3.8, rotate the image resize_0 according to the size described in step S3.5 i The height h and width w and the center point coordinates (x center ,y center ), calculate the reconstructed foreground target image i in the background image I t The bounding box coordinates (x min ,y min )、(x max ,y max ), the bounding box coordinates (x min ,y min )、(x max ,y max ) together with the target category c constitutes the label l of the foreground object i i ;

[0087] S3.9, repeat steps S3.3 to S3.8 until a new foreground target image i and label l are added. i The preset target number n is reached;

[0088] S3.10, output the foreground target image i described in step S3.9 as a new training example, and mark l i Add to the existing set of target markers L t A corresponding tag set is formed in .

[0089] In the image fusion task, in order to achieve a natural transition between the foreground and the background, two aspects need to be focused on: content smoothness and boundary consistency. Content smoothness means that the image gradients of the foreground and background should be as similar as possible to ensure visual consistency within the fusion area. Boundary consistency requires that the pixel values ​​of the foreground and background must be the same at the fusion edge to avoid visual discontinuities or obvious edges. The Poisson fusion described in step S3.7 is implemented by the following algorithm:

[0090] Let the function representing the pixels of the fused image inside Ω be f, and the function representing the pixels of the fused image outside Ω be f * , the smoothing condition can be expressed as:

[0091]

[0092] In formula (4-1), v represents the gradient field of the foreground image to be synthesized.

[0093] The condition to keep the boundary constant can be expressed as:

[0094]

[0095] make

[0096] F=|▽fv 2 =(▽f x -v x ) 2 +(▽f y -v y ) 2 (4-3)

[0097] Substituting into the Euler-Lagrange equation, we have:

[0098]

[0099] In formula (4-4), x represents the horizontal coordinate of the pixel point in the image, and y represents the vertical coordinate of the pixel point in the image.

[0100] F is a function of ▽f, so From this we can get

[0101]

[0102] div v is a known quantity that can be calculated. Its determination process needs to take into account both the foreground target image i and the background image I t This can be achieved by directly using the divergence value of the foreground target image i, or by using the divergence value of the foreground target image i and the background image I t The divergence of is realized by fusing the corresponding points, because the boundaries need to be consistent, and the pixel value on the boundary of the foreground target image i is equal to the background image I t At this point, the boundary conditions and divergence are known, and solving the Poisson equation Δf=div v in the discrete space can obtain the pixel value of the fused image within Ω.

[0103] Step S4 trains the parameter prediction submodule to predict the hyperparameters of the visual enhancement algorithm. Step S4 is specifically implemented according to the following steps:

[0104] The image training set described in step S1 and the new training examples described in step S3.10 are divided into two paths, one path is resized to 416×416 pixels or 640×640 pixels as the input of the visual enhancement submodule (VE module), wherein the visual enhancement submodule (VE module) is a series connection of three visual enhancement algorithms in the order of contrast adjustment, brightness adjustment and white balance operation; the other path is resized to 256×256 pixels as the input of the parameter prediction submodule (Pe-PP module), the parameter prediction submodule (Pe-PP module) infers and outputs the configuration parameters of the visual enhancement submodule (VE module), and outputs them to the visual enhancement submodule (VE module), after the visual enhancement submodule (VE module) receives the configuration parameters, it can perform visual enhancement processing on the input image of 416×416 pixels or 640×640 pixels, and use the processed image as the input of the target detector; Figure 4(a) to Figure 4(d) They are comparison diagrams of different underwater images before and after processing by the VE module after joint training in Example 1. In Figure 4(a), the underwater image has high brightness, normal contrast and color fidelity after processing. In Figure 4(b), the underwater image has color fidelity after processing. In Figure 4(c), the underwater image has color fidelity after processing. In Figure 4(d), the target is clearly identifiable after underwater image processing.

[0105] The Pe-PP module is a lightweight convolutional neural network (PeleeNet). Both the Pe-PP module and the target detector need to update parameters. During the training process, the Pe-PP module and the target detection network share the same loss function, namely the target detection loss. The loss is calculated by the output of the target detector and the corresponding label set, and the parameters of the Pe-PP module and the target detector are updated with the goal of minimizing the loss to obtain a trained target detection model.

[0106] Step S5: Input the underwater image to be detected into the target detection model trained in step S4. The target detection model infers the input image to be detected and outputs the target detection result, thereby completing the target detection task.

[0107] Specifically, after the visual enhancement submodule (VE module) receives the configuration parameters inferred by the parameter prediction submodule (Pe-PP module), it can perform visual enhancement processing on the 416×416 or 640×640 input image and use the processed image as the input of the object detector. The object detector infers the image output by the visual enhancement submodule (VE module) and outputs the object detection result to complete the object detection task. Figure 5(a) to Figure 5(b)The figure is a comparison chart of the target detection results before and after the VE module is processed after the joint training for different images to be detected in Example 1. The right figures in Figure 5(a) and Figure 5(b) are the target detection results after the VE module is processed. The red circles represent the blurred and overlapping targets newly detected by the target detection model after the VE module is processed. The target detection results indicate that the method can achieve better detection results in images to be detected with low brightness, low contrast and less color distortion.

[0108] In order to demonstrate the superiority of Example 1 of the detection method, YOLOX is used as the baseline model and URPC2022 is used as the experimental data set. A comparative experiment is designed to verify the effectiveness of the image data augmentation method based on the visual large model SAM (SAM-AUG) and the image preprocessing method based on the visual enhancement algorithm and joint training (VEJT-IP).

[0109] It can be seen from Tables 1 and 2 that the image data augmentation method (SAM-AUG) of the visual large model SAM in Example 1 of the detection method enhances the model's detection accuracy for small sample categories, and the average accuracy on sea cucumbers, sea urchins, and starfish is improved by 1.92%, 0.6%, and 0.24%, respectively, and the overall improvement is 0.7%, thereby optimizing the overall performance of the model. At the same time, the image preprocessing method based on the visual enhancement algorithm and joint training (VEJT-IP) is capable of improving target detection accuracy, especially YOLOX-Nano, YOLOX-s, and YOLOX-l in mAP@0.5:0.95. Improved by 0.22%, 0.81%, and 0.39%, respectively, indicating that the image preprocessing method can improve the mAP under different positioning accuracy requirements (corresponding to different IoU thresholds).

[0110] Table 1 Comparative experimental results before and after the introduction of SAM-AUG image data augmentation

[0111]

[0112] Table 2 Comparative experimental results for models of different sizes before and after the introduction of the VEJT-IP method

[0113]

[0114] A target detection system based on image data augmentation and image visual enhancement algorithm, applying the target detection method, comprises a target database construction module, a joint training module and a target detection module:

[0115] Target database construction module: input the image training set into the SAM model, segment the image training set according to the "bounding box prompt" in the SAM model, form and output the foreground target mask after segmentation; accurately crop the foreground target mask and the image training set according to the coordinates of the "bounding box prompt" to obtain mask image blocks and foreground target image blocks of the same size as the "bounding box prompt" respectively, filter and screen the mask image blocks and foreground target image blocks to construct the target database D object ;

[0116] Joint training module: From the target database D object A foreground target image i is randomly selected, and the foreground target image i and the image training set are recombined based on a mask to obtain a new training sample; the new training sample and the image training set are jointly trained using a parameter prediction submodule (Pe-PP module), a visual enhancement submodule (VE module) and a target detector to obtain a trained target detection model;

[0117] Object detection module: uses the trained object detection model to detect the image to be detected and outputs the object detection result.

[0118] An electronic device includes: a memory for storing a computer program; the memory, as a carrier for storing resources, may be a read-only memory, a random access memory, a disk or an optical disk, etc., and the resources stored thereon may include an operating system, a computer program, etc., and the storage method may be temporary storage or permanent storage; a processor, for executing the computer program to implement the target detection method, the processor may be composed of an integrated circuit, for example, it may be composed of a single packaged integrated circuit, or it may be composed of a plurality of packaged integrated circuits with the same function or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors and various control chips, etc.

[0119] A computer-readable storage medium for storing a computer program, which implements the target detection method when executed by a processor; the computer-readable storage medium may include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, DVD-ROM, Blu-ray or optical disk memory, hard disk drive (HDD), solid state drive (SSD), card memory (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device.

[0120] The working principle of the present invention is:

[0121] Firstly, new training samples are formed through SAM model and Poisson fusion. The parameter prediction submodule (Pe-PP module), visual enhancement submodule (VE module) and target detector are jointly trained with the new training samples and the original data set to obtain the target detection model. The image to be detected is input into the target detection model. The target detection model divides the input image into two paths. One path is resized to 416×416 or 640×640 as the input of the VE module; the other path is resized to 256×256 as the input of the parameter prediction submodule (Pe-PP module); the parameter prediction The parameter prediction submodule (Pe-PP module) infers and outputs the configuration parameters of the visual enhancement submodule (VE module) and outputs them to the visual enhancement submodule (VE module); after the visual enhancement submodule (VE module) receives the configuration parameters inferred and output by the parameter prediction submodule (Pe-PP module), it can perform visual enhancement processing on the 416×416 or 640×640 input image and use the processed image as the input of the target detector; the target detector infers the image output by the visual enhancement submodule (VE module) and outputs the target detection result to complete the target detection task.

Claims

1. A target detection method based on image data augmentation and image visual enhancement algorithm, characterized in that: The following steps are involved: S1. Input the image training set into the SAM model, segment the image training set according to the "bounding box prompt" in the SAM model, and form and output a foreground object mask after segmentation; S2, the foreground target mask and the image training set described in step S1 are accurately cropped according to the coordinates of the "bounding box prompt" described in step S1, and the mask image blocks and the foreground target image blocks of the same size as the "bounding box prompt" are obtained respectively, and the mask image blocks and the foreground target image blocks are filtered and screened to construct the target database D object ; S3, from the target database D described in step S2 object Randomly select a foreground target image i, and perform mask-based recombination on the foreground target image i and the image training set described in step S1 to obtain a new training sample; S4, using the parameter prediction submodule (Pe-PP module), the visual enhancement submodule (VE module) and the target detector, jointly training the new training examples described in step S3 and the image training set described in step S1 to obtain a trained target detection model; S5. Use the trained target detection model described in step S4 to detect the image to be detected, and output the target detection result.

2. The target detection method according to claim 1, characterized in that: Step S2 is implemented according to the following steps: S2.1, according to the coordinates of the "bounding box hint" in step S1, the foreground target mask in step S1 and the background image in the image training set are respectively cropped to obtain the target area, so as to obtain the mask image block and the foreground target image block of the same size as the "bounding box hint"; S2.2, for the foreground target image blocks and the corresponding mask image blocks obtained in step S2.1 that do not meet the use standard, the data are manually screened according to the principle that the mask image blocks contain the foreground target, contain a single foreground target, and the single foreground target image is clear; S2.3, after the manual screening described in step S2.2, the qualified foreground target image blocks and their corresponding mask image blocks are classified and integrated to obtain the target database D object .

3. The target detection method according to claim 1, characterized in that: Step S3 is implemented according to the following steps: S3.

1. Select a background image I from the image training set t and the set L of existing target markers t , t represents the tth image in the image training set; S3.2, determine the background image I to be used together with the background image I in step S3.1 t The target category c and target number n to be reorganized; S3.3, from the target database D in step S2 object Randomly extract a foreground target image i of target category c from the image, and read the RGB image O of the foreground target image i i and foreground object mask M i ; S3.4, RGB image O of the foreground target image i described in step S3.3 i and foreground object mask M i Perform random rotation or flipping, and generate the rotated image trans_O after processing. i and rotated foreground object mask trans_M i ; S3.5, adjust the rotated image trans_O described in step S3.4 i and rotated foreground object mask trans_M i The size of the background image I t Same, generate resize_O images respectively i and size foreground object mask resize_M i ; S3.6, determine the foreground target image i in step S3.3 in the background image I t The center point coordinates (x center ,y center ), the center point coordinate (x center ,y center ) needs to follow two conditions: the target frame of the newly added foreground target image i and the background image I t The IoU (intersection over union) of the original target box in is less than 0.5, and the target box of the newly added foreground target image i and the background image I t The original target boxes do not contain each other; S3.7, resize_0 the image rotated to the size described in step S3.5 i 、Size foreground object mask resize_M i 、Background Image I t and the center point coordinates (x center ,y center ) is input into the Poisson fusion algorithm to perform Poisson fusion to complete the foreground target image i and the background image I t reorganization; S3.8, rotate the image resize_0 according to the size described in step S3.5 i The height h and width w and the center point coordinates (x center ,y center ), calculate the reconstructed foreground target image i in the background image I t The bounding box coordinates (x min ,y min )、(x max ,y max ), the bounding box coordinates (x min ,y min )、(x max ,y max ) together with the target category c constitutes the label l of the foreground object i i ; S3.9, repeat steps S3.3 to S3.8 until a new foreground target image i and label l are added. i The preset target number n is reached; S3.10, output the foreground target image i described in step S3.9 as a new training example, and mark l i Add to the existing set of target markers L t A corresponding tag set is formed in .

4. The target detection method according to claim 3, characterized in that: The Poisson fusion described in step S3.7 is implemented by the following algorithm: Let the function representing the pixels of the fused image within Ω be f, and the function representing the pixels of the fused image outside Ω be f * , the smoothing condition can be expressed as: In formula (4-1), v represents the gradient field of the foreground image to be synthesized; The condition for keeping the boundary consistent can be expressed as: make F=|▽f-v 2 =(▽f x -v x ) 2 +(▽f y -v y ) 2 (4-3) Substituting formula (4-3) into the Euler-Lagrange equation, we have: In formula (4-4), x represents the horizontal coordinate of the pixel point in the image, and y represents the vertical coordinate of the pixel point in the image; F is a function of ▽f, Therefore, it can be obtained In formula (4-5), Δf=div v is the solution of Poisson's equation.

5. The target detection method according to claim 1, characterized in that: Step S4 is implemented according to the following steps: The image training set described in step S1 and the new training examples described in step S3.10 are divided into two paths. One path is resized to 416×416 pixels or 640×640 pixels as the input of the visual enhancement submodule (VE module), wherein the visual enhancement submodule (VE module) is a series connection of three visual enhancement algorithms in the order of contrast adjustment, brightness adjustment and white balance operation; the other path is resized to 256×256 pixels as the input of the parameter prediction submodule (Pe-PP module), and the parameter prediction submodule (Pe-PP module) infers and outputs the visual The configuration parameters of the enhancement submodule (VE module) are output to the visual enhancement submodule (VE module). After receiving the configuration parameters, the visual enhancement submodule (VE module) can perform visual enhancement processing on the input image of 416×416 pixels or 640×640 pixels, and use the processed image as the input of the target detector, calculate the loss of the output of the target detector and the corresponding label set, and update the parameters of the parameter prediction submodule (Pe-PP module) and the target detector with the goal of minimizing the loss to obtain a trained target detection model.

6. The target detection method according to claim 1, characterized in that: Step S5 is implemented according to the following steps: The image to be detected is input into the trained target detection model described in step S4. The target detection model infers the input image to be detected and outputs the target detection result, thereby completing the target detection task.

7. A target detection system based on image data augmentation and image visual enhancement algorithm, applying the target detection method according to any one of claims 1 to 6, wherein the target detection system comprises a target database construction module, a joint training module and a target detection module: Target database construction module: input the image training set into the SAM model, segment the image training set according to the "bounding box prompt" in the SAM model, form and output the foreground target mask after segmentation; accurately crop the foreground target mask and the image training set according to the coordinates of the "bounding box prompt" to obtain mask image blocks and foreground target image blocks of the same size as the "bounding box prompt" respectively, filter and screen the mask image blocks and foreground target image blocks to construct the target database D object ; Joint training module: From the target database D object Randomly select a foreground target image i from the image training set, and perform mask-based recombination on the foreground target image i and the image training set to obtain a new training sample; Using a parameter prediction submodule (Pe-PP module), a visual enhancement submodule (VE module) and a target detector, the new training examples and the image training set are jointly trained to obtain a trained target detection model; Object detection module: uses the trained object detection model to detect the image to be detected and outputs the object detection result.

8. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the target detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the target detection method according to any one of claims 1 to 6.

Images