Multi-modal guide feature enhanced camouflage target detection method

Abstract

The invention relates to the technical field of camouflage target detection, in particular to a multi-modal guide feature enhanced camouflage target detection method, which comprises the following steps of: acquiring an input image through a camera and preprocessing the input image by using an inference perception filtering module; camouflage features in the preprocessed image are extracted through a backbone network and a receptive field enhancement module, the integrity of the camouflage features is further enhanced by fusing feature information such as text information, morphological features and an infrared spectrum of a camouflage target, and intermediate feature information and deep feature information are transmitted into a near decoder to obtain an initial segmentation image; the initial segmentation image, shallow layer information and multi-modal fusion information are spliced layer by layer to further enrich the edge and texture of the segmentation image, and the segmentation image obtains a prediction result through an activation function. According to the invention, the detection precision of the camouflage target is improved, the integrity of the edge texture of the target is enhanced, and convenience is provided for camouflage target detection.

Description

Technical Field

[0001] The present invention relates to the technical field of camouflaged target detection, and in particular to a camouflaged target detection method with multi-modal guidance feature enhancement. Background Art

[0002] Camouflaged Object Detection (COD) is a type of hidden object detection that is opposite to Salient Object Detection (SOD). Its purpose is to detect camouflaged objects that are highly integrated with the surrounding environment.

[0003] The camouflage phenomenon is prevalent in various fields; for example, the localization detection of diseased cells in the biomedical field, the defect detection of transparent objects in the process manufacturing field, the invasion detection of mimic pests in the ecological agriculture field, the detection of picture splicing effects in the art design field, the detection of camouflage facilities in the military industry, and the salience conversion of hidden objects, etc.

[0004] At present, the camouflage detection methods include YOLO target detection, information search and recognition network, edge feature extraction algorithm, attention mechanism fusion, etc.; for example: the publication numbers are CN202311356780.9 and CN202211548176.1; both use YOLO for camouflage target detection and recognition; however, the YOLO model needs to divide the input image into multiple grids, and needs to predict multiple bounding boxes for each grid. When the camouflage target has complex texture features, the YOLO model will destroy the integrity of the texture edge of the camouflage target to a certain extent, and it also takes a long time to train the model.

[0005] Another example is publication numbers CN202311348228.5 and CN202310316598.4; edge-guided feature enhancement and edge information guidance methods are used, but in most cases this method only performs corresponding processing on the edges of the camouflaged target, and largely ignores other information of the camouflaged target.

[0006] In general, the above existing detection technologies are difficult to be comprehensive when faced with more classified and more concealed objects; therefore, there is an urgent need for a more efficient and more accurate detection method to solve the detection and positioning problems of camouflaged targets. Summary of the invention

[0007] In view of the shortcomings of the existing methods, the present invention solves the problem that the existing YOLO model destroys the integrity of the texture edge of the camouflaged target, resulting in reduced recognition accuracy and long time consumption.

[0008] The technical solution adopted by the present invention is: a camouflaged target detection method enhanced by multi-modal guided features comprises the following steps:

[0009] Step 1: Collect camouflaged target images, create training sets and test sets, and pre-process the camouflaged target images;

[0010] As a preferred implementation of the present invention, the camouflaged target image is preprocessed using an IFF module.

[0011] Step 2: Input the preprocessed feature image into the Res2Net50 backbone network, output the feature information of 5 stages, cross-fuse the feature information of 5 stages, and obtain the cross-fused features;

[0012] As a preferred embodiment of the present invention, step 2 specifically includes:

[0013] Step 21: Downsample the feature information of the X1 stage and the X2 stage by 2 times to obtain the feature

[0014] Step 22: The feature information of the X2 stage is downsampled by 2 times and concatenated with the feature information of the X3 stage to obtain the feature

[0015] Step 23: Upsample the feature information of the X3 stage and the X4 stage by 2 times and upsample the feature information of the X5 stage by 4 times to obtain the feature

[0016] Step 24: Upsample the feature information of the X4 stage by 2 times and upsample the feature information of the X5 stage by 4 times to obtain the feature

[0017] Step 25: Upsample the feature information of the X5 stage by 4 times to obtain the feature

[0018] Step 3: Connect the cross-fused features to 5 RFB-s modules to obtain 5 new features;

[0019] As a preferred embodiment of the present invention, step three specifically includes:

[0020] The characteristics Input the first RFB-s module to obtain the feature r1F1; Input the second RFB-s module to obtain the feature r2F2; Input the third RFB-s module to obtain the feature r3F3; Input the fourth RFB-s module to obtain the feature r4F4; Input the fifth RFB-s module to obtain feature r5F5.

[0021] Step 4: Fuse the multimodal features of the camouflaged target’s text information, morphological features, and infrared spectrum, process the multimodal features using text embedding and image embedding, and perform mode conversion on the embedded multimodal features;

[0022] As a preferred embodiment of the present invention, step 4 specifically includes:

[0023] Step 41, converting text information, morphological feature images, and infrared spectrum images into sequences;

[0024] Step 42: Generate a feature vector T from the text information sequence i ;

[0025] Step 43: Generate feature vectors A from the morphological features and infrared spectrum sequence i and B i , and generate the feature matrix I PR =A i T B i ;

[0026] Step 44: Extract matrix I PR The diagonal elements of the matrix I NE ;

[0027] Step 45: Extraction I NE The diagonal elements generate the eigenvector I i ;

[0028] Step 46: Feature vector I i With the eigenvector T i The transpose of T i T Multiply to generate the characteristic matrix I, and the diagonal elements of I constitute the characteristic vector T i I i .

[0029] Step 5: Use the NCD module to fuse the output features of the third to fifth RFB-s modules to obtain an initial camouflage image; concatenate the initial camouflage image with the output features of the second RFB-s module, and then fuse the concatenated feature map with the output feature information of the first RFB-s module. Concatenate the fused feature map with the multimodal feature information, and output the predicted image through the activation function operation to complete the prediction model construction.

[0030] As a preferred implementation of the present invention, step five specifically includes:

[0031] Step 51: The feature information r3F3, r4F4, r5F5 is used to obtain the initial camouflage image P through NCD. q ;

[0032] Step 52: Characteristic information r2F2 and initial camouflage image P q After splicing, upsample 8 times to obtain the feature map P r ;

[0033] Step 53: The feature information r1F1 is upsampled 8 times and the feature map P r After splicing, we get the feature map P t ;

[0034] Step 54: Use the mode converter to convert the feature G T , G S , G IS Fusion obtains feature information G c ;

[0035] Step 55: feature information G c Perform 8 times upsampling and feature map P t After splicing, the predicted image P is obtained by activating the Sigmoid function o .

[0036] As a preferred embodiment of the present invention, a weighted cross entropy loss function is used to train the prediction model.

[0037] As a preferred embodiment of the present invention, the formula of the weighted cross entropy loss function is:

[0038]

[0039] Among them, the true distribution of labels is y i The predicted distribution is represented by p i Represents, W represents weight, and N represents step size.

[0040] As a preferred implementation manner of the present invention, it also includes: performing target positioning and correction on the predicted image.

[0041] As a preferred embodiment of the present invention, target positioning and correction include:

[0042] First, obtain the predicted image P o The complete edge profile f_edge;

[0043] Secondly, the coordinates of the edge contour f_edge are fused into the predicted image P o In the above example, we generate a new image P N ;

[0044] Next, a minimum bounding rectangle is set according to the highest point, lowest point, leftmost point and rightmost point of the pixel coordinates of f_edge; the minimum bounding rectangle is centralized to generate image P on ;

[0045] Next, image P on Split to generate N units P k , for P k Stretch and shrink according to the threshold ratio to generate a new unit P j ;

[0046] Finally, all units P j Reassemble in the order of segmentation to obtain the corrected image P co .

[0047] Beneficial effects of the present invention:

[0048] 1. The present invention uses an inference perception filtering module to effectively solve the mixed spot pollution and noise pollution in the input image, which is beneficial to improving the accuracy of camouflage target detection;

[0049] 2. The present invention can effectively enhance the semantic morphology and edge details of the camouflaged target by splicing shallow feature information, intermediate feature information, and deep feature information layer by layer;

[0050] 3. The present invention can further enhance the camouflaged target features by using multi-modal feature enhancement;

[0051] 4. The present invention can clearly locate the specific position of the camouflaged target in the image by using target positioning; and use target correction to convert the unclear or relatively abstract camouflaged target in the predicted image into a relatively obvious camouflaged target without losing detail information. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram of the camouflaged target detection network structure enhanced by multi-modal guided features of the present invention;

[0053] Figure 2 This is a schematic diagram of the RFB-s module;

[0054] Figure 3 It is a schematic diagram of feature enhancement module;

[0055] Figure 4 It is a schematic diagram of the hardware device structure;

[0056] Figure 5 Schematic diagram of target correction process;

[0057] Figure 6 It is a schematic diagram of the minimum enclosing rectangle;

[0058] Figure 7 Schematic diagram of the comparison between the predicted results and the real image. DETAILED DESCRIPTION

[0059] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.

[0060] like Figure 1 As shown, a camouflaged target detection method enhanced by multi-modal guided features includes the following steps:

[0061] Step 1: Collect image data sets of camouflaged targets and preprocess the images;

[0062] According to the existing 78 categories of COD-10K, 100 jungle camouflage and camouflage net image data are added, and the camouflaged objects in the images are segmented with complete edge features and instance information as training data sets; 50 images of camouflage net decoration are collected as test data sets using the same method to segment the complete edge features and instance information, which is used to test the practical effect of the network model.

[0063] The SAM network is used to perform instance segmentation and labeling on the disguised objects in the image, and the processed images are sorted and numbered by category.

[0064] like Figure 4 As shown in the figure, a 1080P high-definition camera is used to collect image information in a complex environment and store it in the memory of the Jetson Xavier NX hardware device; the deployed network model reads the video file M in the memory, analyzes and processes the collected video file M through the central processing unit, image processor and software program, and decomposes the video file into image files. The image files are sorted according to the number of decomposed frames as the input images entering the reasoning perception filtering module, and x i (i=1,……,N) represents.

[0065] When the camera collects dynamic information, it is inevitable that it will encounter interference from factors such as light spot reflection and target occlusion. To a large extent, this may cause the collected dynamic information to be significantly different from the original scene, and ultimately the unfiltered image will generate a "contaminated" feature map after entering the convolution of the backbone network.

[0066] In the input image x i Before entering the backbone network, the inference-aware feature filtering module (IFF) is used as an information passing module to eliminate noise interference of feature information and explore high-level semantic information, that is, preprocessing.

[0067] The expression of the inference-aware filtering module is as follows:

[0068]

[0069] Among them, xi k+1 is the feature image generated after iterating k times for the i-th frame image; w i is the convolution kernel, ReLU is the activation function, y i k is the wth value after k iterations i The present invention only needs a small number of iterations to avoid problems such as light spots and noise interference in the input image.

[0070] Step 2: Input the preprocessed feature map into the Res2Net50 network, output the feature information of the five stages, cross-fuse the five, and obtain the cross-fused features;

[0071] Preprocessed image x i k+1 Feature extraction is performed through the backbone network, which is Res2Net50, including: 5 stages X1, X2, X3, X4, and X5, which output 5 corresponding feature information F1, F2, F3, F4, and F5 respectively, that is, the 5 stages correspond to Stage1 to Stage5 of Res2Net50; feature information F4 and F5 are combined into deep feature information, and the output semantic information is used to determine the specific position of the target in the image; feature information F1 and F2 are combined into shallow feature information, and very rich edge detail feature information is output, which is used to optimize the edge details of the camouflaged target and improve the detection accuracy; feature information F3 is used for the connection between contexts as a bridge between shallow feature information and deep feature information, and F3 can enhance the smoothness of edge features.

[0072] Extracting feature information is divided into two parts: extracting complete edge features of the target and extracting edge detail features; image x i k+1 After entering module X1, feature information F1 is obtained. Feature information F1 represents obvious image edge features and weakens a large number of irrelevant details. The purpose is to reduce irrelevant information in the image and reduce the amount of computation for subsequent convolutional layer filtering. Feature information F1 enters module X2 after pooling and step size setting, and feature information F2 can be obtained after convolution operation. F1 and F2 are represented as shallow feature information, and shallow feature information retains a large number of edge details and texture features. After feature information F2 enters modules X3, X4, and X5, intermediate feature information F3 and deep feature information F4 and F5 can be obtained. Deep feature information F4 and F5 retain rich semantic feature information, which is of great significance for determining the location of the target. Intermediate feature information F3 is used as a mid-level feature to connect shallow features and deep features, and is used for contextual connection and construction of continuous edge contours and smooth edge features.

[0073] Step 21: According to the different sizes of the shallow and deep outputs, the present invention performs 2-fold downsampling on the feature information of the X1 stage and the feature information of the X2 stage to obtain feature

[0074] Step 22: Downsample the feature information of stage X2 by 2 times and combine it with the feature information of stage X3 to obtain the feature

[0075] Step 23: Upsample the feature information of the X3 stage, the feature information of the X4 stage by 2 times, and upsample the feature information of the X5 stage by 4 times to obtain the feature information.

[0076] Step 24: Upsample the feature information of the X4 stage by 2 times and upsample the feature information of the X5 stage by 4 times to obtain the feature

[0077] Step 25: Upsample the feature information of the X5 stage by 4 times to obtain the feature

[0078] In the present invention, the upsampling operation is implemented using the interpolate method in torch, the downsampling operation is implemented using the MaxPool2d method in torch, and the concatenation operation is implemented using the Cat method in torch; the feature information after sampling and concatenation is represented as F i (x), the formula is:

[0079]

[0080] In the formula, x i (i=1,……,5) represents the mapping of different stages of Res2Net50. In Formula 2, f represents the features after concatenation of features at different stages.

[0081] Compared with the prior art, the present invention adopts a method of gradually splicing feature information of different stages, which can not only segment the semantic form of the disguised target but also further enrich the edge feature information of the disguised target on this basis; as shown in formula (2), the deepest feature information is represented by F5(x), at this time the resolution of the feature image is the smallest, the semantic form is the most obvious, and the semantic distribution of the disguised target can be roughly observed in the feature map; F4(x) splices the feature information output by stages X4 and X5, at this time the semantic form of F4(x) is relatively obvious compared to F5(x), but the edge information is strengthened; F3(x) splices the feature information output by stages X3, X4, and X5, the edge details of the feature image of F3(x) will appear clearer than that of F4(x), and the coherence of the context connection is also better; F2(x) and F1(x) further enrich the edge information and detail texture of the feature image; these splicing operations are helpful to improve the integrity and accuracy of target measurement to a great extent.

[0082] Step 3: Use 5 enhanced receptive field blocks (RFB-s) as the target retrieval modules, i.e., RFB-s1 to RFB-s5 are the first to fifth RFB-s modules respectively;

[0083] The characteristics Input the first RFB-s module to obtain the feature r1F1; Input the second RFB-s module to obtain the feature r2F2; Input the third RFB-s module to obtain the feature r3F3; Input the fourth RFB-s module to obtain the feature r4F4; Input the fifth RFB-s module to obtain feature r5F5.

[0084] RFB-s combines the advantages of the Inception module and the ASPP module, uses convolution kernels of different step lengths to simulate the size and eccentricity changes of the receptive field in human vision, and makes the receptive field more diverse and refined while keeping the number of parameters unchanged, which significantly improves the accuracy of the model and the ability to recognize tiny details and their contexts; RFB-s is composed of a convolution array constructed with convolution kernels of different sizes, which can increase the receptive field while improving detection speed and accuracy; Figure 2As shown in the figure, the RFB-s module constructs multi-way branches between different convolution kernels to strengthen the connection between the convolution kernels. Three 1×1 convolutions are used to process the input information close to the input segment, and the 1×1, 1×3, 3×1 and 3×3 convolutions and different rates in the middle are used to increase the step size between the convolution kernels and enlarge the receptive area. In the RFB-s structure, the outputs of convolution layers of different sizes and rates are combined and connected to achieve the purpose of fusing different features. Finally, the fused features are compared with the human visual receptive field.

[0085] The present invention uses RFB-s to increase the visual perception area, and can learn more depth information and detail features; the feature information F i (x) Connecting to the enhanced receptive field module can obtain better visual effects by increasing the receptive field and can achieve the accuracy of target acceleration detection and positioning; the present invention loads F in the RFB-s class i (x) parameter method to establish F i (x) The connection between RFB-s; RFB-s uses multi-way branch pooling. Each branch uses convolution kernels of different sizes to correspond to the receptive fields of different regions. The dilated convolution control simulates the eccentricity of the human retina and reshapes the model to generate the final feature information. RFB-s can achieve good results while taking into account speed.

[0086] Compared with the prior art, the present invention converts the feature information F i (x) New feature information r1F1, r2F2, r3F3, r4F4, r5F5 generated after connecting to RFB-s, r i (i=1,……,5), F i The feature information after sampling and splicing is expressed as F i (x); This can not only capture feature information at close range, but also capture feature information at long distances, which can further improve model performance and enhance the ability to capture detailed features.

[0087] Step 4: The present invention integrates multiple modal feature information to achieve feature enhancement as a feature enhancement module to improve the accuracy of camouflaged target detection; Figure 3 As shown, the present invention integrates three modal features of the camouflaged target, namely, text information, morphological features, and infrared spectrum, wherein the modality of text information is used to describe the details of the camouflaged features, the modality of morphological features is used to express various postures and shapes of the target, and the features of the infrared spectrum are used to display the precise instance features of the target in various forms; the feature information F is enriched and strengthened by the detailed information expressed through multiple modalities.

[0088] The present invention uses feature embedding to obtain feature information of text information, morphological features and infrared spectrum; text information uses a textual embedding module (Textual Embed) to enter a morphological transformation module (Modality Interaction), and shape features and infrared spectrum use a visual embedding module (Visual Embed) to enter the morphological transformation module.

[0089] Step 41: Text and images are converted into sequences using the ViLT module (Vision-and-Language Transformer, ViLT), and the sequences are represented by the matrix D L×W Indicates; the token data generated by text and image are integrated to obtain a whole sequence. The length of the input sequence is 1+L+1+W=2+L+W. Therefore, the entire input is represented by the function F input It is expressed as:

[0090] F input (L, W, H) = (2 + L + W) × H (3)

[0091] Among them, L represents the rows of the matrix, W represents the columns of the matrix, and H represents the input amount.

[0092] Step 42: The text information is converted into a feature vector T by a text encoder. i (i=1,……,N), the elements of this vector constitute the set of the entire text information, where the element T i Represents characteristic information of a camouflaged target;

[0093] Step 43: Similarly, use the image encoder to generate a feature vector A from the morphological information and the infrared spectrum information. i (i=1,……,N) and B i (i=1,……,N);

[0094] Extract A i and B i The vector generates the feature matrix I according to the camouflaged target feature information PR , feature matrix I PR The corner line elements are valid edge feature information.

[0095] Eigenvector A i (i=1,……,N) and B i (i=1,……,N) Generate feature matrix I PR The formula is:

[0096] I PR =A i T Bi (4)

[0097]

[0098] Among them, the eigenvector A i T is the feature vector A i (i=1,……,N)transpose.

[0099] Step 44: Extract the matrix I in formula (6) PR The diagonal elements of generate a new matrix I NE The formula is:

[0100]

[0101] Matrix I NE Feature information generated by multimodal fusion of text, shape features and infrared image features.

[0102] Step 45: Extraction I NE The diagonal elements generate the new eigenvector I i (i=1,……,N);

[0103] Step 46: Feature vector I i (i=1,……,N) and the eigenvector T i Transpose T of (i=1,……,N) i T Multiply to generate the feature matrix I, which is the output feature matrix of the multi-modal; among them, the new feature vector T composed of the diagonal elements i I i (i=1,……,N) represents the edge feature information closest to the camouflaged target, Figure 3 The “×” in the figure indicates weakened feature information or irrelevant feature information, which cannot enhance edge feature information and may even generate interference feature information.

[0104] The similarity and matching degree between text and image are calculated by using Image Text Matching (ITM) and Word Patch Alignment (WPA), retaining the internal highly matched feature information and eliminating discontinuous and discrete feature information. The matched feature data is output using the function G sum express:

[0105] Among them, the text feature information is represented by G T , the shape feature information is represented by G S , the infrared spectrum characteristic information is represented by G IS ; G sum Represented as the feature vector Ti I i (i=1,……,N) mapping; similarly G T , G S and G IS Represented as text feature vector T i The mapping of G S and G IS Represented as the morphological feature vector IA i- and infrared feature vector IB i- 's mapping.

[0106] Step 5, the present invention uses a near component decoder (NCD) to fuse deep features and intermediate feature information to design a target recognition module;

[0107] Step 51: The feature information r3F3, r4F4, r5F5 is used to obtain the initial camouflage image P through NCD. q ;

[0108] Step 52: Characteristic information r2F2 and initial camouflage image P q After splicing, upsample 8 times to obtain the feature map P r ;

[0109] Step 53: The feature information r1F1 is upsampled 8 times and the feature map P r After splicing, we get the feature map P t ;

[0110] Step 54, G T , G S , G IS The feature information G is obtained through the mode converter c ;

[0111] Step 55: Feature information G c Perform 8 times upsampling and feature map P t After splicing, the predicted image P is obtained by activating the Sigmoid function o .

[0112] Steps 2 to 5 construct a network model for camouflaged target detection. The present invention uses weighted cross entropy to construct a loss function. The loss function is expressed as L WCE Indicates that the true distribution of the label is y i The model’s predictive distribution is represented by p i Indicates that W represents the weight, N represents the step size, and the formula of the weighted cross entropy loss function is:

[0113]

[0114] Using the weighted cross entropy method to construct the loss function can directly and clearly calculate the training loss value of the model training process, and can also accelerate the convergence of the training model, providing convenience for subsequent adjustment of hyperparameters such as training rounds and learning rate.

[0115] Target positioning and target correction are because when the camouflaged target is captured by the camera, the target may be tilted. Therefore, the camouflaged target needs to be positioned and corrected so that the camouflaged target is in the center of the image for easy observation and comparison. That is, when the camouflaged target is output through the display screen, the output prediction image P o And the predicted corrected image P co Can be used for visual comparison.

[0116] like Figure 5 As shown, the present invention has the function of target positioning and correction, which is implemented based on Python, and locates and corrects the camouflaged target, including:

[0117] First, obtain the predicted image P o The complete edge contour f_edge of the image P can be used for target positioning and target correction using the existing Canny, Sobel and Prewitt edge detection algorithms. o After processing, the complete edge contour f_edge is obtained; Figure 6 It is the complete edge outline of a fish;

[0118] Secondly, the coordinates of the edge contour f_edge are fused into the predicted image P o In the above example, we generate a new image P N , P N Represented as the target positioning image, through P N Display the specific location of the camouflaged target;

[0119] Secondly, the minimum bounding rectangle is set according to the highest point, lowest point, leftmost point and rightmost point of the pixel coordinates of f_edge, where the highest point, lowest point, leftmost point and rightmost point correspond to the four directions of the pixel coordinates, respectively; the minimum bounding rectangle is centralized to generate the image P on ; That is, image P on The camouflaged target is placed in the predicted image P o The original position in is transferred to the center of the image;

[0120] Next, image P on The inconspicuous or abstract camouflage targets in the image are regarded as scenes similar to the camouflage targets surrounding the cylinder. The image P is segmented by the cylindrical surface. on Split to generate N units P k (k=1,……,N), all the units P after segmentationk Stretch and shrink according to the threshold ratio to generate a new unit P j (j=1,...,N), the threshold ratio in this embodiment is 50%; where P on The segmentation to generate N units usually adopts vertical segmentation. This is because the aspect ratio of the image is usually 3:2, and the vertical segmentation method is better than the horizontal segmentation method;

[0121] Finally, all units P j (j=1,……,N) are reassembled on the same plane in the order of segmentation to generate the predicted corrected image P co ;

[0122] Because P o , P on , Unit P k and P co They are all binary images; therefore, compared with the prior art, the present invention can save a lot of redundant operations, and the experimental steps are less than other target positioning and target correction methods.

[0123] Jetson XavierNX calls the display function, through which the display screen will display the input image x i , predicted image P o , the predicted corrected image P co Comparison chart and camouflage target classification information.

[0124] Positioning is to determine the specific position of the camouflaged object in the image, and correction is to restore the unclear or tilted object in the form of a flat image. Compared with the prior art, the target positioning and target correction methods of the present invention can reduce the amount of processor calculation when performing image processing on hardware devices, thereby improving the visualization effect of the predicted image.

[0125] Load the enriched COD-10K dataset to train the model and test the model, and modify the parameters to debug the optimal training results.

[0126] After several rounds of training, the model with the best training effect is selected for test training. According to the test results, the different types of activation functions are selected, and the number of iterations and hyperparameters are modified to optimize the network model.

[0127] The optimized network model is trained again and the best effect model is selected for testing. The best network model is selected from several rounds of training for deployment.

[0128] Assemble the Jetson XavierNX hardware device and connect it to the computer. Follow the hardware manual to complete the system programming and wait for verification to complete.

[0129] The present invention uses the python library according to the model design, and selects the corresponding versions such as CUDA and TensorRT; the present invention uses torch.onnx to export the ONNX model and upload it to the hardware device, uses trtexec to convert the ONNX model into a TensorRT model, and finally implements model prediction through C++TensorRT.

[0130] like Figure 7 , which is a schematic diagram comparing the prediction results of the network model of the present invention and the real image.

[0131] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A camouflaged target detection method enhanced by multi-modal guided features, characterized in that: The following steps are involved: Step 1: Collect camouflaged target images, create training sets and test sets, and pre-process the camouflaged target images; Step 2: Input the preprocessed feature image into the Res2Net50 backbone network, output the feature information of 5 stages, cross-fuse the feature information of 5 stages, and obtain the cross-fused features; Step 3: Connect the cross-fused features to 5 RFB-s modules to obtain 5 new features; Step 4: Fuse the multimodal features of the camouflaged target’s text information, morphological features, and infrared spectrum, process the multimodal features using text embedding and image embedding, and perform mode conversion on the embedded multimodal features; Step 5: Use the NCD module to fuse the output features of the third to fifth RFB-s modules to obtain an initial camouflage image; concatenate the initial camouflage image with the output features of the second RFB-s module, and then fuse the concatenated feature map with the output feature information of the first RFB-s module. Concatenate the fused feature map with the multimodal feature information, and output the predicted image through the activation function operation to complete the prediction model construction.

2. The method for detecting camouflaged targets enhanced by multi-modal guided features according to claim 1, characterized in that: Step 2 specifically includes: Step 21: Downsample the feature information of the X1 stage and the X2 stage by 2 times to obtain the feature Step 22: Downsample the feature information of stage X2 by 2 times and concatenate it with the feature information of stage X3 to obtain the feature Step 23: Upsample the feature information of the X3 stage and the X4 stage by 2 times and upsample the feature information of the X5 stage by 4 times to obtain the feature Step 24: Upsample the feature information of the X4 stage by 2 times and upsample the feature information of the X5 stage by 4 times to obtain the feature Step 25: Upsample the feature information of the X5 stage by 4 times to obtain the feature 3. The method for detecting camouflaged targets enhanced by multi-modal guided features according to claim 2, characterized in that: Step three specifically includes: The characteristics Input the first RFB-s module to obtain the feature r1F1; Input the second RFB-s module to obtain the feature r2F2; Input the third RFB-s module to obtain the feature r3F3; Input the fourth RFB-s module to obtain the feature r4F4; Input the fifth RFB-s module to obtain feature r5F5.

4. The method for detecting camouflaged targets enhanced by multi-modal guided features according to claim 1, characterized in that: Step 4 specifically includes: Step 41, converting text information, morphological feature images and infrared spectrum images into sequences; Step 42: Generate a feature vector T from the text information sequence i ; Step 43: Generate feature vectors A from the morphological features and infrared spectrum sequence i and B i , generate the feature matrix I PR =A i T B i ; Step 44: Extract matrix I PR The diagonal elements of the matrix I NE ; Step 45: Extraction I NE The diagonal elements generate the eigenvector I i ; Step 46: Feature vector I i With the eigenvector T i The transpose of T i T Multiply to generate the characteristic matrix I, and the diagonal elements of I constitute the characteristic vector T i I i .

5. The method for detecting camouflaged targets enhanced by multi-modal guided features according to claims 3 and 4, characterized in that: Step 5 specifically includes: Step 51: The feature information r3F3, r4F4, r5F5 is used to obtain the initial segmentation map P through NCD. q ; Step 52: feature information r2F2 and initial segmentation map P q After splicing, upsample 8 times to obtain the feature map P r ; Step 53: The feature information r1F1 is upsampled 8 times and the feature map P r After splicing, we get the feature map P t ; Step 54: Use the mode converter to convert the feature G T , G S , G IS Fusion obtains feature information G c ; Step 55: feature information G c Perform 8 times upsampling and feature map P t After splicing, the predicted image P is obtained by activating the Sigmoid function o .

6. The method for detecting camouflaged targets enhanced by multi-modal guided features according to claim 5, characterized in that: The prediction model is trained using the weighted cross entropy loss function.

7. The method for detecting camouflaged targets enhanced by multi-modal guided features according to claim 6, characterized in that: The formula for the weighted cross entropy loss function is: Among them, the true distribution of labels is y i The predicted distribution is represented by p i Represents, W represents weight, and N represents step size.

8. The method for detecting camouflaged targets enhanced by multi-modal guided features according to claim 5, characterized in that: Also includes: Perform target localization and correction on the predicted image.

9. The method for detecting camouflaged targets enhanced by multi-modal guided features according to claim 8, characterized in that: Targeting and correction include: First, obtain the predicted image P o The complete edge profile f_edge; Secondly, the coordinates of the edge contour f_edge are fused into the predicted image P o In the above example, we generate a new image P N ; Next, a minimum bounding rectangle is set according to the highest point, lowest point, leftmost point and rightmost point of the pixel coordinates of f_edge; the minimum bounding rectangle is centralized to generate image P on ; Next, image P on Split to generate N units P k , for P k Stretch and shrink according to the threshold ratio to generate a new unit P j ; Finally, all units P j Reassemble in the order of segmentation to obtain the corrected image P co .

10. The method for detecting camouflaged targets enhanced by multi-modal guided features according to claim 1, characterized in that: The IFF module is used to preprocess the camouflaged target image.

Images