A method and device for training a joint detection network of camouflage targets and salient targets

Abstract

The application discloses a combined detection network training method and device for camouflage targets and salient targets and a storage medium. The method comprises the following steps: acquiring a first data set, including a plurality of salient image pairs; acquiring a second data set, including a plurality of camouflage image pairs; pre-processing a plurality of salient image pairs sampled from the first data set, each salient image pair including a first image and a real salient feature image; pre-processing a plurality of camouflage image pairs sampled from the second data set, each camouflage image pair including a second image and a real camouflage feature image; processing the pre-processed plurality of first images and the plurality of second images by using a combined detection network to obtain a plurality of predicted salient feature images and a plurality of predicted camouflage feature images; combining the pre-processed plurality of real salient feature images and the plurality of real camouflage feature images to determine a total loss value and optimize network parameters. The application can simultaneously realize effective detection of camouflage targets and salient targets.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a joint detection network training method of camouflage target and salient target, equipment and storage medium. BACKGROUND

[0002] It is a common technical means to apply a neural network to a camouflage target detection task and a salient target detection task respectively, but since the camouflage target detection task and the salient target detection task are in conflict, if the data of the two tasks are used to train the neural network at the same time without any improvement of the neural network to adapt to the joint detection task, the detection performance of the neural network on the two tasks will be reduced. SUMMARY

[0003] The present application provides a joint detection network training method of camouflage target and salient target, equipment and storage medium to solve one or more technical problems existing in the prior art, at least to provide a beneficial choice or create conditions.

[0004] In a first aspect, a joint detection network training method of camouflage target and salient target is provided, and the method comprises:

[0005] Obtaining a first data set and a second data set, the first data set comprising a plurality of salient image pairs, and the second data set comprising a plurality of camouflage image pairs;

[0006] Sampling a plurality of salient image pairs from the first data set, each salient image pair comprising a first image and a corresponding real salient feature image, and pre-processing the plurality of salient image pairs;

[0007] Sampling a plurality of camouflage image pairs from the second data set, each camouflage image pair comprising a second image and a corresponding real camouflage feature image, and pre-processing the plurality of camouflage image pairs;

[0008] Obtaining a joint detection network and performing feature extraction on the pre-processed plurality of first images and plurality of second images to obtain a plurality of predicted salient feature images and a plurality of predicted camouflage feature images;

[0009] Determining a total loss value according to the plurality of predicted salient feature images, the plurality of predicted camouflage feature images, and the pre-processed plurality of real salient feature images and plurality of real camouflage feature images;

[0010] According to the total loss value, performing parameter optimization on the joint detection network using a backpropagation algorithm.

[0011] Further, the joint detection network comprises an encoder, a linear layer and a decoder connected in sequence.

[0012] The encoder comprises a plurality of original transformers connected in sequence, each original transformer comprising a plurality of multi-head attention layers connected in sequence, a first residual connection layer, a multi-layer perception, and a second residual connection layer;

[0013] The decoder comprises a plurality of improved transformers connected in sequence, each improved transformer being obtained by adding a first distribution learning module for cooperating to perform a salient feature detection task and a second distribution learning module for cooperating to perform a camouflage feature detection task at the output end of the original transformer;

[0014] When the original transformer outputs a feature sequence, the first distribution learning module processes the feature sequence after receiving it through the following expression:

[0015]

[0016] The second distribution learning module processes the feature sequence after receiving it through the following expression:

[0017]

[0018] wherein X1 ′ is the output result of the first distribution learning module, X is the feature sequence, μ s is a first learnable mean, σ s is a first learnable variance, X2 ′ is the output result of the second distribution learning module, μ c is a second learnable mean, μ c is a second learnable variance.

[0019] Further, the obtaining the joint detection network and performing feature extraction on the plurality of first images and the plurality of second images after preprocessing to obtain a plurality of predicted salient feature images and a plurality of predicted camouflage feature images comprises:

[0020] According to the plurality of first images after preprocessing, a plurality of first token sequences corresponding thereto are generated; the plurality of first token sequences are input into the joint detection network for feature extraction, while all second distribution learning modules are disabled, to obtain a plurality of second token sequences corresponding thereto; according to the plurality of second token sequences, a plurality of first detection images corresponding thereto are generated; and the plurality of first detection images are subjected to convolution operation to obtain a plurality of predicted salient feature images corresponding thereto;

[0021] According to the plurality of second images after preprocessing, a plurality of third token sequences corresponding are generated; the plurality of third token sequences are input into the joint detection network for feature extraction, while all first distribution learning modules are disabled, to obtain a plurality of fourth token sequences corresponding; according to the plurality of fourth token sequences, a plurality of second detection images corresponding are generated; the plurality of second detection images are subjected to convolution operation to obtain a plurality of predicted camouflage feature images corresponding.

[0022] Further, the first data set contains a number of significant image pairs equal to the number of camouflage image pairs contained in the second data set, and the number of camouflage image pairs contained in the second data set is denoted as M, and the first data set is obtained by the following way:

[0023] An original data set is obtained, which includes N original significant image pairs and N>M, and each original significant image pair includes an original image and a corresponding original significant feature image;

[0024] The N original images are processed by using the trained significant target detection network to obtain N first predicted significant feature images corresponding;

[0025] The minimum mean square error between each first predicted significant feature image and the associated original significant feature image is determined, and M minimum mean square errors with the smallest values are selected from N minimum mean square errors, and M original images and M original significant feature images associated with the M minimum mean square errors are taken as the first data set.

[0026] Further, the preprocessing of the plurality of significant image pairs includes synchronously scaling and data augmentation on the first image and the real significant feature image contained in each significant image pair.

[0027] Further, the preprocessing of the plurality of camouflage image pairs includes synchronously scaling and data augmentation on the second image and the real camouflage feature image contained in each camouflage image pair.

[0028] Further, the determination of the total loss value according to the plurality of predicted significant feature images, the plurality of predicted camouflage feature images, and the plurality of real significant feature images and the plurality of real camouflage feature images after preprocessing includes:

[0029] A structure loss function is used to determine a first loss value between the plurality of predicted significant feature images and the plurality of real significant feature images after preprocessing, and a second loss value between the plurality of predicted camouflage feature images and the plurality of real camouflage feature images after preprocessing;

[0030] The first loss value and the second loss value are added to obtain the total loss value.

[0031] Further, the method further comprises:

[0032] Obtaining a to-be-detected image containing a camouflage target and a salient target;

[0033] Performing feature extraction on the to-be-detected image by using the trained joint detection network to obtain a corresponding camouflage feature image and a salient feature image.

[0034] In a second aspect, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the joint detection network training method for a camouflage target and a salient target according to the first aspect.

[0035] In a third aspect, a computer readable storage medium is provided, storing a computer program, and the computer program is executed by a processor to implement the joint detection network training method for a camouflage target and a salient target according to the first aspect.

[0036] The present application has at least the following beneficial effects: by introducing two independent distribution learning models in the decoder to learn the characteristics of different target detection tasks, the joint detection network can effectively and reliably detect camouflage features and salient features in the same input image; by limiting the number of salient image pairs contained in the first data set to be equal to the number of camouflage image pairs contained in the second data set, the joint detection network is prevented from being dominated by the task with larger data size during training, i.e., the performance of the joint detection network on the task with smaller data size is improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings are included to provide a further understanding of the technical scheme of the present application, and constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical scheme of the present application, and do not constitute a limitation on the technical scheme of the present application.

[0038] Figure 1 is a flowchart of a joint detection network training method for a camouflage target and a salient target in an embodiment of the present application;

[0039] Figure 2 is a composition diagram of a joint detection network in an embodiment of the present application;

[0040] Figure 3 is a composition diagram of an original transformer in an embodiment of the present application;

[0041] Figure 4 is a composition diagram of an improved transformer in an embodiment of the present application;

[0042] Figure 5is the application effect schematic diagram of the joint detection network in the embodiment of the present application;

[0043] Figure 6 is the hardware structure schematic diagram of the computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0045] It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0047] In addition, the described features, structures or characteristics can be combined in any suitable way in one or more embodiments. In the following description, many specific details are provided to give a sufficient understanding of the embodiments of the present application. However, one skilled in the art will realize that the technical solutions of the present application can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be used. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid obscuring the aspects of the present application.

[0048] The flowchart shown in the drawings is only an exemplary description, not necessarily including all contents and operations / steps, and not necessarily executed in the order described. For example, some operations / steps can be further divided, and some operations / steps can be combined or partially combined, so the actual execution order may be changed according to the actual situation.

[0049] First, some of the nouns involved in the present application are explained as follows:

[0050] Camouflage target detection aims to identify those target regions in the image that are integrated with the background, which are difficult to distinguish due to high similarity in color, texture or lighting conditions with the background.

[0051] Salient object detection, which aims to identify those target regions in an image that can attract the most attention of the human eye, which are easily distinguished from the background due to color contrast, edge clarity or special texture features, and is completely opposite to camouflage target detection.

[0052] Mean Squared Error (MSE), a commonly used indicator to measure the similarity or difference between two images, is used to calculate the average of the square of the difference between the corresponding pixel values of the two images.

[0053] Structural Similarity Loss Function (SSIM), an indicator to measure the quality of an image, considers the brightness, contrast and structural information of the image, and can better capture the structural differences between the predicted image and the actual image.

[0054] Backpropagation algorithm is an algorithm for training artificial neural networks, based on gradient descent method and updating the weights by calculating the partial derivative of the loss function on the network weights, so as to optimize the performance of the network.

[0055] Please refer to Figure 1 , Figure 1 is a flowchart of a joint detection network training method for camouflage targets and salient targets provided by an embodiment of the application, and the method comprises the following steps:

[0056] Step S110, acquiring a first data set and a second data set;

[0057] The first data set comprises a plurality of salient image pairs, and the second data set comprises a plurality of camouflage image pairs.

[0058] Step S120, sampling a plurality of salient image pairs from the first data set, each salient image pair comprising a first image and a corresponding real salient feature image, and pre-processing the plurality of salient image pairs;

[0059] Step S130, sampling a plurality of camouflage image pairs from the second data set, each camouflage image pair comprising a second image and a corresponding real camouflage feature image, and pre-processing the plurality of camouflage image pairs;

[0060] Step S140, acquiring a joint detection network and performing feature extraction on the pre-processed plurality of first images and plurality of second images to obtain a plurality of predicted salient feature images and a plurality of predicted camouflage feature images;

[0061] Step S150, determining a total loss value according to the plurality of predicted salient feature images, the plurality of predicted camouflage feature images, and the plurality of preprocessed real salient feature images and the plurality of real camouflage feature images;

[0062] Step S160, performing parameter optimization on the joint detection network by using a back propagation algorithm according to the total loss value.

[0063] In some embodiments of the present application, the following is described for the acquisition method of the first data set mentioned in step S110:

[0064] First, an original data set is acquired, which includes N original salient image pairs, each original salient image pair including an original image and an original salient feature image associated with the original image, the number of camouflage image pairs included in the second data set is denoted as M, and N > M is defined, N and M are both positive integers greater than zero; as a preferred embodiment, 10556 original salient image pairs and 4040 camouflage image pairs are applied in the present application in total;

[0065] Second, the N original images included in the N original salient images are input into a pre-trained salient object detection network for feature extraction, so as to acquire N first predicted salient feature images associated with the N original images; as a preferred embodiment, the salient object detection network is set to be a SENet network, which is recorded in an English document entitled "A Simple yet Effective Network based on Vision Transformer for Camouflaged Object and Salient Object Detection", and is obtained by training using 10556 original salient image pairs;

[0066] Then, for each first predicted salient feature image and original salient feature image associated with each original image in the N original images, the minimum mean square error between the first predicted salient feature image and the original salient feature image is calculated, in this way, N minimum mean square errors associated with the N original images can be calculated;

[0067] Finally, the N minimum mean square errors are arranged in order from small to large, and then the first M minimum mean square errors are extracted from the front, and the M original salient feature images and M original images associated with the M minimum mean square errors are taken as the first data set, wherein the original salient feature image is a real salient feature image, and the original image is a first image, so that the number of salient image pairs included in the first data set is the same as the number of camouflage image pairs included in the second data set.

[0068] Through the above acquisition mode, data balance between the training data set required for performing the salient feature detection task and the training data set required for performing the camouflage feature detection task can be ensured, so as to avoid the joint detection network being dominated by the task with larger data scale in the training process, that is, to avoid the performance of the joint detection network on the task with smaller data scale being poor, and to shorten the network training time and improve the network training efficiency; in addition, the present application does not directly use the random sampling mode, and can ensure that each real salient feature image contained in the first data set can express the salient feature in the corresponding first image in high quality, so that the joint detection network can have higher salient feature extraction capability after training.

[0069] In some embodiments of the present application, the preprocessing mode mentioned in the above step S120 includes scaling and data augmentation, which is specifically as follows:

[0070] For each salient image pair obtained by sampling, the real salient feature image and the first image contained in the salient image pair are simultaneously scaled to a pre-set first size, and then the scaled real salient feature image and the scaled first image are simultaneously randomly horizontally flipped, and finally the flipped real salient feature image and the flipped first image are simultaneously randomly cropped to a pre-set second size, so as to obtain the preprocessed real salient feature image and the preprocessed first image; as a preferred embodiment, the present application sets the batch size randomly sampled from the first data set to be 16, the first size to be 400*400, and the second size to be 384*384.

[0071] It should be noted that by synchronously operating the first image and the real salient feature image, the pixels between the two images can be kept aligned, so as to ensure that the preprocessing operation does not affect the training effect of the joint detection network when performing the salient feature detection task.

[0072] In some embodiments of the present application, the preprocessing mode mentioned in the above step S130 also includes scaling and data augmentation, which is specifically as follows:

[0073] For each camouflage image pair sampled, the real camouflage feature image and the second image contained in the camouflage image pair are simultaneously scaled to a preset third size, then the scaled real camouflage feature image and the scaled second image are simultaneously randomly horizontally flipped, and finally the flipped real camouflage feature image and the flipped second image are simultaneously randomly cropped to a preset fourth size, so as to obtain a pretreated real camouflage feature image and a pretreated second image; as a preferred embodiment, the batch size randomly sampled from the second data set is also 16, the third size is also 400*400, and the fourth size is also 384*384.

[0074] It should be noted that, by synchronously operating the second image and the real camouflage feature image, the pixels between the two images can be ensured to be aligned, so as to ensure that the pretreatment operation does not affect the training effect of the joint detection network when performing the camouflage feature detection task.

[0075] In some embodiments of the present application, the joint detection network mentioned in the above step S140 comprises an encoder, a linear layer and a decoder connected in sequence, as shown in Figure 2 The encoder comprises a plurality of original transformers connected in sequence, and the decoder comprises a plurality of improved transformers connected in sequence; as an optional embodiment, the present application sets the number of original transformers to 12 and the number of improved transformers to 8, and sets the output dimension of the encoder to 768 and the output dimension of the decoder to 512, at this time the linear layer mainly plays a role of data dimension reduction.

[0076] More specifically, each original transformer (i.e. Transformer block) comprises a multi-head attention layer, a first residual connection layer, a multi-layer perceptron and a second residual connection layer connected in sequence, as shown in Figure 3

[0077] ​In practical applications, when the original transformer receives an input feature sequence, the input feature sequence is subjected to attention mechanism operation in the multi-head attention layer to obtain a first feature sequence; in the first residual connection layer, the first feature sequence is subjected to normalization processing, and then the normalized first feature sequence is added to the input feature sequence to obtain a second feature sequence; in the multi-layer perception, a first linear layer, a second linear layer and a GELU (Gaussian Error Linear Unit) activation function are arranged, the first linear layer can increase the sequence length, the second linear layer can shorten the sequence length to the original state, and the second feature sequence is sequentially subjected to processing by the first linear layer, the GELU activation function and the second linear layer to obtain a third feature sequence; in the second residual connection layer, the third feature sequence is subjected to normalization processing, and then the normalized third feature sequence is added to the second feature sequence and output; it should be noted that the first residual connection layer and the second residual connection layer can improve the stability of network training.

[0078] More specifically, each improved transformer is obtained by adding a first distribution learning module and a second distribution learning module at the output end of the original transformer, as shown in Figure 4 , wherein the first distribution learning module is used to assist in performing a salient feature detection task, the second distribution learning module is used to assist in performing a camouflage feature detection task, and the first distribution learning module and the second distribution learning module have a sequence of execution in actual use;

[0079] In practical applications, when the original transformer outputs a feature sequence, if the first distribution learning module is selected to assist in performing a salient feature detection task, the first distribution learning module processes the received feature sequence using the following expression:

[0080]

[0081] In the expression, X'1 is the output result of the first distribution learning module processing the feature sequence, X is the feature sequence, μ s is a first learnable mean, σ s is a first learnable variance.

[0082] If the second distribution learning module is selected to assist in performing a camouflage feature detection task, the second distribution learning module processes the received feature sequence using the following expression:

[0083]

[0084] In the formula, X'2 is an output result obtained by processing the feature sequence by the second distribution learning module, X is the feature sequence, μ c is a second learnable mean, and μ c is a second learnable variance.

[0085] In some embodiments of the present application, the implementation process of the above step S140 includes but is not limited to the following:

[0086] Step S141, based on the plurality of preprocessed first images, a plurality of first token sequences are generated;

[0087] More specifically, after each preprocessed first image is sequentially subjected to patch segmentation, patch flattening and position encoding, a corresponding first token sequence is formed;

[0088] Step S142, the plurality of first token sequences are input to the joint detection network for feature extraction, in which process all second distribution learning modules are not used (i.e., only all first distribution learning modules are used to assist in performing the salient feature detection task) to obtain a plurality of corresponding second token sequences;

[0089] Step S143, based on the plurality of second token sequences, a plurality of first detection images are generated;

[0090] More specifically, after each second token sequence is sequentially subjected to dimension expansion, patch reconstruction and patch splicing, a corresponding first detection image is formed;

[0091] Step S144, convolution operation is performed on the plurality of first detection images to obtain a plurality of predicted salient feature images associated with the plurality of preprocessed first images;

[0092] Step S145, based on the plurality of preprocessed second images, a plurality of third token sequences are generated;

[0093] More specifically, after each preprocessed second image is sequentially subjected to patch segmentation, patch flattening and position encoding, a corresponding third token sequence is formed;

[0094] Step S146, the plurality of third token sequences are input to the joint detection network for feature extraction, in which process all first distribution learning modules are not used (i.e., only all second distribution learning modules are used to assist in performing the camouflage feature detection task) to obtain a plurality of corresponding fourth token sequences;

[0095] Step S147, based on the plurality of fourth token sequences, a plurality of second detection images are generated;

[0096] More specifically, each fourth token sequence is sequentially subjected to dimension expansion, patch reconstruction and patch splicing to form a corresponding second detection image.

[0097] In step S148, a convolution operation is performed on the plurality of second detection images to obtain a plurality of predicted camouflage feature images associated with the plurality of preprocessed second images.

[0098] As an optional implementation, the manner of generating a token sequence from an image mentioned in the above step S141 and the above step S145 is described as follows:

[0099] In the present application, the size of the image is set to 384*384, the image is evenly divided to obtain 24*24 two-dimensional patches (patch), each two-dimensional patch (patch) has a size of 16*16, each two-dimensional patch (patch) is stretched and flattened into a one-dimensional token, then position encoding is performed to determine the position of the one-dimensional token in the image, the length of the one-dimensional token is 768, and finally 576 one-dimensional tokens converted from the image form a token sequence output.

[0100] As an optional implementation, the manner of generating a token sequence from an image mentioned in the above step S141 and the above step S145 is described as follows:

[0101] Since the output dimension of the decoder is set to 512 in the present application, it is explained that the length of each one-dimensional token contained in the token sequence is 512, at this time, the linear layer mapping manner is adopted to expand the dimension of the token sequence to obtain an intermediate token sequence, so that the length of each one-dimensional token contained in the intermediate token sequence is 768, then the intermediate token sequence is converted into 576 two-dimensional patches (patch) with a size of 16*16*3, wherein 3 represents RGB three channels, finally the 576 two-dimensional patches (patch) are spliced according to the respective corresponding position encoding to form an image output.

[0102] As an optional implementation, the convolution operation mentioned in the above step S144 and the above step S148 is performed by using a 1*1 convolution layer, and the purpose is to compress a three-channel image into a single-channel binary image.

[0103] In some embodiments of the present application, the implementation process of the above step S150 includes but is not limited to the following:

[0104] Step S151, based on a preset structure loss function, a first loss value between the plurality of predicted saliency feature images and the plurality of preprocessed real saliency feature images is calculated.

[0105] More specifically, for each preprocessed first image associated predicted saliency feature image and preprocessed real saliency feature image, the first structure loss value between the predicted saliency feature image and the preprocessed real saliency feature image is calculated by the structure loss function, in this way, a plurality of first structure loss values associated with the plurality of preprocessed first images can be calculated, and the plurality of first structure loss values are averaged to obtain the first loss value; as an optional implementation, the structure loss function can adopt an existing structure similarity loss function;

[0106] Step S152, based on the structure loss function, a second loss value between the plurality of predicted camouflage feature images and the plurality of preprocessed real camouflage feature images is calculated.

[0107] More specifically, for each preprocessed second image associated predicted camouflage feature image and preprocessed real camouflage feature image, the second structure loss value between the predicted camouflage feature image and the preprocessed real camouflage feature image is calculated by the structure loss function, in this way, a plurality of second structure loss values associated with the plurality of preprocessed second images can be calculated, and the plurality of second structure loss values are averaged to obtain the second loss value;

[0108] Step S153, the first loss value and the second loss value are added to obtain a total loss value.

[0109] In some embodiments of the present application, after the training task of the joint detection network is completed through the above step S110 to the above step S160, the trained joint detection network can be put into formal application, which specifically includes the following steps:

[0110] (1) obtaining a to-be-detected image containing a salient target and a camouflage target;

[0111] (2) feature extraction of the to-be-detected image is performed by the trained joint detection network to obtain a saliency feature image and a camouflage feature image associated with the to-be-detected image;

[0112] More specifically, the to-be-detected image is sequentially subjected to patch segmentation, patch flattening and position encoding to form a corresponding first to-be-detected token sequence;

[0113] input the first to-be-tested token sequence into the trained joint detection network for feature extraction, in the process, all the second distribution learning modules are not used (that is, only all the first distribution learning modules are used to assist in performing the salient feature detection task), to obtain a second to-be-tested token sequence; after the second to-be-tested token sequence is sequentially subjected to dimension expansion, patch reconstruction and patch splicing, a first feature image is formed; convolution operation is performed on the first feature image, to obtain a salient feature image associated with the to-be-detected image;

[0114] input the first to-be-tested token sequence into the trained joint detection network for feature extraction, in the process, all the first distribution learning modules are not used (that is, only all the second distribution learning modules are used to assist in performing the camouflage feature detection task), to obtain a third to-be-tested token sequence; after the third to-be-tested token sequence is sequentially subjected to dimension expansion, patch reconstruction and patch splicing, a second feature image is formed; convolution operation is performed on the second feature image, to obtain a camouflage feature image associated with the to-be-detected image.

[0115] In order to verify the effectiveness of the trained joint detection network proposed in the application, two groups of test experiments are mainly carried out here, which are specifically described as follows:

[0116] In the first group of test experiments, a first test image and a real salient feature image associated therewith are obtained, feature extraction is performed on the first test image by the trained joint detection network, to obtain a first salient feature image and a first camouflage feature image, as shown in FIG. 6, it can be known that the difference between the first salient feature image and the real salient feature image is small; Figure 5

[0117] In the second group of test experiments, a second test image and a real camouflage feature image associated therewith are obtained, feature extraction is performed on the second test image by the trained joint detection network, to obtain a second salient feature image and a second camouflage feature image, as shown in FIG. 7, it can be known that the difference between the second camouflage feature image and the real camouflage feature image is small. Figure 5

[0118] In the embodiment of the application, by introducing two independent distribution learning models in the decoder to learn the characteristics of different target detection tasks, the joint detection network can have the ability to effectively and reliably detect camouflage features and salient features in the same input image; by limiting the number of salient image pairs contained in the first data set to be equal to the number of camouflage image pairs contained in the second data set, it is avoided that the joint detection network is dominated by the task with larger data size in the training process, that is, it is avoided that the performance of the joint detection network on the task with smaller data size is deteriorated. ​​

[0119] Further, the present application also provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the joint detection network training method of the camouflage target and the salient target provided in the above embodiments. The computer readable storage medium includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the storage device includes any medium that stores or transmits information in a readable form by a device (such as a computer, a mobile phone, etc.), which can be a read-only memory, a magnetic disk or an optical disk, etc.

[0120] Further, Figure 6 is a schematic diagram of a hardware structure of a computer device provided by the present application. The computer device includes a processor 220, a memory 230, an input unit 240, a display unit 250, and the like. Those skilled in the art can understand that the computer device can further include other components, which are not shown in the figure, without departing from the scope of the present application. Figure 6 The device structure shown in the figure does not constitute a limitation on all devices, and can include more or fewer components or combine certain components. The memory 230 can be used to store the computer program 210 and various functional modules. The processor 220 runs the computer program 210 stored in the memory 230, so as to perform various functional applications and data processing of the device. The memory can be an internal memory or an external memory, or include an internal memory and an external memory. The internal memory can include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, or a random memory. The external memory can include a hard disk, a floppy disk, a U disk, a magnetic tape, and the like. The memory 230 disclosed in the present application includes, but is not limited to, the above-mentioned types of memory. The memory 230 disclosed in the present application is only an example, not a limitation.

[0121] The input unit 240 is configured to receive input of a signal and receive a keyword input by a user. The input unit 240 can include a touch panel and other input devices. The touch panel can collect a touch operation of a user thereon or adjacent thereto (e.g., an operation of the user on or adjacent to the touch panel by using a finger, a stylus, or any suitable object or accessory) and drive a corresponding connection device according to a pre-set program; the other input devices can include one or more of a physical keyboard, function keys (e.g., play control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc. The display unit 250 can be configured to display information input by a user or information provided to the user and various menus of the terminal device. The display unit 250 can take the form of a liquid crystal display, an organic light-emitting diode, etc. The processor 220 is a control center of the terminal device, which connects various parts of the entire device by using various interfaces and lines, performs various functions and processes data by running or executing software programs and / or modules stored in the memory 230 and calling data stored in the memory 230.

[0122] As an embodiment, the computer device includes a processor 220, a memory 230, and a computer program 210, wherein the computer program 210 is stored in the memory 230 and configured to be executed by the processor 220, and the computer program 210 is configured to perform the joint detection network training method of the camouflage target and the salient target in one of the above embodiments.

[0123] The terms "comprise", "have" and any variations thereof in the specification and above drawings are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0124] In this application, it should be understood that "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including single item or any combination of multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0125] While the application has been described in connection with very specific embodiments, it will be understood that the application is not limited to any of these details or embodiments or any special embodiment, but intends to cover all alternatives, modifications and equivalents as can be included within the scope of the application as defined by the appended claims when interpreted in accordance with the full scope of equivalents, rules of construction and the doctrine of equivalents under 35 U.S.C. § 112. Furthermore, the above description is intended to provide a useful description of the application, and is not intended to limit the application to the particular embodiments or examples described. It is being appreciated that those skilled in the art, who read the present disclosure, can easily conceive equivalents and modifications of the present application, and those are intended to be included within the scope of the present application.

Claims

1. A method for training a joint detection network for camouflaged targets and salient targets, characterized in that, The method comprises: obtaining a first data set and a second data set, the first data set comprising a plurality of saliency image pairs, and the second data set comprising a plurality of camouflage image pairs; sampling a plurality of saliency image pairs from the first data set, each saliency image pair comprising a first image and a corresponding real saliency feature image, and pre-processing the plurality of saliency image pairs; sampling a plurality of camouflage image pairs from the second data set, each camouflage image pair comprising a second image and a corresponding real camouflage feature image, and pre-processing the plurality of camouflage image pairs; obtaining a joint detection network and performing feature extraction on the pre-processed plurality of first images and plurality of second images to obtain a plurality of predicted saliency feature images and a plurality of predicted camouflage feature images; determining a total loss value according to the plurality of predicted saliency feature images, the plurality of predicted camouflage feature images, and the pre-processed plurality of real saliency feature images and plurality of real camouflage feature images; performing parameter optimization on the joint detection network using a back propagation algorithm according to the total loss value; wherein the joint detection network comprises an encoder, a linear layer, and a decoder connected in sequence; the encoder comprises a plurality of original transformers connected in sequence, each original transformer comprising a multi-head attention layer, a first residual connection layer, a multi-layer perceptron, and a second residual connection layer connected in sequence; the decoder comprises a plurality of improved transformers connected in sequence, each improved transformer being obtained by adding a first distribution learning module for cooperating to perform a saliency feature detection task and a second distribution learning module for cooperating to perform a camouflage feature detection task at the output end of an original transformer; when the original transformer outputs a feature sequence, the first distribution learning module processes the feature sequence by the following expression after receiving the feature sequence: the second distribution learning module processes the feature sequence by the following expression after receiving the feature sequence: wherein, is an output result of the first distribution learning module, is the feature sequence, is a first learnable mean, is a first learnable variance, is an output result of the second distribution learning module, is a second learnable mean, is a second learnable variance.

2. The combined camouflage target and salient target detection network training method according to claim 1, characterized in that, the obtaining of the joint detection network and the performing of the feature extraction on the pre-processed plurality of first images and plurality of second images to obtain a plurality of predicted saliency feature images and a plurality of predicted camouflage feature images comprises: generating a plurality of first token sequences corresponding to the pre-processed plurality of first images; inputting the plurality of first token sequences into the joint detection network for feature extraction while disabling all second distribution learning modules to obtain a plurality of second token sequences corresponding to the plurality of first token sequences; generating a plurality of first detection images corresponding to the plurality of second token sequences; and performing convolution operation on the plurality of first detection images to obtain a plurality of predicted saliency feature images corresponding to the plurality of first detection images. According to the preprocessed plurality of second images, a plurality of third token sequences corresponding thereto are generated; the plurality of third token sequences are input into the joint detection network for feature extraction while all first distribution learning modules are disabled, so as to obtain a plurality of fourth token sequences corresponding thereto; according to the plurality of fourth token sequences, a plurality of second detection images corresponding thereto are generated; and the plurality of second detection images are subjected to convolution operation, so as to obtain a plurality of predicted camouflage feature images corresponding thereto.

3. The method of claim 1, wherein the method further comprises: The first data set contains a number of significant image pairs equal to the number of camouflage image pairs contained in the second data set, and the number of camouflage image pairs contained in the second data set is denoted as M. The first data set is obtained by the following method: An original data set is obtained, which includes N original significant image pairs and N > M, and each original significant image pair includes an original image and a corresponding original significant feature image; The N original images are processed by using the trained significant target detection network to obtain N first predicted significant feature images corresponding thereto; The minimum mean square error between each first predicted significant feature image and the associated original significant feature image is determined, and M minimum mean square errors with the smallest values are selected from the N minimum mean square errors, and the M original images and M original significant feature images associated with the M minimum mean square errors are taken as the first data set.

4. The method of claim 1, wherein the method further comprises: The preprocessing of the plurality of significant image pairs includes synchronously performing scaling and data augmentation on the first image and the real significant feature image contained in each significant image pair.

5. The method of claim 1, wherein the method further comprises: The preprocessing of the plurality of camouflage image pairs includes synchronously performing scaling and data augmentation on the second image and the real camouflage feature image contained in each camouflage image pair.

6. The method of claim 1, wherein, The determination of the total loss value according to the plurality of predicted significant feature images, the plurality of predicted camouflage feature images, and the plurality of preprocessed real significant feature images and the plurality of real camouflage feature images includes: A structure loss function is used to determine a first loss value between the plurality of predicted significant feature images and the plurality of preprocessed real significant feature images, and a second loss value between the plurality of predicted camouflage feature images and the plurality of preprocessed real camouflage feature images; The first loss value and the second loss value are added to obtain a total loss value.

7. The method of claim 1, wherein the method further comprises: The method further includes: An image to be detected containing a camouflage target and a significant target is obtained; The image to be detected is subjected to feature extraction by using the trained joint detection network to obtain a corresponding camouflage feature image and a significant feature image.

8. A computer device comprising a memory and a processor, said memory having stored thereon a computer program, characterized in that, The processor executes the computer program to implement the joint detection network training method for the camouflage target and the significant target according to any one of claims 1 to 7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the joint detection network training method for the camouflage target and the significant target according to any one of claims 1 to 7.

Images