Camouflage object semantic segmentation method, device and equipment based on self-supervised dual construction model, and storage medium

Abstract

The invention provides a camouflage object semantic segmentation method, device and equipment based on a self-supervised dual construction model, and a storage medium. Relates to the technical field of computer vision. The method comprises the following steps of: an image reconstruction stage: randomly masking pixels in an input image, extracting the masked image through a backbone network to obtain semantic features, and aggregating the semantic features by using a boundary self-adaptive feeling module to reconstruct the masked pixels; in the model refining stage, the weight of a boundary self-adaptive feeling module is randomly initialized, meanwhile, semantic features extracted by a backbone network are reserved, and full-supervised training is carried out to obtain a preliminary segmentation result; and a label reconstruction stage: using a backbone network to extract semantic features from the noise label, the partial label and the complete label so as to carry out feature extraction on the preliminary segmentation result, and then using a distance adaptive asymmetric module to carry out reconstruction so as to obtain a final segmentation result. According to the invention, the camouflage object can be accurately segmented.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, equipment and storage medium for semantic segmentation of camouflaged objects based on self-supervised dual modeling. Background Art

[0002] Semantic segmentation of camouflaged objects is a complex visual task that aims to accurately identify camouflaged objects from images with complex backgrounds and blurred target boundaries. This technology has important application value in fields such as medical image processing (such as lesion detection) and unmanned driving (such as road obstacle detection).

[0003] In the prior art, the task of camouflaged object segmentation mainly relies on fully supervised learning methods, which require a large amount of high-quality labeled data to train deep neural networks. However, in practical applications, data labeling is not only costly, but also has the problem of inconsistent labeling. In addition, fully supervised models are usually highly dependent on the completeness and accuracy of labels, resulting in a significant decrease in segmentation accuracy when labels are partially missing or noisy. Summary of the invention

[0004] The present application provides a method, apparatus, device and storage medium for semantic segmentation of camouflaged objects based on self-supervised dual construction modeling. By proposing a self-supervised mask image reconstruction stage, a model refinement stage, and an optimization stage using label reconstruction, high-precision segmentation of camouflaged objects is achieved, and the method has the advantages of strong noise resistance and fine detail processing.

[0005] In a first aspect, the present application provides a method for semantic segmentation of camouflaged objects based on self-supervised dual modeling, comprising:

[0006] Randomly masking pixels in the image to obtain a mask image, extracting the mask image through a backbone network to obtain a first multi-scale semantic feature, and aggregating the semantic features using a boundary adaptive perception module to reconstruct the masked pixels;

[0007] The weights of the boundary adaptive perception module are randomly initialized, while retaining the first multi-scale semantic features extracted by the backbone network, and full-supervised training is performed to obtain preliminary segmentation results;

[0008] A backbone network is used to extract second multi-scale semantic features from noise labels, partial labels and complete labels to perform feature extraction on the preliminary segmentation result, and then a distance adaptive asymmetric module is used to reconstruct the final segmentation result.

[0009] In a possible design, the mask image is extracted through a backbone network to obtain a first multi-scale semantic feature, and a boundary adaptive perception module is used to aggregate the semantic features to reconstruct the masked pixels, including:

[0010] Using the backbone network to extract semantic features from the mask image to obtain a first semantic feature map, a second semantic feature map, a third semantic feature map, and a fourth semantic feature map;

[0011] The first semantic feature map, the second semantic feature map, the third semantic feature map and the fourth semantic feature map are sent to a boundary adaptive perception module to restore the masked part of the image for self-supervised reconstruction training.

[0012] In one possible design, the weights of the boundary adaptive perception module are randomly initialized while retaining the first multi-scale semantic features extracted by the backbone network, and fully supervised training is performed to obtain preliminary segmentation results, including:

[0013] Based on the boundary adaptive perception module, a random initialization operation is performed, and the first semantic feature map, the second semantic feature map, the third semantic feature map and the fourth semantic feature map are sent to the boundary adaptive perception module for full-supervised training to obtain a preliminary segmentation result.

[0014] In a possible design, a backbone network is used to extract second multi-scale semantic features from noise labels, partial labels, and complete labels to perform feature extraction on the preliminary segmentation result, and then a distance adaptive asymmetric module is used to reconstruct the final segmentation result, including:

[0015] The preliminary segmentation results are randomly masked and noise is randomly added to obtain a partial feature map and a noise feature map;

[0016] Inputting the preliminary segmentation result, the partial feature map and the noise feature map into a backbone network to extract semantic features to obtain a fifth semantic feature map, a sixth semantic feature map, a seventh semantic feature map and an eighth semantic feature map;

[0017] The fifth semantic feature map, the sixth semantic feature map, the seventh semantic feature map and the eighth semantic feature map are sent to a distance adaptive asymmetric module for reconstruction to obtain a final segmentation result.

[0018] In a possible design, the backbone network includes a Transformer backbone network or a convolutional backbone network; the Transformer backbone network implements feature modeling based on a PVTv2 backbone network using a multi-layer multi-head self-attention mechanism and multi-layer linear interpolation sampling; the convolutional backbone network implements feature modeling based on a Res2Net backbone network using multi-layer split channel convolution and multi-layer channel splicing and residual connection;

[0019] The boundary adaptive perception module includes a boundary adaptive perception convolution and a multi-branch channel aggregation structure;

[0020] The boundary adaptive receptive convolution includes an adaptive hole convolution branch, a boundary receptive branch and an attention weighting mechanism;

[0021] The adaptive dilated convolution branch includes four dilated convolutions with 3*3 convolution kernels and dilated coefficients of 1, 3, 5, and 7 respectively connected in sequence;

[0022] The boundary perception branch includes: Gaussian, Sobel and Laplace operations connected in sequence and an average weighted fusion operation;

[0023] The attention weighting mechanism includes sequentially connected convolution, Relu activation function, and Softmax activation function;

[0024] The output result of the boundary adaptive receptive convolution is obtained by element-wise multiplication of the output of the dilated convolution branch and the output of the attention weighting mechanism;

[0025] The multi-branch channel aggregation structure includes a channel Split operation, four parallel boundary adaptive perception convolutions and four parallel CBR operations connected in sequence, and the output result of the multi-branch channel aggregation structure is obtained by merging the outputs of the four CBR operations and feeding them into one CBR operation;

[0026] The CBR operation includes: convolution, normalization and ReLU activation function connected in sequence.

[0027] In a possible design, the distance adaptive asymmetric module includes a distance adaptive asymmetric convolution and a multi-branch channel aggregation structure;

[0028] The distance adaptive asymmetric convolution includes asymmetric convolution, depth convolution and parallel SEA attention mechanism and spatial pixel distance attention weighted mechanism connected in sequence. The output result of the distance adaptive asymmetric convolution is obtained by element-wise multiplication of the output of the SEA attention mechanism and the output of the spatial pixel distance attention weighted mechanism.

[0029] In a second aspect, the present application provides a camouflaged object semantic segmentation device based on self-supervised dual construction modeling, the device comprising:

[0030] A backbone network module is configured to randomly mask pixels in an image to obtain a mask image, extract the mask image through the backbone network to obtain a first multi-scale semantic feature, and use a boundary adaptive perception module to aggregate the semantic features to reconstruct the masked pixels;

[0031] The boundary adaptive perception module is configured to randomly initialize the weights of the boundary adaptive perception module, while retaining the first multi-scale semantic features extracted by the backbone network, and performing full-supervised training to obtain preliminary segmentation results;

[0032] The distance adaptive asymmetric module is configured to use the backbone network to extract second multi-scale semantic features from the noise labels, partial labels and complete labels to perform feature extraction on the preliminary segmentation result, and then use the distance adaptive asymmetric module to reconstruct the final segmentation result.

[0033] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the semantic segmentation method of camouflaged objects based on self-supervised dual modeling as described in the first aspect and various possible designs of the first aspect.

[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the method for semantic segmentation of camouflaged objects based on self-supervised dual modeling as described in the first aspect and various possible designs of the first aspect is implemented.

[0035] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the semantic segmentation method of camouflaged objects based on self-supervised dual modeling as described in the first aspect and various possible designs of the first aspect.

[0036] The method, device, equipment and storage medium for semantic segmentation of camouflaged objects based on self-supervised dual modeling provided by the present application have at least the following beneficial effects:

[0037] The present application provides a method for semantic segmentation of camouflaged objects based on self-supervised dual construction modeling, including the following operations: in the image reconstruction stage, pixels in the input image are randomly masked, the masked image is extracted through a backbone network to obtain semantic features, and a boundary adaptive perception module is used to aggregate semantic features to reconstruct masked pixels; in the model refinement stage, the weights of the boundary adaptive perception module are randomly initialized, while retaining the semantic features extracted by the backbone network, and full supervised training is performed to obtain preliminary segmentation results; in the label reconstruction stage, the backbone network is used to extract semantic features from noise labels, partial labels, and complete labels to extract features from the preliminary segmentation results, and then the distance adaptive asymmetric module is used to reconstruct the final segmentation results. The present invention can accurately segment camouflaged objects.

[0038] In the above operation process, when implementing the segmentation of camouflaged objects in an image, an image is provided as input to the model, and the model will segment out the pixel area containing the camouflaged object in the image. Specifically, the reason why the Transformer backbone network or the convolutional backbone network is used is that the semantic feature extraction encoded by the above backbone network is conducive to the model to better explore the information of the camouflaged object in the image. The reason why the self-supervised dual reconstruction learning method is used is that the self-supervised learning method helps the model learn the feature information hidden inside the image outside the label. The self-supervised reconstruction task allows the model to learn important feature representations from the input image without a large amount of labeled data.

[0039] Specifically, the reason why the boundary-adaptive perception module is used is that fusing adjacent features by weighting different coefficients of dilated convolution with edge information attention mechanism is conducive to the full fusion of multi-scale information. The boundary-adaptive perception module can avoid ignoring the subtle differences between the camouflaged object and the background by paying weighted attention to the boundary area, thereby enhancing the model's perception of the edge of the object. With the help of dilated convolution, this weighted mechanism can capture a wider range of contextual information, thereby achieving accurate segmentation of camouflaged objects; the distance-adaptive asymmetric module is then used because the asymmetric convolution with different coefficients of pixel space attention mechanism is conducive to the model to fully explore the structural details of the camouflaged object. Through asymmetric convolution, the model can more flexibly learn structural information in different directions, enhance the recognition of the details of the camouflaged object, and thus avoid confusion between the detailed features of the camouflaged object and the background area.

[0040] In the task of camouflaged object segmentation, this method achieves accurate segmentation of camouflaged objects by combining self-supervised dual reconstruction learning, boundary adaptive perception module and distance adaptive asymmetric module. It can effectively distinguish camouflaged objects from background areas, and shows high robustness and accuracy, especially in complex camouflage environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] Figure 1 A flowchart of a method for semantic segmentation of camouflaged objects based on self-supervised dual modeling provided in an embodiment of the present application;

[0043] Figure 2 A model structure diagram provided for an embodiment of the present application;

[0044] Figure 3 A schematic diagram of an input image provided in an embodiment of the present application;

[0045] Figure 4A schematic diagram of pixel-level labeling provided in an embodiment of the present application;

[0046] Figure 5 The effect diagram of the camouflaged object semantic segmentation method based on self-supervised dual modeling provided in the embodiment of the present application;

[0047] Figure 6 The effect diagram of HitNet provided in the embodiment of the present application;

[0048] Figure 7 The effect diagram of VSCode provided in the embodiment of the present application;

[0049] Figure 8 The effect diagram of CamoDiffusion provided in the embodiment of this application;

[0050] Fig. 9 A schematic diagram of the structure of a camouflaged object semantic segmentation device based on self-supervised dual modeling provided in an embodiment of the present application.

[0051] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0052] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0053] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0054] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0055] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0056] The embodiment of the present application provides a semantic segmentation method for camouflaged objects based on self-supervised dual construction modeling, including: in an image reconstruction stage, pixels in an input image are randomly masked, the masked image is extracted through a backbone network to obtain semantic features, and a boundary adaptive perception module is used to aggregate the semantic features to reconstruct the masked pixels; in a model refinement stage, the weights of the boundary adaptive perception module are randomly initialized, while the semantic features extracted by the backbone network are retained, and full supervised training is performed to obtain a preliminary segmentation result; in a label reconstruction stage, a backbone network is used to extract semantic features from noise labels, partial labels and complete labels to perform feature extraction on the preliminary segmentation result, and then a distance adaptive asymmetric module is used to reconstruct the final segmentation result.

[0057] Specifically, Figure 1 A flowchart of a method for semantic segmentation of camouflaged objects based on self-supervised dual modeling provided in an embodiment of the present application. Figure 1 As shown, the method for semantic segmentation of disguised objects based on self-supervised dual modeling includes steps S1 to S3.

[0058] In the S1 image reconstruction stage, the pixels in the input image are randomly masked, and the masked image is extracted through the backbone network to obtain semantic features. The boundary adaptive perception module is used to aggregate the semantic features to reconstruct the masked pixels.

[0059] In the S2 model refinement stage, the weights of the boundary adaptive perception module are randomly initialized, while the semantic features extracted by the backbone network are retained, and fully supervised training is performed to obtain preliminary segmentation results.

[0060] In the S3 label reconstruction stage, a backbone network is used to extract semantic features from noise labels, partial labels and complete labels to perform feature extraction on the preliminary segmentation results, and then a distance adaptive asymmetric module is used to reconstruct the final segmentation results.

[0061] like Figure 2 As shown, the embodiment of the present application specifically divides the semantic segmentation task of camouflaged objects based on self-supervised dual construction modeling into the following three processing stages, namely, the image reconstruction stage, the model refinement stage and the label reconstruction stage.

[0062] In the image reconstruction stage, pixels in the input image are randomly masked, the masked image is extracted through the backbone network to obtain semantic features, and the boundary adaptive perception module is used to aggregate the semantic features to reconstruct the masked pixels;

[0063] In the model refinement stage, the weights of the boundary adaptive perception module are randomly initialized, while the semantic features extracted by the backbone network are retained, and full-supervised training is performed to obtain preliminary segmentation results;

[0064] For the label reconstruction stage, a backbone network is used to extract semantic features from noise labels, partial labels and complete labels to perform feature extraction on the preliminary segmentation result, and then a distance adaptive asymmetric module is used to reconstruct the final segmentation result.

[0065] As one or more embodiments, during the execution of the above step S1, pixels in the input image are randomly masked, the masked image is extracted through the backbone network to obtain semantic features, and the boundary adaptive perception module is used to aggregate the semantic features to reconstruct the masked pixels, which specifically includes the following steps:

[0066] S101: Using a backbone network to extract semantic features from a randomly masked input image to obtain a first semantic feature map, a second semantic feature map, a third semantic feature map, and a fourth semantic feature map.

[0067] S102: Based on the first semantic feature map, the second semantic feature map, the third semantic feature map and the fourth semantic feature map, the masked part of the image is restored by sending them to a boundary adaptive perception module for self-supervised reconstruction training.

[0068] The boundary adaptive perception module includes a boundary adaptive perception convolution and a multi-branch channel aggregation structure.

[0069] The boundary adaptive receptive convolution includes: an adaptive void convolution branch, a boundary receptive branch and an attention weighting mechanism.

[0070] The adaptive dilated convolution branch includes: four dilated convolutions whose convolution kernels are 3*3 and dilated coefficients are 1, 3, 5, and 7 respectively, which are connected in sequence.

[0071] The boundary perception branch includes: Gaussian, Sobel and Laplace operations connected in sequence and an average weighted fusion operation.

[0072] The attention weighting mechanism includes: sequentially connected convolution, Relu activation function, and Softmax activation function, and then the output result of the boundary adaptive receptive convolution is obtained by element-wise multiplication of the output of the hole convolution branch and the output of the attention weighting mechanism.

[0073] The multi-branch channel aggregation structure includes: a channel Split operation, four parallel boundary adaptive perception convolutions and four parallel CBR operations connected in sequence, and then the outputs of the four CBR operations are merged and sent to one CBR operation to obtain the output result of the multi-branch channel aggregation structure.

[0074] The CBR operation includes: sequentially connected convolution, normalization and ReLU activation function.

[0075] As one or more embodiments, during the execution of step S2, based on the semantic features, the weights of the boundary adaptive perception module are randomly initialized, while retaining the semantic features extracted by the backbone network, and performing full supervised training to obtain a preliminary segmentation result, specifically including the following steps:

[0076] S201: Randomly initialize the weights of the boundary adaptive perception module.

[0077] S202: retain the semantic features extracted by the backbone network, add the camouflaged object image labels, perform full-supervised training to refine the model and obtain preliminary segmentation results.

[0078] As one or more embodiments, during the execution of step S3, based on the preliminary segmentation result, a backbone network is used to extract semantic features from noise labels, partial labels and complete labels to perform feature extraction on the preliminary segmentation result, and then a distance adaptive asymmetric module is used to reconstruct the final segmentation result, specifically including:

[0079] S301: randomly masking and randomly adding noise to the preliminary segmentation result to obtain a partial feature map and a noise feature map.

[0080] S302: Input the preliminary segmentation result, the partial feature map and the noise feature map into the backbone network to extract semantic features to obtain a fifth semantic feature map, a sixth semantic feature map, a seventh semantic feature map and an eighth semantic feature map.

[0081] S303: Send the fifth semantic feature map, the sixth semantic feature map, the seventh semantic feature map and the eighth semantic feature map to a distance adaptive asymmetric module for reconstruction to obtain a final segmentation result.

[0082] The distance adaptive asymmetric convolution includes: sequentially connected asymmetric convolution, depth convolution and parallel SEA attention mechanism and spatial pixel distance attention weighted mechanism, and then element-wise multiplication of the output of the SEA attention mechanism and the output of the spatial pixel distance attention weighted mechanism is performed to obtain the output result of the distance adaptive asymmetric convolution.

[0083] The Transformer backbone network includes but is not limited to any implementation method: a PVTv2 backbone network that implements feature modeling based on a multi-layer multi-head self-attention mechanism and multi-layer linear interpolation sampling;

[0084] The convolutional backbone network includes but is not limited to any implementation method: a Res2Net backbone network that implements feature modeling based on multi-layer split channel convolution and multi-layer channel splicing and residual connection.

[0085] Deep neural network training and parameter initialization: In an embodiment of the present invention, the parameter initialization weight parameters of the backbone network adopt PVTv2 obtained by fully supervised pre-training on the ImageNet dataset.

[0086] Training optimization details: In the embodiment of the present application, AdamW optimizer is used for training, the training rounds are set to 300 rounds, the initial learning rate is set to 0.0001, the batch size is 8, and the weight decay rate is 0.0001.

[0087] like Figure 3 As shown, it is a schematic diagram of the input image of this embodiment, Figure 4 is the pixel-level label, Figure 5 This is a rendering of the method according to this embodiment. Figures 6 to 8 Table 1 shows the quantitative comparison results of the method of the embodiment of the present application and the ten most advanced semantic segmentation methods of camouflaged objects based on self-supervised dual modeling.

[0088]

[0089]

[0090] Table 1 Quantitative comparison

[0091] The present application also provides a semantic segmentation device for camouflaged objects based on self-supervised dual modeling. Fig. 9 As shown, the camouflaged object semantic segmentation device based on self-supervised dual modeling includes:

[0092] The backbone network module 901 is configured to randomly mask pixels in the image to obtain a mask image, extract the mask image through the backbone network to obtain a first multi-scale semantic feature, and use a boundary adaptive perception module to aggregate the semantic features to reconstruct the masked pixels;

[0093] The boundary adaptive perception module 902 is configured to randomly initialize the weights of the boundary adaptive perception module, while retaining the first multi-scale semantic features extracted by the backbone network, and performing full-supervised training to obtain a preliminary segmentation result;

[0094] The distance adaptive asymmetric module 903 is configured to use the backbone network to extract second multi-scale semantic features from the noise labels, partial labels and complete labels to perform feature extraction on the preliminary segmentation result, and then use the distance adaptive asymmetric module to reconstruct the final segmentation result.

[0095] In some embodiments, the backbone network module is further configured to:

[0096] Using the backbone network to extract semantic features from the mask image to obtain a first semantic feature map, a second semantic feature map, a third semantic feature map, and a fourth semantic feature map;

[0097] The first semantic feature map, the second semantic feature map, the third semantic feature map and the fourth semantic feature map are sent to a boundary adaptive perception module to restore the masked part of the image for self-supervised reconstruction training.

[0098] In some embodiments, the boundary adaptive perception module is further configured to:

[0099] Based on the boundary adaptive perception module, a random initialization operation is performed, and the first semantic feature map, the second semantic feature map, the third semantic feature map and the fourth semantic feature map are sent to the boundary adaptive perception module for full-supervised training to obtain a preliminary segmentation result.

[0100] In some embodiments, the distance adaptive asymmetric module is further configured to:

[0101] The preliminary segmentation results are randomly masked and noise is randomly added to obtain a partial feature map and a noise feature map;

[0102] Inputting the preliminary segmentation result, the partial feature map and the noise feature map into a backbone network to extract semantic features to obtain a fifth semantic feature map, a sixth semantic feature map, a seventh semantic feature map and an eighth semantic feature map;

[0103] The fifth semantic feature map, the sixth semantic feature map, the seventh semantic feature map and the eighth semantic feature map are sent to a distance adaptive asymmetric module for reconstruction to obtain a final segmentation result.

[0104] In some embodiments, the backbone network includes a Transformer backbone network or a convolutional backbone network; the Transformer backbone network implements feature modeling based on a PVTv2 backbone network with multi-layer multi-head self-attention mechanism and multi-layer linear interpolation sampling; the convolutional backbone network implements feature modeling based on a Res2Net backbone network with multi-layer split channel convolution and multi-layer channel splicing and residual connection;

[0105] The boundary adaptive perception module includes a boundary adaptive perception convolution and a multi-branch channel aggregation structure;

[0106] The boundary adaptive receptive convolution includes an adaptive hole convolution branch, a boundary receptive branch and an attention weighting mechanism;

[0107] The adaptive dilated convolution branch includes four dilated convolutions with 3*3 convolution kernels and dilated coefficients of 1, 3, 5, and 7 respectively connected in sequence;

[0108] The boundary perception branch includes: Gaussian, Sobel and Laplace operations connected in sequence and an average weighted fusion operation;

[0109] The attention weighting mechanism includes sequentially connected convolution, Relu activation function, and Softmax activation function;

[0110] The output result of the boundary adaptive receptive convolution is obtained by element-wise multiplication of the output of the dilated convolution branch and the output of the attention weighting mechanism;

[0111] The multi-branch channel aggregation structure includes a channel Split operation, four parallel boundary adaptive perception convolutions and four parallel CBR operations connected in sequence, and the output result of the multi-branch channel aggregation structure is obtained by merging the outputs of the four CBR operations and feeding them into one CBR operation;

[0112] The CBR operation includes: convolution, normalization and ReLU activation function connected in sequence.

[0113] In some embodiments, the distance adaptive asymmetric module includes a distance adaptive asymmetric convolution and a multi-branch channel aggregation structure;

[0114] The distance adaptive asymmetric convolution includes asymmetric convolution, depth convolution and parallel SEA attention mechanism and spatial pixel distance attention weighted mechanism connected in sequence. The output result of the distance adaptive asymmetric convolution is obtained by element-wise multiplication of the output of the SEA attention mechanism and the output of the spatial pixel distance attention weighted mechanism.

[0115] An embodiment of the present application provides an electronic device, which may include: a processor and a memory, wherein the processor and the memory may communicate with each other; illustratively, the processor and the memory communicate with each other via a communication bus.

[0116] The processor executes the computer execution instructions stored in the memory, so that the processor executes the scheme in the above embodiment. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components.

[0117] The communication bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The system bus can be divided into an address bus, a data bus, a control bus, etc. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0118] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0119] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the technical solution of the method for semantic segmentation of camouflaged objects based on self-supervised dual modeling in the above embodiment.

[0120] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution of the semantic segmentation method of camouflaged objects based on self-supervised dual construction modeling in the above embodiment.

[0121] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0122] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0123] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0124] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0125] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0126] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0127] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0128] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0129] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a main control device.

[0130] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for semantic segmentation of camouflaged objects based on self-supervised dual modeling, characterized in that: The method comprises: Randomly masking pixels in the image to obtain a mask image, extracting the mask image through a backbone network to obtain a first multi-scale semantic feature, and aggregating the semantic features using a boundary adaptive perception module to reconstruct the masked pixels; The weights of the boundary adaptive perception module are randomly initialized, while retaining the first multi-scale semantic features extracted by the backbone network, and full-supervised training is performed to obtain preliminary segmentation results; A backbone network is used to extract second multi-scale semantic features from noise labels, partial labels and complete labels to perform feature extraction on the preliminary segmentation result, and then a distance adaptive asymmetric module is used to reconstruct the final segmentation result.

2. The method for semantic segmentation of camouflaged objects based on self-supervised dual modeling according to claim 1, characterized in that: The mask image is extracted through a backbone network to obtain a first multi-scale semantic feature, and a boundary adaptive perception module is used to aggregate the semantic features to reconstruct the masked pixels, including: Using the backbone network to extract semantic features from the mask image to obtain a first semantic feature map, a second semantic feature map, a third semantic feature map, and a fourth semantic feature map; The first semantic feature map, the second semantic feature map, the third semantic feature map and the fourth semantic feature map are sent to a boundary adaptive perception module to restore the masked part of the image for self-supervised reconstruction training.

3. The method for semantic segmentation of camouflaged objects based on self-supervised dual modeling according to claim 2, characterized in that: The weights of the boundary adaptive perception module are randomly initialized, while retaining the first multi-scale semantic features extracted by the backbone network, and full-supervised training is performed to obtain preliminary segmentation results, including: Based on the boundary adaptive perception module, a random initialization operation is performed, and the first semantic feature map, the second semantic feature map, the third semantic feature map and the fourth semantic feature map are sent to the boundary adaptive perception module for full-supervised training to obtain a preliminary segmentation result.

4. The method for semantic segmentation of camouflaged objects based on self-supervised dual modeling according to claim 1, characterized in that: The backbone network is used to extract second multi-scale semantic features from the noise labels, partial labels and complete labels to perform feature extraction on the preliminary segmentation result, and then a distance adaptive asymmetric module is used to reconstruct the final segmentation result, including: The preliminary segmentation results are randomly masked and noise is randomly added to obtain a partial feature map and a noise feature map; Inputting the preliminary segmentation result, the partial feature map and the noise feature map into a backbone network to extract semantic features to obtain a fifth semantic feature map, a sixth semantic feature map, a seventh semantic feature map and an eighth semantic feature map; The fifth semantic feature map, the sixth semantic feature map, the seventh semantic feature map and the eighth semantic feature map are sent to a distance adaptive asymmetric module for reconstruction to obtain a final segmentation result.

5. The method for semantic segmentation of camouflaged objects based on self-supervised dual modeling according to claim 1, characterized in that: The backbone network includes a Transformer backbone network or a convolutional backbone network; the Transformer backbone network implements feature modeling based on a PVTv2 backbone network with multi-layer multi-head self-attention mechanism and multi-layer linear interpolation sampling; the convolutional backbone network implements feature modeling based on a Res2Net backbone network with multi-layer split channel convolution and multi-layer channel splicing and residual connection; The boundary adaptive perception module includes a boundary adaptive perception convolution and a multi-branch channel aggregation structure; The boundary adaptive receptive convolution includes an adaptive hole convolution branch, a boundary receptive branch and an attention weighting mechanism; The adaptive dilated convolution branch includes four dilated convolutions with 3*3 convolution kernels and dilated coefficients of 1, 3, 5, and 7 respectively connected in sequence; The boundary perception branch includes: Gaussian, Sobel and Laplace operations connected in sequence and an average weighted fusion operation; The attention weighting mechanism includes sequentially connected convolution, Relu activation function, and Softmax activation function; The output result of the boundary adaptive receptive convolution is obtained by element-wise multiplication of the output of the dilated convolution branch and the output of the attention weighting mechanism; The multi-branch channel aggregation structure includes a channel Split operation, four parallel boundary adaptive perception convolutions and four parallel CBR operations connected in sequence, and the output result of the multi-branch channel aggregation structure is obtained by merging the outputs of the four CBR operations and feeding them into one CBR operation; The CBR operation includes: convolution, normalization and ReLU activation function connected in sequence.

6. The method for semantic segmentation of camouflaged objects based on self-supervised dual modeling according to claim 1, characterized in that: The distance adaptive asymmetric module includes a distance adaptive asymmetric convolution and a multi-branch channel aggregation structure; The distance adaptive asymmetric convolution includes asymmetric convolution, depth convolution and parallel SEA attention mechanism and spatial pixel distance attention weighted mechanism connected in sequence. The output result of the distance adaptive asymmetric convolution is obtained by element-wise multiplication of the output of the SEA attention mechanism and the output of the spatial pixel distance attention weighted mechanism.

7. A camouflaged object semantic segmentation device based on self-supervised dual modeling, characterized in that: The device comprises: A backbone network module is configured to randomly mask pixels in an image to obtain a mask image, extract the mask image through the backbone network to obtain a first multi-scale semantic feature, and use a boundary adaptive perception module to aggregate the semantic features to reconstruct the masked pixels; The boundary adaptive perception module is configured to randomly initialize the weights of the boundary adaptive perception module, while retaining the first multi-scale semantic features extracted by the backbone network, and performing full-supervised training to obtain preliminary segmentation results; The distance adaptive asymmetric module is configured to use the backbone network to extract second multi-scale semantic features from the noise labels, partial labels and complete labels to perform feature extraction on the preliminary segmentation result, and then use the distance adaptive asymmetric module to reconstruct the final segmentation result.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the camouflaged object semantic segmentation method based on self-supervised dual modeling as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method for semantic segmentation of camouflaged objects based on self-supervised dual modeling according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for semantic segmentation of camouflaged objects based on self-supervised dual modeling as claimed in any one of claims 1 to 6 is implemented.

Images