Method, device, equipment and storage medium for semantic segmentation of camouflaged objects based on self-supervised dual modeling

Abstract

The present application provides a method, apparatus, device and storage medium for semantic segmentation of camouflaged objects based on self-supervised dual construction modeling. It relates to the field of computer vision technology. The method includes: an image reconstruction stage, in which pixels in the input image are randomly masked, the masked image is extracted through a backbone network to obtain semantic features, and a boundary adaptive perception module is used to aggregate the semantic features to reconstruct the masked pixels; a model refinement stage, the weights of the boundary adaptive perception module are randomly initialized, while retaining the semantic features extracted by the backbone network, and full-supervised training is performed to obtain preliminary segmentation results; a label reconstruction stage, a backbone network is used to extract semantic features from noise labels, partial labels and complete labels to perform feature extraction on the preliminary segmentation results, and then a distance adaptive asymmetric module is used to reconstruct the final segmentation results. The present application can achieve accurate segmentation of camouflaged objects.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, apparatus, device and storage medium for semantic segmentation of camouflaged objects based on self-supervised dual modeling. Background Art

[0002] Semantic segmentation of camouflaged objects is a complex visual task that aims to accurately identify camouflaged objects in images with complex backgrounds and blurred object boundaries. This technology has important applications in fields such as medical image processing (such as lesion detection) and autonomous driving (such as road obstacle detection).

[0003] Existing technologies for camouflaged object segmentation rely primarily on fully supervised learning methods, which require large amounts of high-quality labeled data to train deep neural networks. However, in practical applications, data labeling is not only costly but also subject to label inconsistency. Furthermore, fully supervised models often rely heavily on the completeness and accuracy of labels, resulting in a significant drop in segmentation accuracy when labels are partially missing or noisy. Summary of the Invention

[0004] The present application provides a method, apparatus, device and storage medium for semantic segmentation of camouflaged objects based on self-supervised dual construction modeling. By proposing a self-supervised mask image reconstruction stage, a model refinement stage, and an optimization stage using label reconstruction, high-precision segmentation of camouflaged objects is achieved, with advantages such as strong noise resistance and fine detail processing.

[0005] In a first aspect, the present application provides a method for semantic segmentation of camouflaged objects based on self-supervised dual modeling, comprising:

[0006] Randomly masking pixels in the image to obtain a mask image, extracting the mask image through a backbone network to obtain a first multi-scale semantic feature, and aggregating the semantic features using a boundary adaptive perception module to reconstruct the masked pixels;

[0007] Randomly initialize the weights of the boundary adaptive perception module while retaining the first multi-scale semantic features extracted by the backbone network, and perform fully supervised training to obtain preliminary segmentation results;

[0008] A backbone network is used to extract second multi-scale semantic features from the noise labels, partial labels and complete labels to perform feature extraction on the preliminary segmentation result, and then a distance adaptive asymmetric module is used to reconstruct the final segmentation result.

[0009] In one possible design, the mask image is extracted through a backbone network to obtain first multi-scale semantic features, and a boundary adaptive perception module is used to aggregate the semantic features to reconstruct the masked pixels, including:

[0010] Using the backbone network to extract semantic features from the mask image to obtain a first semantic feature map, a second semantic feature map, a third semantic feature map, and a fourth semantic feature map;

[0011] The first semantic feature map, the second semantic feature map, the third semantic feature map and the fourth semantic feature map are sent to a boundary adaptive perception module to restore the masked part of the image for self-supervised reconstruction training.

[0012] In one possible design, the weights of the boundary adaptive perception module are randomly initialized while retaining the first multi-scale semantic features extracted by the backbone network. Fully supervised training is performed to obtain preliminary segmentation results, including:

[0013] Based on the boundary adaptive perception module, a random initialization operation is performed, and the first semantic feature map, the second semantic feature map, the third semantic feature map and the fourth semantic feature map are sent to the boundary adaptive perception module for full-supervised training to obtain a preliminary segmentation result.

[0014] In one possible design, a backbone network is used to extract second multi-scale semantic features from the noisy labels, partial labels, and complete labels to perform feature extraction on the preliminary segmentation result, and then a distance adaptive asymmetric module is used to reconstruct the final segmentation result, including:

[0015] Randomly masking and randomly adding noise to the preliminary segmentation results to obtain a partial feature map and a noise feature map;

[0016] Inputting the preliminary segmentation result, the partial feature map and the noise feature map into a backbone network to extract semantic features to obtain a fifth semantic feature map, a sixth semantic feature map, a seventh semantic feature map and an eighth semantic feature map;

[0017] The fifth semantic feature map, the sixth semantic feature map, the seventh semantic feature map and the eighth semantic feature map are sent to a distance adaptive asymmetric module for reconstruction to obtain a final segmentation result.

[0018] In one possible design, the backbone network includes a Transformer backbone network or a convolutional backbone network; the Transformer backbone network implements feature modeling based on a PVTv2 backbone network using a multi-layer multi-head self-attention mechanism and multi-layer linear interpolation sampling; the convolutional backbone network implements feature modeling based on a Res2Net backbone network using multi-layer split channel convolution and multi-layer channel splicing and residual connections;

[0019] The boundary adaptive perception module includes a boundary adaptive perception convolution and a multi-branch channel aggregation structure;

[0020] The boundary adaptive receptive convolution includes an adaptive hole convolution branch, a boundary receptive branch and an attention weighting mechanism;

[0021] The adaptive dilated convolution branch includes four dilated convolutions with 3*3 convolution kernels and dilated coefficients of 1, 3, 5, and 7 respectively, which are connected in sequence.

[0022] The boundary perception branch includes: Gaussian, Sobel and Laplace operations connected in sequence and average weighted fusion operation;

[0023] The attention weighting mechanism includes sequentially connected convolution, Relu activation function, and Softmax activation function;

[0024] The output of the boundary adaptive receptive convolution is obtained by element-wise multiplication of the output of the dilated convolution branch and the output of the attention weighting mechanism;

[0025] The multi-branch channel aggregation structure includes a channel Split operation, four parallel boundary adaptive perception convolutions, and four parallel CBR operations connected in sequence. The output result of the multi-branch channel aggregation structure is obtained by merging the outputs of the four CBR operations and feeding them into one CBR operation;

[0026] The CBR operation includes: convolution, normalization and ReLU activation function connected in sequence.

[0027] In one possible design, the distance adaptive asymmetric module includes a distance adaptive asymmetric convolution and a multi-branch channel aggregation structure;

[0028] The distance-adaptive asymmetric convolution includes sequentially connected asymmetric convolution, depthwise convolution, and parallel SEA attention mechanism and spatial pixel distance attention weighting mechanism. The output result of the distance-adaptive asymmetric convolution is obtained by element-wise multiplication of the output of the SEA attention mechanism and the output of the spatial pixel distance attention weighting mechanism.

[0029] In a second aspect, the present application provides a semantic segmentation device for disguised objects based on self-supervised dual modeling, the device comprising:

[0030] A backbone network module is configured to randomly mask pixels in an image to obtain a mask image, extract the mask image through the backbone network to obtain a first multi-scale semantic feature, and aggregate the semantic features using a boundary adaptive perception module to reconstruct the masked pixels;

[0031] The boundary adaptive perception module is configured to randomly initialize the weights of the boundary adaptive perception module while retaining the first multi-scale semantic features extracted by the backbone network, and performing fully supervised training to obtain preliminary segmentation results;

[0032] The distance adaptive asymmetric module is configured to use the backbone network to extract second multi-scale semantic features from the noise labels, partial labels and complete labels to perform feature extraction on the preliminary segmentation result, and then use the distance adaptive asymmetric module to reconstruct the final segmentation result.

[0033] In a third aspect, an embodiment of the present application provides an electronic device comprising: at least one processor and a memory; the memory storing computer-executable instructions; the at least one processor executing the computer-executable instructions stored in the memory, so that the at least one processor performs the semantic segmentation method for camouflaged objects based on self-supervised dual modeling as described in the first aspect and various possible designs of the first aspect.

[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method for semantic segmentation of camouflaged objects based on self-supervised dual modeling as described in the first aspect and various possible designs of the first aspect is implemented.

[0035] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the semantic segmentation method of camouflaged objects based on self-supervised dual modeling as described in the first aspect and various possible designs of the first aspect.

[0036] The method, apparatus, device, and storage medium for semantic segmentation of camouflaged objects based on self-supervised dual modeling provided by this application have at least the following beneficial effects:

[0037] The present application provides a method for semantic segmentation of camouflaged objects based on self-supervised dual-construction modeling, which includes the following operations: an image reconstruction phase, in which pixels in the input image are randomly masked, the masked image is extracted through a backbone network to obtain semantic features, and a boundary adaptive perception module is used to aggregate the semantic features to reconstruct the masked pixels; a model refinement phase, in which the weights of the boundary adaptive perception module are randomly initialized while retaining the semantic features extracted by the backbone network, and fully supervised training is performed to obtain preliminary segmentation results; a label reconstruction phase, in which semantic features are extracted from noisy labels, partial labels, and complete labels using the backbone network to perform feature extraction on the preliminary segmentation results, and then a distance adaptive asymmetric module is used to reconstruct the final segmentation results. The present invention achieves accurate segmentation of camouflaged objects.

[0038] During the aforementioned process, when segmenting camouflaged objects in an image, an image is input to the model, which then segments the pixel regions containing the camouflaged objects. Specifically, a Transformer or convolutional backbone network is used because the semantic feature extraction encoded by these networks facilitates the model's exploration of information about camouflaged objects in the image. Self-supervised dual reconstruction learning is employed because it helps the model learn hidden feature information within the image, independent of labels. The self-supervised reconstruction task enables the model to learn important feature representations from the input image even without extensive labeled data.

[0039] Specifically, the boundary-adaptive perception module is adopted because fusing adjacent features using an edge information attention mechanism with weighted dilated convolutions of different coefficients facilitates the full integration of multi-scale information. The boundary-adaptive perception module can enhance the model's ability to perceive object edges by weighting boundary regions, avoiding ignoring subtle differences between camouflaged objects and the background. This weighting mechanism, aided by dilated convolutions, can capture a wider range of contextual information, enabling accurate segmentation of camouflaged objects. The distance-adaptive asymmetric module is subsequently adopted because asymmetric convolutions with weighted coefficients using a pixel-space attention mechanism facilitate the model's full exploration of the structural details of camouflaged objects. Through asymmetric convolutions, the model can more flexibly learn structural information in different directions, enhancing the recognition of camouflaged object details and thus preventing confusion between the details of camouflaged objects and background areas.

[0040] In the task of camouflaged object segmentation, this method achieves accurate segmentation of camouflaged objects by combining self-supervised dual reconstruction learning, boundary adaptive perception module and distance adaptive asymmetric module. It can effectively distinguish camouflaged objects from background areas, and shows high robustness and accuracy, especially in complex camouflage environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] Figure 1 A flowchart of a method for semantic segmentation of camouflaged objects based on self-supervised dual modeling provided in an embodiment of the present application;

[0043] Figure 2 A model structure diagram provided for an embodiment of the present application;

[0044] Figure 3 Schematic diagram of input image provided in the embodiment of the present application;

[0045] Figure 4Schematic diagram of pixel-level labeling provided in an embodiment of the present application;

[0046] Figure 5 This is a rendering of the semantic segmentation method for camouflaged objects based on self-supervised dual modeling provided in an embodiment of the present application;

[0047] Figure 6 This is a rendering of the HitNet provided in the embodiment of this application;

[0048] Figure 7 The renderings of VSCode provided in the embodiments of this application;

[0049] Figure 8 This is the effect diagram of CamoDiffusion provided in the embodiment of this application;

[0050] Figure 9 A schematic diagram of the structure of a camouflaged object semantic segmentation device based on self-supervised dual modeling provided in an embodiment of the present application.

[0051] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0052] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0053] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0054] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned. They should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0055] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0056] An embodiment of the present application provides a semantic segmentation method for camouflaged objects based on self-supervised dual construction modeling, comprising: an image reconstruction stage, in which pixels in an input image are randomly masked, the masked image is extracted through a backbone network to obtain semantic features, and a boundary adaptive perception module is used to aggregate the semantic features to reconstruct the masked pixels; a model refinement stage, in which the weights of the boundary adaptive perception module are randomly initialized while retaining the semantic features extracted by the backbone network, and fully supervised training is performed to obtain preliminary segmentation results; a label reconstruction stage, in which semantic features are extracted from noisy labels, partial labels, and complete labels using a backbone network to perform feature extraction on the preliminary segmentation results, and then reconstructed using a distance adaptive asymmetric module to obtain a final segmentation result.

[0057] Specifically, Figure 1 This is a flowchart of a method for semantic segmentation of camouflaged objects based on self-supervised dual modeling provided in an embodiment of the present application. Figure 1 As shown, the method for semantic segmentation of disguised objects based on self-supervised dual modeling includes steps S1 to S3.

[0058] In the S1 image reconstruction stage, the pixels in the input image are randomly masked, and the masked image is extracted through the backbone network to obtain semantic features. The boundary adaptive perception module is used to aggregate the semantic features to reconstruct the masked pixels.

[0059] In the S2 model refinement stage, the weights of the boundary adaptive perception module are randomly initialized while retaining the semantic features extracted by the backbone network, and fully supervised training is performed to obtain preliminary segmentation results.

[0060] In the S3 label reconstruction stage, a backbone network is used to extract semantic features from noise labels, partial labels, and complete labels to perform feature extraction on the preliminary segmentation results, and then a distance adaptive asymmetric module is used to reconstruct the final segmentation results.

[0061] like Figure 2 As shown, the embodiment of the present application divides the semantic segmentation task of camouflaged objects based on self-supervised dual modeling into the following three processing stages, namely, the image reconstruction stage, the model refinement stage and the label reconstruction stage.

[0062] In the image reconstruction stage, pixels in the input image are randomly masked, the masked image is extracted through the backbone network to obtain semantic features, and the boundary adaptive perception module is used to aggregate the semantic features to reconstruct the masked pixels;

[0063] In the model refinement stage, the weights of the boundary adaptive perception module are randomly initialized while retaining the semantic features extracted by the backbone network, and fully supervised training is performed to obtain preliminary segmentation results;

[0064] In the label reconstruction stage, a backbone network is used to extract semantic features from noise labels, partial labels and complete labels to perform feature extraction on the preliminary segmentation results, and then a distance adaptive asymmetric module is used to reconstruct the final segmentation results.

[0065] As one or more embodiments, during the execution of step S1 above, pixels in the input image are randomly masked, the masked image is extracted through a backbone network to obtain semantic features, and the semantic features are aggregated using a boundary adaptive perception module to reconstruct the masked pixels, specifically including the following steps:

[0066] S101: Using a backbone network to extract semantic features from the randomly masked input image to obtain a first semantic feature map, a second semantic feature map, a third semantic feature map, and a fourth semantic feature map.

[0067] S102: Based on the first semantic feature map, the second semantic feature map, the third semantic feature map and the fourth semantic feature map, the masked part of the image is restored by feeding them into a boundary adaptive perception module for self-supervised reconstruction training.

[0068] The boundary adaptive perception module includes a boundary adaptive perception convolution and a multi-branch channel aggregation structure.

[0069] The boundary adaptive receptive convolution includes: an adaptive void convolution branch, a boundary receptive branch and an attention weighting mechanism.

[0070] The adaptive dilated convolution branch includes: four dilated convolutions whose convolution kernels are 3*3 and whose dilated coefficients are 1, 3, 5, and 7 respectively, connected in sequence.

[0071] The boundary perception branch includes: sequentially connected Gaussian, Sobel and Laplace operations and an average weighted fusion operation.

[0072] The attention weighting mechanism includes: sequentially connected convolution, Relu activation function, and Softmax activation function, and then the output of the boundary adaptive receptive convolution is obtained by element-wise multiplication of the output of the void convolution branch and the output of the attention weighting mechanism.

[0073] The multi-branch channel aggregation structure includes: a channel Split operation, four parallel boundary adaptive perception convolutions and four parallel CBR operations connected in sequence, and then the outputs of the four CBR operations are merged and sent to one CBR operation to obtain the output result of the multi-branch channel aggregation structure.

[0074] The CBR operation includes: sequentially connected convolution, normalization and ReLU activation functions.

[0075] As one or more embodiments, during the execution of step S2, based on the semantic features, the weights of the boundary adaptive perception module are randomly initialized while retaining the semantic features extracted by the backbone network, and fully supervised training is performed to obtain a preliminary segmentation result, specifically including the following steps:

[0076] S201: Randomly initialize the weights of the boundary adaptive perception module.

[0077] S202: Retain the semantic features extracted by the backbone network, add the camouflaged object image labels, perform full-supervised training, and refine the model to obtain preliminary segmentation results.

[0078] As one or more embodiments, during the execution of step S3 above: based on the preliminary segmentation result, using a backbone network to extract semantic features from the noise labels, partial labels, and complete labels to perform feature extraction on the preliminary segmentation result, and then using a distance adaptive asymmetric module to reconstruct the final segmentation result, specifically including:

[0079] S301: randomly masking and randomly adding noise to the preliminary segmentation result to obtain a partial feature map and a noise feature map.

[0080] S302: Inputting the preliminary segmentation result, the partial feature map and the noise feature map into a backbone network to extract semantic features to obtain a fifth semantic feature map, a sixth semantic feature map, a seventh semantic feature map and an eighth semantic feature map.

[0081] S303: Send the fifth semantic feature map, the sixth semantic feature map, the seventh semantic feature map and the eighth semantic feature map to a distance adaptive asymmetric module for reconstruction to obtain a final segmentation result.

[0082] The distance-adaptive asymmetric convolution includes: sequentially connected asymmetric convolution, depth convolution, and parallel SEA attention mechanism and spatial pixel distance attention weighting mechanism, and then element-wise multiplication of the output of the SEA attention mechanism and the output of the spatial pixel distance attention weighting mechanism is performed to obtain the output result of the distance-adaptive asymmetric convolution.

[0083] The Transformer backbone network includes but is not limited to any implementation: a PVTv2 backbone network that implements feature modeling based on a multi-layer multi-head self-attention mechanism and multi-layer linear interpolation sampling;

[0084] The convolutional backbone network includes but is not limited to any implementation method: a Res2Net backbone network that implements feature modeling based on multi-layer split channel convolution and multi-layer channel splicing and residual connection.

[0085] Deep neural network training and parameter initialization: In an embodiment of the present invention, the parameter initialization weight parameters of the backbone network adopt PVTv2 obtained by fully supervised pre-training on the ImageNet dataset.

[0086] Training optimization details: In the embodiment of the present application, the AdamW optimizer is used for training, the number of training rounds is set to 300 rounds, the initial learning rate is set to 0.0001, the batch size is 8, and the weight decay rate is 0.0001.

[0087] like Figure 3 , which is a schematic diagram of the input image of this embodiment, Figure 4 is a pixel-level label, Figure 5 This is a rendering of the method proposed in this embodiment. Figures 6 to 8 Table 1 shows the quantitative comparison results of the method of the embodiment of the present application and the ten most advanced semantic segmentation methods for camouflaged objects based on self-supervised dual modeling.

[0088]

[0089]

[0090] Table 1 Quantitative comparison

[0091] The present application also provides a semantic segmentation device for camouflaged objects based on self-supervised dual modeling. Figure 9 As shown, the camouflaged object semantic segmentation device based on self-supervised dual modeling includes:

[0092] The backbone network module 901 is configured to randomly mask pixels in the image to obtain a mask image, extract the mask image through the backbone network to obtain first multi-scale semantic features, and aggregate the semantic features using a boundary adaptive perception module to reconstruct the masked pixels;

[0093] The boundary adaptive perception module 902 is configured to randomly initialize the weights of the boundary adaptive perception module while retaining the first multi-scale semantic features extracted by the backbone network, and perform fully supervised training to obtain preliminary segmentation results;

[0094] The distance adaptive asymmetric module 903 is configured to use the backbone network to extract second multi-scale semantic features from the noise labels, partial labels and complete labels to perform feature extraction on the preliminary segmentation result, and then use the distance adaptive asymmetric module to reconstruct the final segmentation result.

[0095] In some embodiments, the backbone network module is further configured to:

[0096] Using the backbone network to extract semantic features from the mask image to obtain a first semantic feature map, a second semantic feature map, a third semantic feature map, and a fourth semantic feature map;

[0097] The first semantic feature map, the second semantic feature map, the third semantic feature map and the fourth semantic feature map are sent to a boundary adaptive perception module to restore the masked part of the image for self-supervised reconstruction training.

[0098] In some embodiments, the boundary adaptive perception module is further configured to:

[0099] Based on the boundary adaptive perception module, a random initialization operation is performed, and the first semantic feature map, the second semantic feature map, the third semantic feature map and the fourth semantic feature map are sent to the boundary adaptive perception module for full-supervised training to obtain a preliminary segmentation result.

[0100] In some embodiments, the distance adaptive asymmetric module is further configured to:

[0101] Randomly masking and randomly adding noise to the preliminary segmentation results to obtain a partial feature map and a noise feature map;

[0102] Inputting the preliminary segmentation result, the partial feature map and the noise feature map into a backbone network to extract semantic features to obtain a fifth semantic feature map, a sixth semantic feature map, a seventh semantic feature map and an eighth semantic feature map;

[0103] The fifth semantic feature map, the sixth semantic feature map, the seventh semantic feature map and the eighth semantic feature map are sent to a distance adaptive asymmetric module for reconstruction to obtain a final segmentation result.

[0104] In some embodiments, the backbone network includes a Transformer backbone network or a convolutional backbone network; the Transformer backbone network implements feature modeling based on a PVTv2 backbone network using a multi-layer multi-head self-attention mechanism and multi-layer linear interpolation sampling; the convolutional backbone network implements feature modeling based on a Res2Net backbone network using multi-layer split channel convolution and multi-layer channel splicing and residual connection;

[0105] The boundary adaptive perception module includes a boundary adaptive perception convolution and a multi-branch channel aggregation structure;

[0106] The boundary adaptive receptive convolution includes an adaptive hole convolution branch, a boundary receptive branch and an attention weighting mechanism;

[0107] The adaptive dilated convolution branch includes four dilated convolutions with 3*3 convolution kernels and dilated coefficients of 1, 3, 5, and 7 respectively, which are connected in sequence.

[0108] The boundary perception branch includes: Gaussian, Sobel and Laplace operations connected in sequence and average weighted fusion operation;

[0109] The attention weighting mechanism includes sequentially connected convolution, Relu activation function, and Softmax activation function;

[0110] The output of the boundary adaptive receptive convolution is obtained by element-wise multiplication of the output of the dilated convolution branch and the output of the attention weighting mechanism;

[0111] The multi-branch channel aggregation structure includes a channel Split operation, four parallel boundary adaptive perception convolutions, and four parallel CBR operations connected in sequence. The output result of the multi-branch channel aggregation structure is obtained by merging the outputs of the four CBR operations and feeding them into one CBR operation;

[0112] The CBR operation includes: convolution, normalization and ReLU activation function connected in sequence.

[0113] In some embodiments, the distance adaptive asymmetric module includes a distance adaptive asymmetric convolution and a multi-branch channel aggregation structure;

[0114] The distance-adaptive asymmetric convolution includes sequentially connected asymmetric convolution, depthwise convolution, and parallel SEA attention mechanism and spatial pixel distance attention weighting mechanism. The output result of the distance-adaptive asymmetric convolution is obtained by element-wise multiplication of the output of the SEA attention mechanism and the output of the spatial pixel distance attention weighting mechanism.

[0115] An embodiment of the present application provides an electronic device, which may include a processor and a memory, wherein the processor and the memory can communicate with each other; illustratively, the processor and the memory communicate with each other via a communication bus.

[0116] The processor executes the computer-executable instructions stored in the memory, so that the processor implements the solutions in the above embodiments. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0117] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. System buses can be categorized as address buses, data buses, and control buses. Transceivers enable communication between the database access device and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) or non-volatile memory.

[0118] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0119] An embodiment of the present application also provides a computer-readable storage medium having computer instructions stored therein. When the computer instructions are executed on a computer, the computer is caused to execute the technical solution of the method for semantic segmentation of camouflaged objects based on self-supervised dual modeling in the above embodiment.

[0120] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the semantic segmentation method of camouflaged objects based on self-supervised dual modeling in the above embodiment.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0122] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of these modules may be selected to implement the solution of this embodiment based on actual needs.

[0123] In addition, the functional modules in the various embodiments of the present application may be integrated into a single processing unit, or each module may exist physically separately, or two or more modules may be integrated into a single unit. The above-mentioned modules may be implemented in the form of hardware or hardware plus software functional units.

[0124] The above-mentioned integrated module implemented in the form of a software functional module can be stored in a computer-readable storage medium. The above-mentioned software functional module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0125] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASICs). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0126] The memory may include a high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk.

[0127] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, and control buses.

[0128] The storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0129] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a main control device.

[0130] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A semantic segmentation method for camouflaged objects based on self-supervised dual modeling, characterized by: The method comprises: Randomly masking pixels in the image to obtain a mask image, extracting the mask image through a backbone network to obtain a first multi-scale semantic feature, and aggregating the semantic features using a boundary adaptive perception module to reconstruct the masked pixels; Randomly initialize the weights of the boundary adaptive perception module while retaining the first multi-scale semantic features extracted by the backbone network, and perform fully supervised training to obtain preliminary segmentation results; Using a backbone network to extract second multi-scale semantic features from the noise labels, partial labels and complete labels to perform feature extraction on the preliminary segmentation result, and then using a distance adaptive asymmetric module to reconstruct the final segmentation result; The backbone network includes a Transformer backbone network or a convolutional backbone network; the Transformer backbone network implements feature modeling based on a PVTv2 backbone network with multi-layer multi-head self-attention mechanism and multi-layer linear interpolation sampling; the convolutional backbone network implements feature modeling based on a Res2Net backbone network with multi-layer split channel convolution and multi-layer channel splicing and residual connection; The boundary adaptive perception module includes a boundary adaptive perception convolution and a multi-branch channel aggregation structure; The boundary adaptive receptive convolution includes an adaptive hole convolution branch, a boundary receptive branch and an attention weighting mechanism; The adaptive dilated convolution branch includes four dilated convolutions with 3*3 convolution kernels and dilated coefficients of 1, 3, 5, and 7 respectively, which are connected in sequence. The boundary perception branch includes: Gaussian, Sobel and Laplace operations connected in sequence and average weighted fusion operation; The attention weighting mechanism includes sequentially connected convolution, Relu activation function, and Softmax activation function; The output of the boundary adaptive receptive convolution is obtained by element-wise multiplication of the output of the dilated convolution branch and the output of the attention weighting mechanism; The multi-branch channel aggregation structure includes a channel Split operation, four parallel boundary adaptive perception convolutions, and four parallel CBR operations connected in sequence. The output result of the multi-branch channel aggregation structure is obtained by merging the outputs of the four CBR operations and feeding them into one CBR operation; The CBR operation includes: convolution, normalization and ReLU activation functions connected in sequence; The distance adaptive asymmetric module includes a distance adaptive asymmetric convolution and a multi-branch channel aggregation structure; The distance-adaptive asymmetric convolution includes sequentially connected asymmetric convolution, depthwise convolution, and parallel SEA attention mechanism and spatial pixel distance attention weighting mechanism. The output result of the distance-adaptive asymmetric convolution is obtained by element-wise multiplication of the output of the SEA attention mechanism and the output of the spatial pixel distance attention weighting mechanism.

2. The method for semantic segmentation of camouflaged objects based on self-supervised dual modeling according to claim 1, characterized in that: Extracting the mask image through a backbone network to obtain a first multi-scale semantic feature, and aggregating the semantic features using a boundary adaptive perception module to reconstruct the masked pixels, including: Using the backbone network to extract semantic features from the mask image to obtain a first semantic feature map, a second semantic feature map, a third semantic feature map, and a fourth semantic feature map; The first semantic feature map, the second semantic feature map, the third semantic feature map and the fourth semantic feature map are sent to a boundary adaptive perception module to restore the masked part of the image for self-supervised reconstruction training.

3. The method for semantic segmentation of camouflaged objects based on self-supervised dual modeling according to claim 2, characterized in that: Randomly initialize the weights of the boundary adaptive perception module while retaining the first multi-scale semantic features extracted by the backbone network. Perform fully supervised training to obtain preliminary segmentation results, including: Based on the boundary adaptive perception module, a random initialization operation is performed, and the first semantic feature map, the second semantic feature map, the third semantic feature map and the fourth semantic feature map are sent to the boundary adaptive perception module for full-supervised training to obtain a preliminary segmentation result.

4. The method for semantic segmentation of camouflaged objects based on self-supervised dual modeling according to claim 1, characterized in that: The backbone network is used to extract second multi-scale semantic features from the noise labels, partial labels, and complete labels to perform feature extraction on the preliminary segmentation result, and then a distance adaptive asymmetric module is used to reconstruct the final segmentation result, including: Randomly masking and randomly adding noise to the preliminary segmentation results to obtain a partial feature map and a noise feature map; Inputting the preliminary segmentation result, the partial feature map and the noise feature map into a backbone network to extract semantic features to obtain a fifth semantic feature map, a sixth semantic feature map, a seventh semantic feature map and an eighth semantic feature map; The fifth semantic feature map, the sixth semantic feature map, the seventh semantic feature map and the eighth semantic feature map are sent to a distance adaptive asymmetric module for reconstruction to obtain a final segmentation result.

5. A semantic segmentation device for camouflaged objects based on self-supervised dual modeling, characterized in that: The device comprises: A backbone network module is configured to randomly mask pixels in an image to obtain a mask image, extract the mask image through the backbone network to obtain a first multi-scale semantic feature, and aggregate the semantic features using a boundary adaptive perception module to reconstruct the masked pixels; The boundary adaptive perception module is configured to randomly initialize the weights of the boundary adaptive perception module while retaining the first multi-scale semantic features extracted by the backbone network, and performing fully supervised training to obtain preliminary segmentation results; a distance adaptive asymmetric module configured to use the backbone network to extract second multi-scale semantic features from the noisy labels, partial labels, and complete labels to perform feature extraction on the preliminary segmentation result, and then reconstruct the final segmentation result using the distance adaptive asymmetric module; The backbone network includes a Transformer backbone network or a convolutional backbone network; the Transformer backbone network implements feature modeling based on a PVTv2 backbone network with multi-layer multi-head self-attention mechanism and multi-layer linear interpolation sampling; the convolutional backbone network implements feature modeling based on a Res2Net backbone network with multi-layer split channel convolution and multi-layer channel splicing and residual connection; The boundary adaptive perception module includes a boundary adaptive perception convolution and a multi-branch channel aggregation structure; The boundary adaptive receptive convolution includes an adaptive hole convolution branch, a boundary receptive branch and an attention weighting mechanism; The adaptive dilated convolution branch includes four dilated convolutions with 3*3 convolution kernels and dilated coefficients of 1, 3, 5, and 7 respectively, which are connected in sequence. The boundary perception branch includes: Gaussian, Sobel and Laplace operations connected in sequence and average weighted fusion operation; The attention weighting mechanism includes sequentially connected convolution, Relu activation function, and Softmax activation function; The output of the boundary adaptive receptive convolution is obtained by element-wise multiplication of the output of the dilated convolution branch and the output of the attention weighting mechanism; The multi-branch channel aggregation structure includes a channel Split operation, four parallel boundary adaptive perception convolutions, and four parallel CBR operations connected in sequence. The output result of the multi-branch channel aggregation structure is obtained by merging the outputs of the four CBR operations and feeding them into one CBR operation; The CBR operation includes: convolution, normalization and ReLU activation functions connected in sequence; The distance adaptive asymmetric module includes a distance adaptive asymmetric convolution and a multi-branch channel aggregation structure; The distance-adaptive asymmetric convolution includes sequentially connected asymmetric convolution, depthwise convolution, and parallel SEA attention mechanism and spatial pixel distance attention weighting mechanism. The output result of the distance-adaptive asymmetric convolution is obtained by element-wise multiplication of the output of the SEA attention mechanism and the output of the spatial pixel distance attention weighting mechanism.

6. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the semantic segmentation method for camouflaged objects based on self-supervised dual modeling according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the semantic segmentation method for camouflaged objects based on self-supervised dual modeling according to any one of claims 1 to 4.

8. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method for semantic segmentation of camouflaged objects based on self-supervised dual modeling according to any one of claims 1 to 4.

Images