Semantic segmentation method, semantic segmentation device, equipment and computer storage medium

Abstract

The invention provides a semantic segmentation method, a semantic segmentation device, semantic segmentation equipment and a computer storage medium. The semantic segmentation method comprises the steps that an unlabeled image is input into a semantic segmentation network, a prediction result graph of the unlabeled image is acquired, and the prediction result graph comprises a plurality of pseudo-label areas; inputting the label-free image into a segmentation cutting model, and obtaining a detection result image of the label-free image; determining a false label proportion of each false label area in an image area corresponding to the detection result image; deleting the pseudo label areas with the pseudo label proportions smaller than a preset threshold value from the prediction result image to obtain a screened label image; wherein the screening label image is used for training the semantic segmentation network. Through the semantic segmentation method, unreasonable areas in the prediction result graph are filtered by segmenting the contradiction between the detection result of all the models and the pseudo labels, so that the pseudo labels are filtered, the credibility of the pseudo labels can be effectively enhanced, and the overall prediction precision of the models is improved.

Description

Technical Field

[0001] The present application relates to the technical field of semantic segmentation, and in particular to a semantic segmentation method, a semantic segmentation apparatus, a semantic segmentation device, and a computer storage medium. Background Art

[0002] With the continuous development of aerospace technology, the observation capabilities of high-resolution remote sensing satellites have been continuously improved. How to efficiently use these data to guide industrial and agricultural production has become an important problem. Semantic segmentation of remote sensing images is one of the important tasks. Remote sensing semantic segmentation refers to the division of remote sensing images into different landforms such as woodland, grassland, building land, and water. In the context of global digitalization and intelligence, the research on remote sensing image semantic segmentation algorithms has very important research significance and application value.

[0003] Traditional remote sensing semantic segmentation technology relies on Bayesian classification, clustering and other technologies, which cannot adapt to diverse practical needs. The development of deep learning technology has brought new opportunities for remote sensing image semantic segmentation, but the training process of deep learning requires a large amount of labeled data, and it is expensive to obtain pixel-level semantic labels. This brings new challenges to the semantic segmentation task. Semi-supervised learning can effectively solve this problem by using a small amount of labeled data and a large amount of unlabeled data to jointly train the model.

[0004] However, when using unlabeled data to train the semantic segmentation network, a large number of pseudo labels will be generated, and unreasonable pseudo labels will cause errors in the training of the semantic segmentation network, affecting the overall prediction accuracy of the model.

[0005] ‌ Summary of the invention In order to solve the above technical problems, the present application proposes a semantic segmentation method, a semantic segmentation apparatus, a semantic segmentation device and a computer storage medium.

[0006] In order to solve the above technical problems, the present application proposes a semantic segmentation method, which includes: Inputting the unlabeled image into a semantic segmentation network to obtain a prediction result map of the unlabeled image, wherein the prediction result map includes a plurality of pseudo-label regions; Input the unlabeled image into a segmentation model to obtain a detection result image of the unlabeled image; Determine the pseudo-label ratio of each pseudo-label area in the image area corresponding to the detection result image; Deleting the pseudo-label area whose pseudo-label ratio is less than a preset threshold from the prediction result image to obtain a filtered label image; Wherein, the screened label images are used to train the semantic segmentation network.

[0007] Wherein, the semantic segmentation method further includes: The pseudo-label area whose pseudo-label proportion is greater than a preset threshold is covered with the noise area in the prediction result image according to the pseudo-label excess area in the detection result image.

[0008] Wherein, the semantic segmentation method further includes: Get a random area in the original image; The random area is filled with a target image of a specified type to generate a labeled image or an unlabeled image for training the semantic segmentation network.

[0009] Before filling the random area with a target image of a specified type, the semantic segmentation method further includes: The prompt word corresponding to the specified type is input into the segmentation model to generate the target image.

[0010] Wherein, obtaining the prediction result map of the unlabeled image includes: Extracting image features and semantic features of the unlabeled image through a transformer model of the semantic segmentation network; According to the fusion features of the image features and the semantic features, a prediction result map of the unlabeled image is output.

[0011] The converter model includes an encoder, which includes a multi-scale linear attention layer, a first normalized splicing layer, a first feedforward layer, and a second normalized splicing layer connected in sequence; Among them, the multi-scale linear attention layer includes a second-scale attention layer and several first-scale attention layers connected to the second-scale attention layer, the first-scale attention layer is used to extract the self-attention features of each image block of the unlabeled image division, and the second-scale attention layer splices the self-attention features of all image blocks and extracts the mutual attention features between all image blocks.

[0012] The converter model also includes a decoder, which includes a first multi-scale linear attention layer, a third normalized splicing layer, a second multi-scale linear attention layer, a fourth normalized splicing layer, a second feedforward layer and a fifth normalized splicing layer connected in sequence; wherein the image features output by the second normalized splicing layer of the encoder are connected to the input side of the second multi-scale linear attention layer.

[0013] In order to solve the above technical problems, the present application also proposes a semantic segmentation device, which includes: a semantic segmentation module, a segmentation module, a region confirmation module and a label screening module; wherein, The semantic segmentation module is used to input the unlabeled image into the semantic segmentation network to obtain a prediction result map of the unlabeled image, wherein the prediction result map includes a plurality of pseudo-label regions; The segmentation module is used to input the unlabeled image into the segmentation model to obtain a detection result image of the unlabeled image; The region confirmation module is used to determine the pseudo-label proportion of each pseudo-label region in the image region corresponding to the detection result image; The label screening module is used to delete the pseudo-label area whose pseudo-label ratio is less than a preset threshold from the prediction result image to obtain a screened label image; Wherein, the screened label images are used to train the semantic segmentation network.

[0014] To solve the above technical problems, the present application also proposes a semantic segmentation device, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the semantic segmentation method as described above.

[0015] In order to solve the above technical problems, the present application also proposes a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the above semantic segmentation method.

[0016] Compared with the prior art, the beneficial effects of the present application are as follows: the semantic segmentation device inputs the unlabeled image into the semantic segmentation network to obtain the prediction result map of the unlabeled image, wherein the prediction result map includes several pseudo-label areas; the unlabeled image is input into the segmentation everything model to obtain the detection result map of the unlabeled image; the pseudo-label proportion of each pseudo-label area in the corresponding image area of ​​the detection result map is determined; the pseudo-label areas whose pseudo-label proportion is less than a preset threshold are deleted from the prediction result map to obtain a filtered label image; wherein the filtered label image is used to train the semantic segmentation network. Through the above-mentioned semantic segmentation method, the contradiction between the detection result of the segmentation everything model and the pseudo-label is utilized to filter out the unreasonable areas in the prediction result map, thereby filtering the pseudo-labels, which can effectively enhance the credibility of the pseudo-labels and thus improve the overall prediction accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 It is a flowchart of an embodiment of a semantic segmentation method provided by the present application; Figure 2 It is a model flow chart of an embodiment of a semantic segmentation network provided by the present application; Figure 3 is a flowchart of another embodiment of the semantic segmentation method provided by the present application; Figure 4 It is a flowchart of another embodiment of the semantic segmentation method provided by the present application; Figure 5 It is a schematic diagram of the framework of an embodiment of a semantic segmentation network provided by the present application; Figure 6 It is a structural diagram of an embodiment of a semantic segmentation device provided by the present application; Figure 7 is a structural diagram of an embodiment of a semantic segmentation device provided by the present application; Figure 8 It is a structural diagram of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] The problem that this patent proposal aims to solve is the problem of insufficient labels in the semantic segmentation of remote sensing images. The model can be trained using a small amount of labeled data and a large amount of unlabeled data. It can achieve relatively good results on some typical remote sensing data, such as the ISPRS (International Society for Photogrammetry and Remote) remote sensing dataset, the LOVEDA remote sensing dataset, and the MINIFRANCE remote sensing dataset. Taking the ISPRS dataset as an example, ISPRS provides two of the most advanced airborne image datasets for urban classification and three-dimensional building reconstruction test projects. The dataset uses a digital surface model (DSM) generated by high-resolution orthogonal photos and corresponding dense image matching technology.

[0021] Both datasets cover urban scenes. Vaihingen is a relatively small village with many individual buildings and small multi-storey buildings; Postdam is a typical historical city with large building complexes, narrow streets and dense settlement structures. Each dataset has been manually classified into the 6 most common land cover categories.

[0022] Based on the above-mentioned semi-supervised training scheme and traditional pseudo-labeling method, this application designs a pseudo-label filter in combination with the SAM (SegmentAnything Model) model, and uses the contradiction between the SAM prediction results and the pseudo-labels to filter out unreasonable pseudo-labels and improve the overall accuracy of pseudo-labels.

[0023] Please refer to Figure 1 and Figure 2 , Figure 1 is a flowchart of an embodiment of a semantic segmentation method provided by the present application. Figure 2 It is a model flow chart of an embodiment of a semantic segmentation network provided in this application.

[0024] The semantic segmentation method of the present application is applied to a semantic segmentation device, wherein the semantic segmentation device of the present application can be a server, a terminal device, or a system composed of a server and a terminal device. Accordingly, the various parts of the semantic segmentation device, such as various units, sub-units, modules, and sub-modules, can all be set in the server, can all be set in the terminal device, or can be set in the server and the terminal device respectively.

[0025] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here.

[0026] Figure 2 A semi-supervised remote sensing semantic segmentation network based on pseudo label screening is proposed, which uses a small amount of labeled data and a large amount of unlabeled data as model input. For unlabeled data, this application uses the model to generate a large number of pseudo labels for training.

[0027] In order to reduce the misleading of the model by erroneous pseudo-labels, this application introduced the SAM pseudo-label screening mechanism based on the traditional pseudo-labeling method, which effectively improved the generalization ability of the model, reduced the cost of data annotation, and realized pixel-level semantic segmentation of remote sensing images.

[0028] like Figure 1 As shown, the specific steps are as follows: Step S11: input the unlabeled image into the semantic segmentation network to obtain a prediction result map of the unlabeled image, wherein the prediction result map includes a plurality of pseudo-label regions.

[0029] In the embodiment of the present application, the semantic segmentation device inputs unlabeled data, such as unlabeled images, into Figure 2 The semantic segmentation network shown in FIG. 1 outputs a prediction result map of the unlabeled image. Each pixel pseudo-label in the prediction result map indicates that the pixel point in the corresponding unlabeled image belongs to a certain type of land object.

[0030] Among them, there will inevitably be unreasonable pseudo labels in the pseudo labels in the prediction result graph. The SAM pseudo label screening mechanism provided in this application is to screen out these unreasonable pseudo labels, thereby improving the accuracy of the prediction result graph of the unlabeled image.

[0031] It should be noted that the pseudo-label area is a rectangular area of ​​the image formed by connecting adjacent pixels of the same pseudo-label, wherein the rectangular area of ​​the image is not necessarily full of pixels of the pseudo-label, and there may be other pixels introduced to form the rectangular area of ​​the image.

[0032] Step S12: Input the unlabeled image into the segmentation model to obtain a detection result map of the unlabeled image.

[0033] In the embodiment of the present application, the semantic segmentation device inputs the unlabeled image into Figure 2The SAM model shown outputs a detection result map of the unlabeled image. Similarly, each pixel label in the detection result map indicates that the pixel point in the corresponding unlabeled image belongs to a certain type of object. It should be noted that the SAM model is identified as a trained segmentation model in this application, and the detection result map it outputs can be considered as the correct detection result of the unlabeled image.

[0034] Furthermore, since the SAM model can theoretically segment targets of all types of objects, in order to further improve the screening effect of the detection result image on the prediction result image, the semantic segmentation device of the present application can input corresponding prompt words to the SAM model according to the object type of the unlabeled image, and directionally detect various objects existing in the unlabeled image.

[0035] Step S13: Determine the pseudo-label ratio of each pseudo-label region in the image region corresponding to the detection result image.

[0036] In an embodiment of the present application, the semantic segmentation device compares the prediction result map and the detection result map of the unlabeled image to determine the pseudo-label ratio of each pseudo-label area in the prediction result map to the corresponding image area in the detection result map.

[0037] For example, there is a pseudo-label area with a forest type in the prediction result image, that is, the pixels in the area are predicted to be forest pixels. Further, the semantic segmentation device obtains all pixels in the image area of ​​the pseudo-label area corresponding to the forest in the detection result image, and then counts the proportion of these pixels in the detection result image that belong to the forest label.

[0038] Step S14: Delete the pseudo-label areas whose pseudo-label proportion is less than a preset threshold from the prediction result image to obtain a filtered label image.

[0039] In an embodiment of the present application, the semantic segmentation device deletes the pseudo-label area in the detection result image where the pseudo-label ratio is less than a preset threshold from the prediction result image, and considers that the prediction result of this pseudo-label area is very contradictory and the pseudo-label of this area should be deleted.

[0040] Furthermore, the semantic segmentation device can also detect the pseudo-label area in the result map where the pseudo-label ratio is greater than a preset threshold and cover the noise area in the prediction result map in the prediction result map. The noise area can be determined by the additional area of ​​the area composed of the same label as the pseudo-label in the detection result map relative to the pseudo-label area in the prediction result map.

[0041] It should be noted that the filtered label images after proposing unreasonable pseudo labels can be used as labeled images to continue training the semantic segmentation network.

[0042] In a specific embodiment, for unlabeled data, the semantic segmentation device first obtains a pseudo label p1 through a semantic segmentation network, and then uses a target detection method in SAM, inputs the type of ground objects as a prompt word, and detects various ground objects in a targeted manner to obtain a detection result graph q1. The semantic segmentation device compares the detection result graph q1 with the unfiltered pseudo label p1 as a screening basis for the pseudo label filter.

[0043] For example, if the area p1(i) in p1 belongs to forest land, but the forest land ratio in the area q1(i) detected by SAM is less than the threshold r, then it is considered that the prediction result of this area is very contradictory and should be deleted. On the contrary, if the forest land ratio in the prediction result q1(i) is higher than the threshold r, then the SAM result is used to absorb the unreasonable noise area.

[0044] In the present application, the semantic segmentation device inputs an unlabeled image into a semantic segmentation network to obtain a prediction result map of the unlabeled image, wherein the prediction result map includes several pseudo-label regions; inputs the unlabeled image into a segmentation model to obtain a detection result map of the unlabeled image; determines the pseudo-label ratio of each pseudo-label region in the corresponding image region of the detection result map; deletes the pseudo-label regions whose pseudo-label ratio is less than a preset threshold from the prediction result map to obtain a filtered label image; wherein the filtered label image is used to train the semantic segmentation network. Through the above-mentioned semantic segmentation method, the contradiction between the detection results of the segmentation model and the pseudo-label is utilized to filter out unreasonable areas in the prediction result map, thereby filtering the pseudo-labels, which can effectively enhance the credibility of the pseudo-labels and thus improve the overall prediction accuracy of the model.

[0045] Furthermore, in order to solve the problem of insufficient number of data sets in the prior art, the present application also provides another semantic segmentation method, which uses a data expansion module to expand the training data set.

[0046] Semi-supervised learning often falls into overfitting due to insufficient data diversity. Traditional operations such as translation and flipping are difficult to provide diverse features. This application uses the SAM-inpainting model for data enhancement, thereby improving the final segmentation accuracy of the model.

[0047] Please refer to Figure 3 , Figure 3 It is a flowchart of another embodiment of the semantic segmentation method provided by the present application.

[0048] like Figure 3 As shown, the specific steps are as follows: Step S21: Acquire a random area in the original image.

[0049] In the embodiment of the present application, the semantic segmentation device randomly selects an area in the original image and uses the area as the coverage area, that is, the area to be enhanced.

[0050] Furthermore, the semantic segmentation device may also block the area to be enhanced, and may cover the area with a black rectangle or set the pixel value to 0 to simulate the missing area of ​​the object.

[0051] Step S22: Fill the random area with the target image of the specified type to generate a labeled image or an unlabeled image for training the semantic segmentation network.

[0052] In the embodiment of the present application, the semantic segmentation device fills the random area determined in step S21 with the target image of the specified type, thereby generating a new training image, which can be a labeled image or an unlabeled image. The semantic segmentation device obtains the filled image, that is, the enhanced data, and merges the enhanced data with the original data into a new data set.

[0053] Furthermore, the semantic segmentation device may also input the image preprocessed in step S21 and the prompt word of the specified type into the SAM-inpainting model to perform a filling operation to fill the area to be enhanced with the specified target object type.

[0054] In addition, in order to improve Figure 2 The segmentation efficiency of the semantic segmentation network shown in the figure is improved. This application improves the Attention mechanism based on Transformer and designs a hierarchical attention mechanism. First, attention calculation is performed within each window, and then the attention mechanism is applied again in a larger range (such as a group of multiple adjacent windows). The scope of attention is gradually expanded, thereby improving the perception range of the model.

[0055] Please refer to Figure 4 , Figure 4 It is a flowchart of another embodiment of the semantic segmentation method provided by the present application.

[0056] like Figure 4 As shown, the specific steps are as follows: Step S31: Extract image features and semantic features of the unlabeled image through the transformer model of the semantic segmentation network.

[0057] Step S32: outputting a prediction result map of the unlabeled image according to the fusion features of the image features and the semantic features.

[0058] In the embodiment of the present application, the semantic segmentation device uses the Transformer model to construct a semantic segmentation network ML-Transformer and designs a hierarchical attention mechanism. Figure 5, Figure 5 It is a schematic diagram of the framework of an embodiment of a semantic segmentation network provided in this application.

[0059] like Figure 5 As shown, the transformer model (ML-Transformer) includes an encoder, which includes a multi-scale linear attention layer (ML-Attention), a first normalization concatenation layer (Add&Norm), a first feedforward layer (FeedForward), and a second normalization concatenation layer (Add&Norm) connected in sequence.

[0060] The multi-scale linear attention layer (ML-Attention) includes a second-scale attention layer (Attention) and a plurality of first-scale attention layers (SA) connected to the second-scale attention layer. The first-scale attention layer is used to extract the self-attention features of each image block of the unlabeled image division. The second-scale attention layer concatenates the self-attention features of all image blocks and extracts the mutual attention features between all image blocks.

[0061] The transformer model (ML-Transformer) also includes a decoder, which includes a first multi-scale linear attention layer (ML-Attention), a third normalized splicing layer (Add&Norm), a second multi-scale linear attention layer (ML-Attention), a fourth normalized splicing layer (Add&Norm), a second feed forward layer (Feed Forward) and a fifth normalized splicing layer (Add&Norm) connected in sequence; wherein the image features output by the second normalized splicing layer (Add&Norm) of the encoder are connected to the input side of the second multi-scale linear attention layer (ML-Attention).

[0062] First, the attention calculation is performed inside each window, and then the attention mechanism is applied again in a larger scope (such as a group of multiple adjacent windows), gradually expanding the scope of attention. The semantic segmentation network adopts an encoder-decoder structure as a whole. First, the basic features of the image are extracted, and then the Transformer is used as a decoder to analyze its semantic features. The specific operations of the Transformer are as follows: Input encoding: The input image is divided into multiple image blocks and the pixel values ​​of each block are converted into embedded features. These features will be passed to the Transformer model through the embedding layer.

[0063] Self-attention mechanism: In the Transformer model, each block interacts with other blocks to obtain richer feature representations by calculating the relationship between blocks. Through the self-attention mechanism, the model can automatically learn the important connections between each block and other blocks.

[0064] Encoder and Decoder: Transformer models usually consist of an encoder and a decoder. The encoder is responsible for learning the representation of image features, while the decoder uses the encoder's output and combines it with contextual information to generate the final segmentation result.

[0065] Multi-scale fusion: Transformer network can also improve semantic segmentation performance by utilizing multi-scale information. By introducing image patches at multiple different scales and fusing features at different scales, it can provide a more global and detailed feature representation.

[0066] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0067] The semantic segmentation method of the present application creates a pseudo-label filter based on SAM, which can filter out erroneous areas by utilizing the contradictions predicted by the SAM model and the Transformer model in the present application, and can effectively enhance the credibility of the pseudo-labels, thereby improving the overall prediction accuracy of the model.

[0068] The semantic segmentation method of the present application performs data enhancement through SAM-inpainting, which can provide new generative features compared to traditional flipping and translation, thereby improving the generalization ability of the model.

[0069] The semantic segmentation method of this application designs a hierarchical Attention mechanism based on traditional Attention, applies the attention mechanism in a larger range (such as a group of multiple adjacent windows) to expand the perception range of attention.

[0070] To implement the above semantic segmentation method, this application also proposes a semantic segmentation device, please refer to Figure 6 , Figure 6 It is a structural diagram of an embodiment of a semantic segmentation device provided in the present application.

[0071] The semantic segmentation device 500 of this embodiment includes: a semantic segmentation module 51 , a segmentation module 52 , a region confirmation module 53 and a label screening module 54 .

[0072] The semantic segmentation module 51 is used to input the unlabeled image into the semantic segmentation network to obtain a prediction result map of the unlabeled image, wherein the prediction result map includes a plurality of pseudo-label regions.

[0073] The segmentation module 52 is used to input the unlabeled image into a segmentation model to obtain a detection result image of the unlabeled image.

[0074] The region confirmation module 53 is used to determine the pseudo-label ratio of each pseudo-label region in the image region corresponding to the detection result image.

[0075] The label filtering module 54 is used to delete the pseudo-label areas whose pseudo-label proportion is less than a preset threshold from the prediction result image to obtain a filtered label image.

[0076] Wherein, the screened label images are used to train the semantic segmentation network.

[0077] To implement the above semantic segmentation method, this application also proposes a semantic segmentation device, please refer to Figure 7 , Figure 7 It is a structural diagram of an embodiment of a semantic segmentation device provided in this application.

[0078] The semantic segmentation device 400 of this embodiment includes a processor 41 , a memory 42 , an input / output device 43 , and a bus 44 .

[0079] The processor 41 , the memory 42 , and the input / output device 43 are respectively connected to the bus 44 . The memory 42 stores program data, and the processor 41 is used to execute the program data to implement the semantic segmentation method described in the above embodiment.

[0080] In the embodiment of the present application, the processor 41 may also be referred to as a CPU (Central Processing Unit). The processor 41 may be an integrated circuit chip having the ability to process signals. The processor 41 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 41 may also be any conventional processor, etc.

[0081] This application also provides a computer storage medium, please continue to refer to Figure 8, Figure 8 It is a structural diagram of an embodiment of a computer storage medium provided in the present application. The computer storage medium 600 stores a computer program 61. When the computer program 61 is executed by a processor, it is used to implement the semantic segmentation method of the above embodiment.

[0082] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0083] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A semantic segmentation method, characterized in that: The semantic segmentation method comprises: Inputting the unlabeled image into a semantic segmentation network to obtain a prediction result map of the unlabeled image, wherein the prediction result map includes a plurality of pseudo-label regions; Input the unlabeled image into a segmentation model to obtain a detection result image of the unlabeled image; Determine the pseudo-label ratio of each pseudo-label area in the image area corresponding to the detection result image; Deleting the pseudo-label area whose pseudo-label ratio is less than a preset threshold from the prediction result image to obtain a filtered label image; Wherein, the screened label images are used to train the semantic segmentation network.

2. The semantic segmentation method according to claim 1, characterized in that: The semantic segmentation method further includes: The pseudo-label area whose pseudo-label proportion is greater than a preset threshold is covered with the noise area in the prediction result image according to the pseudo-label excess area in the detection result image.

3. The semantic segmentation method according to claim 1, characterized in that: The semantic segmentation method further includes: Get a random area in the original image; The random area is filled with a target image of a specified type to generate a labeled image or an unlabeled image for training the semantic segmentation network.

4. The semantic segmentation method according to claim 3, characterized in that: Before filling the random area with a target image of a specified type, the semantic segmentation method further includes: The prompt word corresponding to the specified type is input into the segmentation model to generate the target image.

5. The semantic segmentation method according to claim 1, characterized in that: The obtaining of the prediction result graph of the unlabeled image comprises: Extracting image features and semantic features of the unlabeled image through a transformer model of the semantic segmentation network; According to the fusion features of the image features and the semantic features, a prediction result map of the unlabeled image is output.

6. The semantic segmentation method according to claim 5, characterized in that: The converter model includes an encoder, wherein the encoder includes a multi-scale linear attention layer, a first normalized splicing layer, a first feed-forward layer, and a second normalized splicing layer connected in sequence; Among them, the multi-scale linear attention layer includes a second-scale attention layer and several first-scale attention layers connected to the second-scale attention layer, the first-scale attention layer is used to extract the self-attention features of each image block of the unlabeled image division, and the second-scale attention layer splices the self-attention features of all image blocks and extracts the mutual attention features between all image blocks.

7. The semantic segmentation method according to claim 6, characterized in that: The converter model also includes a decoder, which includes a first multi-scale linear attention layer, a third normalized splicing layer, a second multi-scale linear attention layer, a fourth normalized splicing layer, a second feedforward layer and a fifth normalized splicing layer connected in sequence; wherein the image features output by the second normalized splicing layer of the encoder are connected to the input side of the second multi-scale linear attention layer.

8. A semantic segmentation device, characterized in that: The semantic segmentation device comprises: a semantic segmentation module, a segmentation module, a region confirmation module and a label screening module; wherein, The semantic segmentation module is used to input the unlabeled image into the semantic segmentation network to obtain a prediction result map of the unlabeled image, wherein the prediction result map includes a plurality of pseudo-label regions; The segmentation module is used to input the unlabeled image into the segmentation model to obtain a detection result image of the unlabeled image; The region confirmation module is used to determine the pseudo-label ratio of each pseudo-label region in the image region corresponding to the detection result image; The label screening module is used to delete the pseudo-label area whose pseudo-label ratio is less than a preset threshold from the prediction result image to obtain a screened label image; Wherein, the screened label images are used to train the semantic segmentation network.

9. A semantic segmentation device, characterized in that: The semantic segmentation device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the semantic segmentation method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the semantic segmentation method according to any one of claims 1 to 7.

Images