Unsupervised image semantic segmentation method, device, terminal and storage medium

Abstract

The present invention relates to the field of image processing technology. The present invention discloses an unsupervised image semantic segmentation method, device, terminal and storage medium, which can improve the accuracy of image segmentation. The unsupervised image semantic segmentation method comprises acquiring a plurality of original images; inputting all the original images into a semantic attention network, the semantic attention network matches and fuses the pixel-level representation generated in the first latent space and the semantic-level representation generated in the second latent space of each original image, and outputs a segmentation result corresponding to each original image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to an unsupervised image semantic segmentation method, device, terminal and storage medium. Background Art

[0002] Existing unsupervised image semantic segmentation methods usually use mutual information to assist unsupervised image semantic segmentation models in learning meaningful image representations. Since the original intention of mutual information is to maximize the distance between different image representations and the similarity between different enhanced versions of the same image, this unsupervised image semantic segmentation model based on mutual information pays too much attention to image-level representations or classification-friendly representations during training, and ignores the fine-grained information and structured semantic layout between pixels, making it difficult for the model to accurately distinguish the categories of adjacent pixels during segmentation, resulting in representation collapse, which leads to inaccurate image segmentation of the model. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an unsupervised image semantic segmentation method, device, terminal and storage medium, which can improve the accuracy of image segmentation. The embodiments of the present application are mainly implemented through the following technical solutions:

[0004] A first aspect of an embodiment of the present application provides an unsupervised image semantic segmentation method, comprising:

[0005] Acquire multiple original images;

[0006] All the original images are input into a semantic attention network, which matches and fuses the pixel-level representation generated in the first latent space and the semantic-level representation generated in the second latent space for each original image, and outputs a segmentation result corresponding to each original image.

[0007] According to an embodiment of the present application, the step of inputting all the original images into a semantic attention network, the semantic attention network matching and fusing the pixel-level representation generated in the first latent space and the semantic-level representation generated in the second latent space for each original image, and outputting a segmentation result corresponding to each original image includes:

[0008] Inputting all the original images into the pixel-level encoder of the semantic attention network, the pixel-level encoder processes the pixels of each original image and maps them to the first latent space to obtain a pixel-level representation corresponding to each original image;

[0009] Cut each original image into a plurality of first images to be processed, and linearly map each first image to be processed to the second latent space through a convolution operation to obtain a plurality of second images to be processed corresponding to each original image;

[0010] Inputting a plurality of second images to be processed corresponding to each original image into a semantic generator of the semantic attention network, wherein the semantic generator performs semantic processing on all the second images to be processed to obtain a semantic level representation corresponding to each original image;

[0011] Inputting the pixel-level representation and the semantic-level representation corresponding to each of the original images into the semantic attention module of the semantic attention network for matching processing to obtain an aligned representation corresponding to the original image;

[0012] The pixel-level representation and the aligned representation corresponding to each of the original images are input into the consistency retainer of the semantic attention network for fusion processing to obtain a segmentation result corresponding to the original image.

[0013] According to an embodiment of the present application, the step of inputting all the original images into the pixel-level encoder of the semantic attention network, wherein the pixel-level encoder processes the pixels of each original image and maps them to the first latent space, and obtaining a pixel-level representation corresponding to each original image includes:

[0014] Inputting all the original images into a pixel sensor of the pixel-level encoder, wherein the pixel sensor generates a pixel-by-pixel region representation corresponding to each pixel in each original image based on a neighboring receptive field of each pixel in each original image;

[0015] All pixel-by-pixel region representations are input to a pixel encoder of the pixel-level encoder, and the pixel encoder maps all the pixel-by-pixel region representations to the first latent space to obtain the pixel-level representation corresponding to each of the original images.

[0016] According to an embodiment of the present application, in the step of inputting a plurality of second images to be processed corresponding to each original image into the semantic generator of the semantic attention network, the semantic generator performs semantic processing on all the second images to be processed, and obtaining a semantic level representation corresponding to each original image includes:

[0017] The encoder based on the transformer in the semantic generator extracts semantic information from each block of the second image to be processed, and generates batch-level semantic information corresponding to the original image;

[0018] The multi-head generator of the semantic generator generates classification information corresponding to the original image based on the batch-level semantic information;

[0019] The information block matcher of the semantic generator generates the semantic level representation based on the classification information and the pixel level representation.

[0020] According to one embodiment of the present application, the calculation formula of the semantic attention module is defined as:

[0021] ;

[0022] in, For the pixel-level characterization, for The transpose of is the scaling factor, is the sum of the semantic-level representations.

[0023] According to one embodiment of the present application, the training method of the semantic attention network includes:

[0024] Obtain a training data set and a real label set, wherein each training data in the training data set has a one-to-one correspondence with a real label in the real label set;

[0025] Input the target training data into an initial semantic attention network and output a predicted segmentation result, wherein the initial semantic attention network includes an initial pixel-level encoder, an initial semantic generator, an initial consistency retainer and an initial semantic attention module;

[0026] Calculating a loss function based on the predicted segmentation result and a true label corresponding to the target training data;

[0027] Adjusting the network parameters of the initial semantic attention network based on the loss function to form the semantic attention network;

[0028] The target training data belongs to any training data in the training data set.

[0029] According to one embodiment of the present application, the calculation formula of the loss function is:

[0030] ;

[0031] in, represents the total number of samples, Representative The true labels, Representative Predicted segmentation results.

[0032] A second aspect of the embodiments of the present application provides an unsupervised image semantic segmentation device, comprising:

[0033] An acquisition module, used for acquiring multiple original images;

[0034] The segmentation result output module is used to input all the original images into the semantic attention network, which matches and fuses the pixel-level representation generated by each original image in the first latent space and the semantic-level representation generated in the second latent space, and outputs the segmentation result corresponding to each original image.

[0035] According to a third aspect of an embodiment of the present application, a terminal device is provided, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory, and executing the steps of the unsupervised image semantic segmentation method described in the first aspect of the embodiment of the present application.

[0036] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program enables a computer to execute the steps of the unsupervised image semantic segmentation method described in the first aspect of the embodiment of the present application.

[0037] The beneficial effects of the embodiments of the present application include:

[0038] The present application designs a semantic attention network, which matches and fuses the pixel-level representation generated in the first latent space and the semantic-level representation generated in the second latent space for each of the multiple original images, and outputs a segmentation result corresponding to each original image. The present application fuses the pixel-level representation and the semantic-level representation, which not only enables the semantic attention network to learn a better overall image representation (i.e., image-level representation), but also pays attention to the fine-grained information between pixels and the attention to the structured semantic layout. Therefore, compared with the prior art, the embodiments of the present application can accurately distinguish the categories of adjacent pixels during semantic segmentation, thereby effectively avoiding the occurrence of representation collapse and improving the accuracy of image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 This is a comparison chart of unsupervised image semantic segmentation methods;

[0041] Figure 2A flowchart of the unsupervised image semantic segmentation method of the present invention in some embodiments;

[0042] Figure 3 It is a flowchart of the unsupervised image semantic segmentation method of the present invention in some embodiments;

[0043] Figure 4 It is a structural block diagram of the unsupervised image semantic segmentation device of the present invention in some embodiments;

[0044] Figure 5 It is a principle block diagram of the terminal device of the present invention in some embodiments. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0046] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0047] The terms "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0048] The terms "comprises," "includes," or any other variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or elements is not necessarily limited to those steps or elements explicitly listed but may include other steps or elements not explicitly listed or inherent to such process, method, product, or apparatus.

[0049] The term "fine-grained information" in the field of image processing technology refers to the details of the image, such as color, texture, edges, and corners. "Fine-grained information between pixels" refers to the subtle differences in color, brightness, and texture between adjacent pixels. In image segmentation tasks, fine-grained information helps to accurately distinguish different objects or regions in an image. For example, Figure 1 The picture with number 1 contains sheep, white clouds, sky and mountains. In the image analysis of this picture, fine-grained information can identify the area where the sheep are located in the picture.

[0050] The term "structured semantic layout", in the field of image processing technology, refers to extracting key information (such as objects, attributes, relationships, etc.) from an image and representing this information in a structured form.

[0051] Unless otherwise defined, all technical and scientific terms used in the specification of this application have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in the specification of this application includes any and all combinations of one or more related listed items.

[0052] Since unsupervised image semantic segmentation models lack direct supervision information, they must rely on specific constraints to achieve effective semantic learning. Based on this, unsupervised image semantic segmentation methods need to meet two core requirements: (1) the method should be able to establish the connection between pixels and semantics; (2) it should have a specific supervision mechanism to promote the effect of semantic learning.

[0053] Early unsupervised image semantic segmentation methods, such as Figure 1 The method flow shown in the label 1 is mainly to use deep neural networks to learn feature representations in images, and use these features to perform pixel-level classification and clustering to achieve semantic segmentation. This method usually clusters pixels of low-level features into a few semantic groups, often accompanied by some artificially designed prior knowledge, such as different data enhancement techniques. The supervision of the deep neural network in this method comes from the pseudo-labels generated by the clustering algorithm. Its basic assumption is that in the latent space, pixels within the same semantic object should have relatively similar low-level image representations. However, this low-level image representation often cannot fully represent high-level semantic information. Therefore, this clustering-based method cannot effectively strengthen the low-level representation to express semantic information without considering high-level semantic information, and thus fails to meet the second core requirement mentioned above.

[0054] The limitation of clustering-based methods is that they ignore high-level semantics and over-rely on low-level features, which leads to insufficient accuracy and generalization ability of semantic segmentation. In order to solve the problems of clustering-based methods, an unsupervised image semantic segmentation method based on mutual information has emerged.

[0055] The unsupervised image semantic segmentation method based on mutual information uses mutual information as a supervisory signal to generate high-level semantic information. For details, refer to Figure 1 The method flow shown in the label 2 is to obtain the segmentation result by classifying the enhanced image through a deep neural network, and then introduce the maximization of mutual information as an optimization goal or constraint to improve the consistency between the segmentation result and the true label. That is, the method is trained by increasing the distance between different image representations while enhancing the similarity between the representations of different enhanced versions of the same image. However, this design makes the unsupervised image semantic segmentation model focus on image-level representations or classification-friendly representations, ignoring or missing the fine-grained information and structured semantic layout between pixels, that is, ignoring the subtle differences between adjacent pixels in color and texture, as well as the structuring of semantics, making it difficult for the model to accurately distinguish the categories of adjacent pixels during segmentation, resulting in the phenomenon of representation collapse, resulting in inaccurate image segmentation of the model.

[0056] In order to solve the problems caused by the unsupervised image semantic segmentation method based on mutual information, the inventors designed a semantic attention network applied to the unsupervised image semantic segmentation method. Figure 1 In the method flow of label 3, the semantic attention network (i.e., the deep neural network in label 3) maps the pixel-level representation (i.e., the pixel-level embedded representation in label 3) and the semantic-level representation (i.e., the semantic embedded representation in label 3) to the latent space at the same time, and then matches them through the semantic attention module (i.e., the semantic attention in label 3), thereby not only enabling the semantic attention network to learn a better overall image representation (i.e., image-level representation), but also simultaneously focusing on the fine-grained information between pixels (i.e., the subtle differences between adjacent pixels in terms of color and texture, etc.) and structured semantic layout (i.e., semantic structuring). The inventor also applies supervisory information to the output of the semantic attention module.

[0057] The specific implementation of the present application is further described below in conjunction with the accompanying drawings.

[0058] like Figure 2 The above is a flowchart of an unsupervised image semantic segmentation method provided in the first aspect of the embodiment of the present application. Figure 2 In the method, the unsupervised image semantic segmentation method comprises:

[0059] S1. Acquire multiple original images.

[0060] In the embodiment of the present application, the multiple original images can be regarded as a data set, which is represented as , are original images, and the resolution of each original image is defined as (H, W).

[0061] S2. All the original images are input into a semantic attention network (SAN), which matches and fuses the pixel-level representation generated in the first latent space and the semantic-level representation generated in the second latent space for each original image, and outputs a segmentation result corresponding to each original image.

[0062] The semantic attention network is mainly used for unsupervised image semantic segmentation tasks. The semantic attention network includes a pixel-level encoder, a semantic generator, a consistency retainer and a semantic attention module.

[0063] Through the above technical solution, the present application can not only enable the semantic attention network to learn a better overall image representation (i.e., image-level representation), but also pay attention to the fine-grained information between pixels and the attention of structured semantic layout. Therefore, compared with the prior art, the embodiment of the present application can accurately distinguish the categories of adjacent pixels during semantic segmentation, thereby effectively avoiding the occurrence of representation collapse and improving the accuracy of image segmentation.

[0064] In some embodiments, reference Figure 3 As shown, the S2 step includes:

[0065] S21. All the original images are input into the pixel-level encoder of the semantic attention network. The pixel-level encoder processes the pixels of each original image and maps them to the first latent space to obtain a pixel-level representation corresponding to each original image.

[0066] Furthermore, the step S21 includes:

[0067] S211. Input all the original images into a pixel sensor of the pixel-level encoder, wherein the pixel sensor generates a pixel-by-pixel region representation corresponding to each pixel in each original image based on an adjacent receptive field of each pixel in each original image.

[0068] More specifically, the pixel sensor adopts a first convolutional layer, a first pooling layer and other structures to slide on each of the original images in a sliding window (or convolution kernel) manner, extracts features of each pixel and its neighborhood, and obtains the pixel-by-pixel region representation corresponding to each pixel.

[0069] Each of the pixel-by-pixel region representations is a high-dimensional feature map, which may at least include resolution, color, bit depth, texture, hue, saturation, brightness, color channel, and hierarchical composition information of the image.

[0070] The dimension of the pixel-by-pixel region representation is defined as P / 2, where P is the dimension of the pixel-level representation and the semantic-level representation.

[0071] exist Figure 3 In , an original image is represented as (B, C, H, W), where B represents the number of batches, C represents the number of channels of the original image, and (H, W) is the resolution of the original image. The pixel-by-pixel region representation is represented as (B, P / 2, H, W), where B represents the number of batches, P / 2 represents the dimension of the pixel-by-pixel region representation, and (H, W) is the resolution of the pixel-by-pixel region representation.

[0072] Step S211 is a specific step of focusing on the fine-grained information between pixels.

[0073] S212: Input all pixel-by-pixel region representations to a pixel encoder of the pixel-level encoder, and the pixel encoder maps all the pixel-by-pixel region representations to the first latent space to obtain the pixel-level representation corresponding to each of the original images.

[0074] The pixel encoder is composed of multiple second convolutional layers, activation layers and second pooling layers. These hierarchical structures are stacked and combined to gradually map all pixel-by-pixel region representations of the input to a higher-dimensional feature space, i.e., the first latent space. In this application, the first latent space can be understood as dimension P.

[0075] In this application, the pixel-level representation can be expressed as (P, H, W), denoted as ,in, is the symbol of space dimension, , P, H and W have the same meanings as above, so they will not be repeated here. Figure 3 In , the pixel-level representation is denoted as (B, P, H, W).

[0076] S22, cutting each original image into a plurality of first images to be processed A, and linearly mapping each first image to be processed to the second latent space through a convolution operation, to obtain a plurality of second images to be processed corresponding to each original image.

[0077] In the embodiment of the present application, each original image is cut into K small blocks of first images A to be processed.

[0078] The second latent space is a P-dimensional space.

[0079] S23. Input multiple blocks of second images to be processed corresponding to each original image into the semantic generator of the semantic attention network. The semantic generator performs semantic processing on all the second images to be processed to obtain a semantic-level representation corresponding to each original image.

[0080] Furthermore, the step S23 includes:

[0081] S231 . The transformer-based encoder in the semantic generator extracts semantic information from each block of the second image to be processed, and generates batch-level semantic information corresponding to the original image.

[0082] The transformer-based encoder is a vision transformer, which extracts semantic information from all the second images to be processed and generates an image representation of dimension (B, K, P), that is, the batch-level semantic information, where B represents the number of batches, K is the number of generated information blocks, and K is calculated by K=(H / c)×(W / c), c is the resolution of each information block, and the information block refers to the second image to be processed. The meaning of P is the same as that mentioned above, so it is not repeated here.

[0083] S232. The multi-head generator of the semantic generator generates classification information corresponding to the original image based on the batch-level semantic information.

[0084] The multi-head generator remaps the batch-level semantic information to a specified number of semantic representation centers (i.e., the classification information) ,in, is the symbol of spatial dimension, N is the semantic quantity of the entire data set (i.e., the plurality of original images), P is the dimension, It can be understood that the multi-head generator has P dimensions and generates N cluster centers in these P dimensions. The cluster centers refer to the number of semantics, and the purpose of the cluster centers is to classify each pixel. It should be understood that the classification process can be understood as a semantic structuring process.

[0085] Label each semantic representation with a category probability prediction ,in, is an N-dimensional probability simplex.

[0086] S233: The information block matcher of the semantic generator generates the semantic level representation based on the classification information and the pixel level representation.

[0087] The information block matcher classifies the pixel-level representation based on the N cluster centers, selects a suitable semantic category for the pixel-level representation, and thus generates the semantic-level representation. The semantic-level representation is represented as ,in, , , .exist Figure 3 In , the semantic level representation is represented as (B, N, P).

[0088] In this application, the function of the semantic generator can be understood as aggregating the batch-level semantic information, arranging semantic categories for each image, and then converting the semantic level representation and the pixel-level characterization Input into the semantic attention module of the semantic attention network to calculate semantic attention.

[0089] S24, inputting the pixel-level representation and the semantic-level representation corresponding to each of the original images into the semantic attention module of the semantic attention network for matching processing to obtain an aligned representation corresponding to the original image. Figure 3 In , the alignment representation is represented as (B, C, H, W).

[0090] In an embodiment of the present application, the semantic attention module dynamically constructs the relationship between the pixel-level representation and the semantic-level representation through an attention mechanism.

[0091] The attention mechanism can calculate the weight of each value vector through the compatibility function of the query and its corresponding key.

[0092] The compatibility function is used to calculate the similarity between the query and the key, thereby affecting the allocation of attention.

[0093] In the embodiment of the present application, the calculation formula of the semantic attention module is defined as:

[0094] ;

[0095] in, For the pixel-level characterization, is the sum of the semantic level representations, is the scaling factor, for The transpose of .

[0096] Furthermore, the calculation formula of the pixel-level representation is: ,in, are the multiple original images, , N is the total number of the original images, is the pixel level encoder.

[0097] Furthermore, the calculation formula of the sum of the semantic level representations is:

[0098] ;

[0099] in, is the semantic generator, are the multiple original images, , N is the total number of the original images.

[0100] In some embodiments, the calculation formula of the attention mechanism adopted by the semantic attention module in the embodiments of the present application is as follows:

[0101] ;

[0102] Where Q is the query, K is the key, and V is the value. is the scaling factor, yes The scaling factor can be determined by those skilled in the art according to actual needs.

[0103] Furthermore, the calculation formula of the query is: ; The calculation formula of the key is ; The value is calculated by ; In these three formulas, x is the input image, , and It is a learnable network, and the learnable network can be determined by those skilled in the art according to actual needs.

[0104] This application adopts the semantic attention module to better generate pixel-level features and semantic-level features.

[0105] S25, inputting the pixel-level representation and the aligned representation corresponding to each of the original images into the consistency retainer of the semantic attention network for fusion processing, and obtaining the segmentation result corresponding to the original image (i.e. Figure 3 “Semantic Segmentation Results” in ).

[0106] The consistency preserver is a pixel decoder. The pixel decoder contains a shallow 1×1 convolutional neural network, and the pixel-level representation and the alignment representation share the 1×1 convolutional neural network to minimize the alignment embedding, the pixel-level representation and the distance between the input image, strengthening the consistency between the pixel-level representation and the semantic-level representation.

[0107] In some embodiments, the training method of the semantic attention network includes:

[0108] S31, obtaining a training data set and a real label set, wherein each training data in the training data set has a one-to-one correspondence with a real label in the real label set.

[0109] S32. Input the target training data into the initial semantic attention network and output the predicted segmentation result, wherein the initial semantic attention network includes an initial pixel-level encoder, an initial semantic generator, an initial consistency retainer and an initial semantic attention module, and the target training data belongs to any training data in the training data set.

[0110] It should be understood that the initial pixel-level encoder is identical to the pixel-level encoder in structure, and the difference between the two lies in the difference in network parameters. The initial semantic generator is identical to the semantic generator in structure, and the difference between the two lies in the difference in network parameters. The initial consistency retainer is identical to the consistency retainer in structure, and the difference between the two lies in the difference in network parameters. The initial semantic attention module is identical to the semantic attention module in structure, and the difference between the two lies in the difference in network parameters.

[0111] S33. Calculate a loss function based on the predicted segmentation result and the true label corresponding to the target training data.

[0112] S34. Adjust the network parameters of the initial semantic attention network based on the loss function to form the semantic attention network.

[0113] In some embodiments, the loss function is calculated as:

[0114] ;

[0115] in, represents the total number of samples, Representative The true labels, Representative Predicted segmentation results.

[0116] exist Figure 3 middle, and is the loss function. The formula and The formula is the same as This can be achieved through the existing loss function, which will not be described in this article.

[0117] In some embodiments, the embodiments of the present application have state-of-the-art performance in the application scenarios of the COCO-Stuff-3 dataset, the COCO-Stuff-15 dataset, the COCO-Stuff-27 dataset, the Cityscapes dataset, and the POTSDAM dataset.

[0118] like Figure 4 , which is a structural block diagram of an unsupervised image semantic segmentation device provided in the second aspect of the embodiment of the present application. Figure 4 In the embodiment, the unsupervised image semantic segmentation device 400 comprises:

[0119] An acquisition module 401 is used to acquire a plurality of original images;

[0120] The segmentation result output module 402 is used to input all the original images into the semantic attention network, and the semantic attention network matches and fuses the pixel-level representation generated in the first latent space and the semantic-level representation generated in the second latent space of each original image, and outputs the segmentation result corresponding to each original image. A third aspect of the embodiment of the present application provides a terminal device, and the principle block diagram of the terminal device can be as follows Figure 5 As shown. The terminal device includes a processor, a memory, a network interface, a display screen and a temperature sensor connected via a system bus. Among them, the processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an unsupervised image semantic segmentation method is implemented. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor is pre-set inside the terminal device to detect the operating temperature of the internal device.

[0121] Those skilled in the art will understand that Figure 5 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal device to which the scheme of the present invention is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0122] In some embodiments, the terminal device includes a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the steps of the unsupervised image semantic segmentation method described in the first aspect of the above-mentioned embodiment of the present application.

[0123] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program enables a computer to execute the steps of the unsupervised image semantic segmentation method described in the first aspect of the embodiment of the present application.

[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0125] The technical features of the above embodiments can be combined without changing the basic principles of the present invention. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the scope of patent protection of the present application shall be subject to the attached claims.

Claims

1. An unsupervised image semantic segmentation method, characterized in that: include: Acquire multiple original images; Inputting all the original images into a semantic attention network, the semantic attention network matches and fuses the pixel-level representation generated in the first latent space and the semantic-level representation generated in the second latent space for each original image, and outputs a segmentation result corresponding to each original image; The step of inputting all the original images into the semantic attention network, matching and fusing the pixel-level representation generated in the first latent space and the semantic-level representation generated in the second latent space of each original image, and outputting the segmentation result corresponding to each original image comprises: Inputting all the original images into the pixel-level encoder of the semantic attention network, the pixel-level encoder processes the pixels of each original image and maps them to the first latent space to obtain a pixel-level representation corresponding to each original image; Cut each original image into a plurality of first images to be processed, and linearly map each first image to be processed to the second latent space through a convolution operation to obtain a plurality of second images to be processed corresponding to each original image; Inputting a plurality of second images to be processed corresponding to each original image into a semantic generator of the semantic attention network, wherein the semantic generator performs semantic processing on all the second images to be processed to obtain a semantic level representation corresponding to each original image; Inputting the pixel-level representation and the semantic-level representation corresponding to each of the original images into the semantic attention module of the semantic attention network for matching processing to obtain an aligned representation corresponding to the original image; The pixel-level representation and the aligned representation corresponding to each of the original images are input into the consistency retainer of the semantic attention network for fusion processing to obtain a segmentation result corresponding to the original image.

2. The unsupervised image semantic segmentation method according to claim 1, characterized in that: The step of inputting all the original images into the pixel-level encoder of the semantic attention network, wherein the pixel-level encoder processes the pixels of each original image and maps them to the first latent space, and obtaining the pixel-level representation corresponding to each original image comprises: Inputting all the original images into a pixel sensor of the pixel-level encoder, wherein the pixel sensor generates a pixel-by-pixel region representation corresponding to each pixel in each original image based on a neighboring receptive field of each pixel in each original image; All pixel-by-pixel region representations are input to a pixel encoder of the pixel-level encoder, and the pixel encoder maps all the pixel-by-pixel region representations to the first latent space to obtain the pixel-level representation corresponding to each of the original images.

3. The unsupervised image semantic segmentation method according to claim 1, characterized in that: The step of inputting a plurality of second images to be processed corresponding to each original image into the semantic generator of the semantic attention network, wherein the semantic generator performs semantic processing on all the second images to be processed to obtain a semantic level representation corresponding to each original image comprises: The encoder based on the transformer in the semantic generator extracts semantic information from each block of the second image to be processed, and generates batch-level semantic information corresponding to the original image; The multi-head generator of the semantic generator generates classification information corresponding to the original image based on the batch-level semantic information; The information block matcher of the semantic generator generates the semantic level representation based on the classification information and the pixel level representation.

4. The unsupervised image semantic segmentation method according to claim 1, characterized in that: The calculation formula of the semantic attention module is defined as: ； in, For the pixel-level characterization, is the scaling factor, is the sum of the semantic level representations, for The transpose of .

5. The unsupervised image semantic segmentation method according to claim 1, characterized in that: The training method of the semantic attention network includes: Obtain a training data set and a real label set, wherein each training data in the training data set has a one-to-one correspondence with a real label in the real label set; Input the target training data into an initial semantic attention network and output a predicted segmentation result, wherein the initial semantic attention network includes an initial pixel-level encoder, an initial semantic generator, an initial consistency retainer and an initial semantic attention module; Calculating a loss function based on the predicted segmentation result and a true label corresponding to the target training data; Adjusting the network parameters of the initial semantic attention network based on the loss function to form the semantic attention network; The target training data belongs to any training data in the training data set.

6. The unsupervised image semantic segmentation method according to claim 5, characterized in that: The calculation formula of the loss function is: ； in, represents the total number of samples, Representative The true labels, Representative Predicted segmentation results.

7. An unsupervised image semantic segmentation device, characterized in that: include: An acquisition module, used for acquiring multiple original images; A segmentation result output module, used to input all the original images into a semantic attention network, wherein the semantic attention network matches and fuses the pixel-level representation generated in the first latent space and the semantic-level representation generated in the second latent space of each original image, and outputs a segmentation result corresponding to each original image; The segmentation result output module is also used to input all the original images into the pixel-level encoder of the semantic attention network, and the pixel-level encoder processes the pixels of each original image and maps them to the first latent space to obtain a pixel-level representation corresponding to each original image; cut each original image into multiple blocks of first images to be processed, and linearly map each block of the first images to be processed to the second latent space through a convolution operation to obtain multiple blocks of second images to be processed corresponding to each original image; Inputting a plurality of second images to be processed corresponding to each original image into the semantic generator of the semantic attention network, the semantic generator performs semantic processing on all the second images to be processed to obtain a semantic-level representation corresponding to each original image; inputting the pixel-level representation and the semantic-level representation corresponding to each original image into the semantic attention module of the semantic attention network for matching processing to obtain an aligned representation corresponding to the original image; The pixel-level representation and the aligned representation corresponding to each of the original images are input into the consistency retainer of the semantic attention network for fusion processing to obtain a segmentation result corresponding to the original image.

8. A terminal device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the steps of the unsupervised image semantic segmentation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program enables a computer to execute the steps of the unsupervised image semantic segmentation method according to any one of claims 1 to 6.

Images