Weak supervision incremental learning semantic segmentation method, equipment and medium

Abstract

The invention discloses a weak supervision incremental learning semantic segmentation method and device, and a medium. The method comprises the steps of obtaining a to-be-segmented picture; inputting a to-be-segmented picture into the trained semantic segmentation model, and outputting a segmentation result; wherein the semantic segmentation model comprises an encoder, a decoder, an affinity network and a lightweight multi-scale segmentation head; the encoder is used for carrying out feature extraction on an input image, and the decoder is used for carrying out category prediction according to the extracted features; the affinity network is used for acquiring affinity information according to the features extracted by the encoder as an affinity matrix for optimizing the segmentation head result; the lightweight multi-scale segmentation head is used for obtaining a rough new class segmentation result according to the output of the encoder. According to the method, the affinity network is introduced, the rough segmentation result is refined through the affinity matrix obtained through the affinity value, the effect is better, the affinity network is directly designed into the network as a module in the framework, end-to-end training prediction is achieved, and the effect is good.

Description

Technical Field

[0001] The present invention relates to the field of image segmentation technology, and in particular to a weakly supervised incremental learning semantic segmentation method, device and medium. Background Art

[0002] With the development of semantic segmentation technology, it has played an important role in various tasks across various industries. However, although existing semantic segmentation methods have achieved remarkable results in many application scenarios, in practical applications, models often need to be updated with new types and new scenarios over time. In addition, the pixel-by-pixel annotation process required for semantic segmentation is expensive and time-consuming, especially for objects with irregular shapes. Although annotations for old categories can be easily obtained through experience accumulation, newly discovered categories still need to be re-annotated pixel by pixel, which makes learning these new categories more difficult.

[0003] In the past, weakly supervised learning was used to solve the problem of lack of pixel-level labels, and incremental learning technology was used to solve the problem of learning new categories. However, during production time, these two problems often occur at the same time. New categories also require new labels, and these new labels require a lot of manpower. In recent years, a new research direction has emerged, namely the weakly supervised incremental learning framework. The semantic segmentation task of this framework happens to solve the two problems mentioned above at the same time. However, this framework is still in its early stages and the framework structure is not yet perfect. It only learns new categories through simple class activation layers, such as Figure 1 As shown, the learning effect is poor and the efficiency is low, and the learning effect of new categories needs to be improved. Summary of the Invention

[0004] In order to at least solve one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide an end-to-end weakly supervised incremental learning semantic segmentation method, device and medium combined with affinity network optimization.

[0005] The first technical solution adopted by the present invention is:

[0006] A weakly supervised incremental learning semantic segmentation method includes the following steps:

[0007] Get the image to be segmented;

[0008] Input the image to be segmented into the trained semantic segmentation model and output the segmentation result;

[0009] The semantic segmentation model includes an encoder, a decoder, an affinity network and a lightweight multi-scale segmentation head; the encoder is used to extract features from the input image, and the decoder is used to predict categories based on the extracted features; the affinity network is used to obtain affinity information based on the features extracted by the encoder, which serves as an affinity matrix for optimizing the segmentation head results; the lightweight multi-scale segmentation head is used to obtain a rough new category segmentation result based on the output of the encoder.

[0010] Furthermore, the semantic segmentation model is trained in the following way:

[0011] Apply the weights obtained from the encoder-decoder structure training in step t-1 to step t;

[0012] During step t training, the encoder-decoder weights are frozen, and the affinity network and lightweight multi-scale segmentation head are trained using labeled data from the old categories. Unlabeled data is input, and the predictions of the lightweight multi-scale segmentation head are enhanced using the affinity matrix obtained by the affinity network. The resulting predictions are the semantic segmentation model's predictions for the new categories in the unlabeled data.

[0013] The prediction results of the new category are spliced with the prediction results of the encoder-decoder for the old category to obtain the prediction results of the semantic segmentation model for all categories of the unlabeled data. The prediction results are supervised, allowing the semantic segmentation model to learn new categories incrementally with weak supervision.

[0014] Furthermore, the affinity network is specifically used to extract affinity information of relevant positions from feature information of different levels extracted during the encoding process of the encoder; the feature information of different levels includes deep feature information, sub-layer feature information and shallow feature information.

[0015] Furthermore, the input feature map during affinity network training is feature information of different depth levels generated during the decoder process, and the affinity label is an affinity matrix composed of affinity values between different coordinates in the results of the segmentation head in the framework.

[0016] Furthermore, the affinity tag is obtained by taking a weighted average of the affinity value calculated according to the L1 distance and the cosine similarity calculated according to the cosine similarity;

[0017] The calculation formula of the affinity tag is as follows:

[0018]

[0019] Where A ij L1 is the affinity value calculated based on the L1 distance, A ij cosis the cosine similarity calculated based on the cosine similarity, A ij The affinity value is the weighted average of the two; f aff (x i ,y i ) is the position coordinate (x i ,y i ) affinity value, f aff (x j ,y j ) is the position coordinate (x j ,y j ) affinity value; C is the number of channels of the feature map to be calculated, subscript c is the feature channel to be calculated; ‖‖1 is the L1 distance; λ1 and λ2 are weight coefficients.

[0020] Furthermore, the affinity network only calculates affinity values for coordinate pairs within a radius of γ, and the set S of coordinate pairs of calculated affinity is:

[0021]

[0022] Where, d((x i ,y i ),(x j ,y j )) represents the coordinate (x i ,y i ) and coordinates (x j ,y j ) between the two locations;

[0023] Divide S into three subsets and aggregate the losses obtained from each subset:

[0024] First, divide S into two subsets of positive and negative pairs:

[0025]

[0026]

[0027] Where, Affinity value A ij The result of binarization;

[0028] According to the segmentation result of the semantic segmentation head, the category attributes of the coordinate pairs corresponding to the segmentation result of the semantic segmentation head are further Decompose into context-related and prospects The cross entropy loss for each subset is calculated as follows:

[0029]

[0030] The affinity network is supervised and trained through the loss function, so that the affinity network can extract and generate affinity information from the encoder process to form an affinity matrix.

[0031] Furthermore, the affinity matrix A is used to calculate the random walk probability matrix T, and the formula is as follows:

[0032]

[0033] Where A° β is the Hadamard power of the affinity matrix, β is a hyperparameter greater than 1; D is a diagonal matrix, focusing on the element D ii The value of is calculated as above formula;

[0034] The propagation is achieved by multiplying the random walk probability matrix T by CAM. The semantic segmentation model iteratively performs this propagation until a predefined number of iterations t. The formula is as follows:

[0035]

[0036] Where M c The output of the segmentation head is vectorized and optimized through the diffusion matrix T. The affinity network is used to learn semantic affinity and perform semantic propagation.

[0037] Furthermore, the lightweight multi-scale segmentation head includes multiple depth-separable convolutions of different scales. Through the depth-separable convolution kernels of different scales, the model receptive field is improved while reducing the network architecture and parameter amount. The feature extractions of different scales are fused together by addition. A 1*1 convolution is used to reduce the channel and extract deep semantic information. The deep semantic information is then fused with the shallow semantic information extracted from only a 3*3 convolution. The segmentation result is then refined from the rich semantic knowledge obtained from this fusion.

[0038] After obtaining the rough new category segmentation result in the segmentation head, this rough data is refined through the affinity matrix obtained by the affinity network, and then spliced with the old category segmentation result obtained by the original encoder-decoder network to finally obtain the complete output result.

[0039] Furthermore, the affinity network optimizes the segmentation head result as follows:

[0040] The segmentation head obtains a rough prediction effect M c , the affinity information A obtained through the affinity network is used to calculate the random walk probability matrix T, and then through iterative diffusion, the prediction results are optimized through the diffusion results, and finally the segmentation head obtains a more accurate output result.

[0041] The second technical solution adopted by the present invention is:

[0042] An electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, the code set or the instruction set is loaded and executed by the processor to implement a weakly supervised incremental learning semantic segmentation method as described above.

[0043] The third technical solution adopted by the present invention is:

[0044] A computer-readable storage medium stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a weakly supervised incremental learning semantic segmentation method as described above.

[0045] The fourth technical solution adopted by the present invention is:

[0046] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned weakly supervised incremental learning semantic segmentation method.

[0047] The present invention has the following beneficial effects: It introduces an affinity network, which refines rough segmentation results using an affinity matrix derived from affinity values, achieving better results. Furthermore, the affinity network is directly integrated into the network as a module within the framework, enabling end-to-end training and prediction with excellent results. Furthermore, the lightweight multi-scale segmentation head is used to decode deep semantic information, achieving higher efficiency and better accuracy than a single class activation layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 This is a schematic diagram of the existing weakly supervised incremental learning framework;

[0050] Figure 2 Schematic diagram of a weakly supervised incremental learning framework provided by an embodiment of the present invention;

[0051] Figure 3 is a schematic diagram of an affinity network in an embodiment of the present invention;

[0052] Figure 4 is a structural diagram of a lightweight semantic segmentation head in an embodiment of the present invention;

[0053] Figure 5 Schematic diagram of the relationship between the affinity network and the segmentation head in the framework of the present invention in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0055] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0056] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0057] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0058] Example 1

[0059] This embodiment provides a weakly supervised incremental learning semantic segmentation method, including the following steps:

[0060] S1. Get the image to be segmented;

[0061] S2. Input the image to be segmented into the trained semantic segmentation model and output the segmentation result;

[0062] The semantic segmentation model includes an encoder, a decoder, an affinity network and a lightweight multi-scale segmentation head; the encoder is used to extract features from the input image, and the decoder is used to predict categories based on the extracted features; the affinity network is used to obtain affinity information based on the features extracted by the encoder, which serves as an affinity matrix for optimizing the segmentation head results; the lightweight multi-scale segmentation head is used to obtain a rough new category segmentation result based on the output of the encoder.

[0063] See also Figure 1 In the existing framework, new categories are learned only through simple class activation layers, which has poor learning effect and low efficiency. Figure 2 In this embodiment, the affinity network and the lightweight multi-scale segmentation head are introduced into the weakly supervised incremental learning framework. The affinity matrix obtained by the affinity value is used to refine the rough segmentation results, which has better effect. The affinity network is directly designed into the network as a module in the framework to achieve end-to-end training and prediction, with good effect.

[0064] Specifically, see Figure 2 The working principle of the framework provided in this embodiment is as follows: the weights obtained from the encoder-decoder structure training in step t-1 are applied to step t; when the training in step t is executed, the affinity network and the multi-scale segmentation head are first trained using the labeled data of the old category. The training at this stage requires freezing the weights of the encoder-decoder; then, the unlabeled data is input into the network, and the prediction results of the lightweight multi-scale segmentation head are enhanced by the affinity matrix obtained by the affinity network. The result obtained is the prediction result of the framework for the new category in the unlabeled data. At the same time, it is spliced with the previous encoder-decoder prediction results for the old category to obtain the recognition effect of the entire framework for all categories of the unlabeled data. The prediction result is supervised so that the entire framework can learn new categories incrementally with weak supervision.

[0065] The framework proposed in this embodiment is supplemented below with reference to the accompanying drawings and specific implementation methods.

[0066] (1) Affinity Network

[0067] This framework extracts feature information at different levels from the encoder encoding process, mixes this information and inputs it into the affinity network, and obtains affinity information through the prediction of the affinity network. This affinity information is used as the optimization matrix for optimizing the segmentation head results.

[0068] The affinity network requires the ability to extract affinity information of relevant positions from feature information input at different levels. The accuracy of this prediction is guaranteed by the effective training of the affinity network.

[0069] See also Figure 3 The training of the affinity network relies on the input feature map and affinity labels to learn how to predict the relationship between pixels. The input feature map selected by the affinity network in the framework of the embodiment of the present invention is the feature information of different depth levels generated in the decoder process, and the affinity label comes from the affinity matrix composed of the affinity values between different coordinates in the results of the segmentation head in the framework.

[0070] The affinity tag designed in this framework is calculated by the affinity between a pair of feature vectors based on the weighted average of their L1 distance and spatial cosine similarity:

[0071] A ij L1 =exp{-||f aff (x i ,y i )-f aff (x j ,y j )||1}

[0072]

[0073] Among them, A ij L1 is the affinity value calculated based on the L1 distance, A ij cos is the cosine similarity calculated based on the cosine similarity, A ij The affinity value is the weighted average of the two.

[0074] Since the correlation of distant coordinate pairs is weakened, and it is necessary to balance the computational cost and the effect gain brought by semantic information, the affinity network only calculates the affinity value for the coordinate pairs within a radius of γ. To represent the set of coordinate pairs for which we calculate affinity:

[0075]

[0076] Due to the problem of class imbalance, it is not advisable to directly use the information of these sets to train the affinity network. The class distribution in is obviously biased towards the positive class distribution, because negative pairs usually only exist at the edge of the object. At the same time, in the subset of positive pairs, the number of background pairs is obviously greater than the number of object pairs, because the background is larger than the object area in many photos. To solve this problem, this embodiment will Divide into three subsets and aggregate the losses obtained from each subset. First, Divided into two subsets of positive and negative pairs:

[0077]

[0078] And further classify the coordinate pairs corresponding to the segmentation results of the lightweight semantic segmentation head into Decompose into context-related and prospects Therefore, the cross entropy loss for each subset is calculated as follows:

[0079]

[0080] This framework uses the above loss function to supervise the training of the affinity network, so that the affinity network can extract and generate affinity information from the encoder process to form an affinity matrix.

[0081] In previous weakly supervised semantic segmentation models, the affinity network is often an auxiliary network independent of the segmentation network. In the framework of the embodiment of the present invention, the affinity module is directly embedded in the framework to achieve end-to-end training prediction effect, rather than treating it as an independent network. This allows us to directly apply the affinity results to the model prediction results. The local semantic affinity predicted by the affinity network is converted into an affinity matrix, which enables the random walk to be aware of the boundaries of the image and prompts it to diffuse the activation score within these boundaries. We use the affinity matrix A to calculate the random walk probability matrix T, and the formula is as follows:

[0082]

[0083] Among them, A° β is the Hadamard power of the affinity matrix, and the diagonal matrix is row-normalized. Propagation is achieved by multiplying the random walk matrix by the CAM. This model iterates this propagation until a predefined number of iterations t is reached. The formula is as follows:

[0084]

[0085] Among them, M c The output of the segmentation head is vectorized and optimized through the diffused matrix T. The affinity network is used to learn semantic affinity and perform semantic propagation. This not only significantly improves the effect of weakly supervised semantic segmentation without additional data or stronger supervision, but also enables the final segmentation model to more accurately capture the complete shape of the target object, improving the weakly supervised incremental learning framework's ability to understand new knowledge.

[0086] (2) Lightweight multi-scale segmentation head

[0087] See also Figure 4 The lightweight multi-scale segmentation head designed by this framework is composed of multiple depth-separable convolutions of different scales. The depth-separable convolution kernels of different scales can improve the receptive field of the model while reducing the network architecture and the number of parameters. The feature extractions of different scales are fused together by addition. The channel is reduced by a 1*1 convolution to extract deep semantic information. The deep semantic information is then fused with the shallow semantic information extracted by only a 3*3 convolution. The segmentation result is then refined from the rich semantic knowledge obtained from this fusion.

[0088] A rough new category segmentation result is obtained from this segmentation head. This rough data is then refined through the affinity matrix obtained by the affinity network, and then spliced with the old category segmentation result obtained by the original encoder-decoder network. Finally, a complete output result is obtained. The entire network structure is supervised by this final result, so that the framework has the ability to segment new category data.

[0089] (3) The impact of affinity network on segmentation head results

[0090] See also Figure 5 , the segmentation head obtains a rough prediction effect M c The affinity information A obtained through the affinity network is used to calculate the random walk probability matrix T. This is then used through iterative diffusion to optimize the predictions, ultimately enabling the segmentation head to achieve more accurate outputs. In this weakly supervised incremental learning framework, the segmentation head predicts new categories within the incremental framework. This improved performance helps generate more accurate pseudo-labels in subsequent processes. Ultimately, through supervision of the entire encoder-decoder model, the framework enables superior incremental learning of unlabeled data.

[0091] Example 2

[0092] An embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the following Figure 2 A weakly supervised incremental learning semantic segmentation method shown.

[0093] It is understood that the memory may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the data storage area may store data created based on the use of the server, etc.

[0094] The processor may include one or more processing cores. The processor utilizes various interfaces and circuits to connect various components within the server. It executes various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, as well as accessing data stored in memory. Optionally, the processor may be implemented using at least one of the following hardware forms: digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor may integrate one or a combination of a central processing unit (CPU) and a modem. The CPU primarily processes the operating system and application programs, while the modem handles wireless communications. It is understood that the modem may not be integrated into the processor and may be implemented separately via a single chip.

[0095] Since the electronic device is an electronic device corresponding to a weakly supervised incremental learning semantic segmentation method of an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0096] Example 3

[0097] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 2 A weakly supervised incremental learning semantic segmentation method shown.

[0098] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0099] Since the storage medium is a storage medium corresponding to a weakly supervised incremental learning semantic segmentation method of an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0100] Example 4

[0101] In some possible implementations, various aspects of the method of the embodiments of the present invention may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps of a weakly supervised incremental learning semantic segmentation method according to various exemplary embodiments of the present application described above in this specification. The executable computer program code or "code" for executing each embodiment may be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0102] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0103] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0104] The above embodiments are intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made based on the essence of the present invention are intended to be covered by the scope of protection of the present invention.

Claims

1. A weakly supervised incremental learning semantic segmentation method, characterized by: The following steps are involved: Get the image to be segmented; Input the image to be segmented into the trained semantic segmentation model and output the segmentation result; The semantic segmentation model includes an encoder, a decoder, an affinity network and a lightweight multi-scale segmentation head; the encoder is used to extract features from the input image, and the decoder is used to predict categories based on the extracted features; the affinity network is used to obtain affinity information based on the features extracted by the encoder, which serves as an affinity matrix for optimizing the segmentation head results; the lightweight multi-scale segmentation head is used to obtain a rough new category segmentation result based on the output of the encoder.

2. A weakly supervised incremental learning semantic segmentation method according to claim 1, characterized in that The semantic segmentation model is trained in the following way: Apply the weights obtained from the encoder-decoder structure training in step t-1 to step t; During step t training, the encoder-decoder weights are frozen, and the affinity network and lightweight multi-scale segmentation head are trained using the old category labeled data. Unlabeled data is input, and the prediction results of the lightweight multi-scale segmentation head are enhanced by the affinity matrix obtained by the affinity network. The obtained result is the prediction result of the semantic segmentation model for the new category in the unlabeled data; The prediction results of the new category are spliced with the prediction results of the encoder-decoder for the old category to obtain the prediction results of the semantic segmentation model for all categories of the unlabeled data. The prediction results are supervised, allowing the semantic segmentation model to learn new categories incrementally with weak supervision.

3. The weakly supervised incremental learning semantic segmentation method according to claim 1, characterized in that The input feature map during affinity network training is feature information of different depth levels generated during the decoder process, and the affinity label is an affinity matrix composed of affinity values between different coordinates in the results of the segmentation head in the framework.

4. The weakly supervised incremental learning semantic segmentation method according to claim 3, characterized in that: The affinity tag is obtained by weighted average of the affinity value calculated according to the L1 distance and the cosine similarity calculated according to the cosine similarity; The calculation formula of the affinity tag is as follows: A ij L1 =exp{-||f aff (x i ,y i )-f aff (x j ,y j )||1} Where A ij L1 is the affinity value calculated based on the L1 distance, A ij cos is the cosine similarity calculated based on the cosine similarity, A ij The affinity value is the weighted average of the two; f aff (x i ,y i ) is the position coordinate (x i ,y i ) affinity value, f aff (x j ,y j ) is the position coordinate (x j ,y j ) affinity value; C is the number of channels of the feature map to be calculated, subscript c is the feature channel to be calculated; ‖‖1 is the L1 distance; λ1 and λ2 are weight coefficients.

5. The weakly supervised incremental learning semantic segmentation method according to claim 4, characterized in that: The affinity network only calculates affinity values for coordinate pairs within a radius of γ. The set of coordinate pairs S of calculated affinity is: Where, d((x i ,y i ),(x j ,y j )) represents the coordinate (x i ,y i ) and coordinates (x j ,y j ) between the two locations; Divide S into three subsets and aggregate the losses obtained from each subset: First, divide S into two subsets of positive and negative pairs: Where, Affinity value A ij The result of binarization; According to the segmentation result of the semantic segmentation head, the category attributes of the coordinate pairs corresponding to the segmentation result of the semantic segmentation head are further Decompose into context-related and prospects The cross entropy loss for each subset is calculated as follows: The affinity network is supervised and trained through the loss function, so that the affinity network can extract and generate affinity information from the encoder process to form an affinity matrix.

6. The weakly supervised incremental learning semantic segmentation method according to claim 3, characterized in that: The affinity matrix A is used to calculate the random walk probability matrix T. The formula is as follows: T=D -1 A °β ,in Where A °β is the Hadamard power of the affinity matrix, β is a hyperparameter greater than 1; D is a diagonal matrix; The propagation is achieved by multiplying the random walk probability matrix T by CAM. The semantic segmentation model iteratively performs this propagation until a predefined number of iterations t. The formula is as follows: Where m c The output of the segmentation head is vectorized and optimized through the diffusion matrix T. The affinity network is used to learn semantic affinity and perform semantic propagation.

7. The weakly supervised incremental learning semantic segmentation method according to claim 1, characterized in that: The lightweight multi-scale segmentation head includes multiple depth-separable convolutions of different scales. Through the depth-separable convolution kernels of different scales, the model receptive field is improved while reducing the network architecture and parameter quantity. The feature extractions of different scales are fused together by summing. A 1*1 convolution is used to reduce the channel and extract deep semantic information. The deep semantic information is then fused with the shallow semantic information extracted from only a 3*3 convolution. The segmentation result is then refined from the rich semantic knowledge obtained from this fusion. After obtaining the rough new category segmentation result in the segmentation head, this rough data is refined through the affinity matrix obtained by the affinity network, and then spliced with the old category segmentation result obtained by the original encoder-decoder network to finally obtain the complete output result.

8. The weakly supervised incremental learning semantic segmentation method according to claim 1, characterized in that: The affinity network optimizes the segmentation head results as follows: The segmentation head obtains a rough prediction effect M c , the affinity information A obtained through the affinity network is used to calculate the random walk probability matrix T, and then through iterative diffusion, the prediction results are optimized through the diffusion results, and finally the segmentation head obtains a more accurate output result.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

Images