Weakly supervised incremental learning semantic segmentation method, device and medium

Abstract

The application discloses a weakly supervised incremental learning semantic segmentation method and device and a medium, wherein the method comprises the following steps: obtaining a picture to be segmented; inputting the picture to be segmented into a trained semantic segmentation model to output a segmentation result; wherein the semantic segmentation model comprises an encoder, a decoder, an affinity network and a light multi-scale segmentation head; the encoder is used for feature extraction of an input image, and the decoder is used for category prediction according to the extracted features; the affinity network is used for obtaining affinity information according to the features extracted by the encoder, as an affinity matrix for optimizing the segmentation head result; and the light multi-scale segmentation head is used for obtaining a rough new category segmentation result according to the output of the encoder. The application introduces the affinity network, the affinity matrix obtained through the affinity value is used for refining the rough segmentation result, the effect is better, and the affinity network is directly designed into the network as a module in the framework, end-to-end training and prediction are realized, and the effect is good.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and in particular to a weakly supervised incremental learning semantic segmentation method, device, and medium. Background Technology

[0002] With the development of semantic segmentation technology, it has played a vital role in various tasks across different industries. However, despite the significant achievements of existing semantic segmentation methods in many application scenarios, in practical applications, models often need to be updated over time to handle new types and scenarios. Furthermore, the pixel-by-pixel annotation process required for semantic segmentation is both expensive and time-consuming, especially for irregularly shaped objects. While annotations for older categories can be relatively easily obtained through experience, newly discovered categories still require re-annotation pixel by pixel, making the learning of these new categories more difficult.

[0003] Previously, the two problems mentioned above were addressed by using weakly supervised learning to solve the lack of pixel-level labels and incremental learning techniques to solve the problem of learning new categories. However, in production environments, these two problems often arise simultaneously, and new categories also require new annotations, which consume a significant amount of manual labor. In recent years, a new research direction has emerged: weakly supervised incremental learning frameworks. These frameworks can simultaneously solve both problems mentioned above in semantic segmentation tasks. However, these frameworks are still in their early stages, with an imperfect structure, and currently only learn new categories through simple class activation layers, such as... Figure 1 As shown, the learning effect is poor and the efficiency is low; the learning effect of new categories needs to be improved. Summary of the Invention

[0004] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide an end-to-end weakly supervised incremental learning semantic segmentation method, device and medium that combines affinity network optimization.

[0005] The first technical solution adopted in this invention is:

[0006] A weakly supervised incremental learning semantic segmentation method includes the following steps:

[0007] Obtain the image to be segmented;

[0008] Input the image to be segmented into the trained semantic segmentation model, and output the segmentation result;

[0009] The semantic segmentation model includes an encoder, a decoder, an affinity network, and a lightweight multi-scale segmentation head. The encoder is used to extract features from the input image, and the decoder is used to predict the category based on the extracted features. The affinity network is used to obtain affinity information based on the features extracted by the encoder, which serves as the affinity matrix for optimizing the segmentation head result. The lightweight multi-scale segmentation head is used to obtain a coarse new category segmentation result based on the encoder output.

[0010] Furthermore, the semantic segmentation model is trained in the following manner:

[0011] Apply the weights obtained from training the encoder-decoder structure in step t-1 to step t;

[0012] During training in step t, the weights of the encoder-decoder are frozen, and the affinity network and lightweight multi-scale segmentation head are trained using labeled data of the old categories. Unlabeled data is input, and the prediction results of the lightweight multi-scale segmentation head are enhanced by the affinity matrix obtained by the affinity network. The result is the prediction result of the semantic segmentation model for the new category in the unlabeled data.

[0013] The prediction results of the new category are concatenated with the prediction results of the encoder-decoder for the old category to obtain the prediction results of the semantic segmentation model for all categories of unlabeled data. This prediction result is then supervised, enabling the semantic segmentation model to incrementally and weakly learn new categories.

[0014] Furthermore, the affinity network is specifically used to extract affinity information at relevant positions from feature information at different levels extracted during the encoder encoding process; the feature information at different levels includes deep feature information, sub-layer feature information, and shallow feature information.

[0015] Furthermore, the input feature map during the training of the affinity network consists of feature information at different depth levels generated during the decoder process, and the affinity label is the affinity matrix composed of affinity values ​​between different coordinates in the results from the segmentation head in the frame.

[0016] Furthermore, the affinity tag is obtained by weighted average of the affinity value calculated based on L1 distance and the cosine similarity calculated based on cosine similarity;

[0017] The formula for calculating the affinity label is as follows:

[0018]

[0019] In the formula, A ij L1 A is the affinity value calculated based on the L1 distance. ij cosA is the cosine similarity calculated based on cosine similarity. ij The affinity value is the weighted average of the two; f aff (x i ,y i ) is the position coordinate (x i ,y i The affinity value of f aff (x j ,y j ) is the position coordinate (x j ,y j The affinity value of ) is ; C is the number of channels of the feature map to be calculated, and the subscript c is the feature channel to be calculated; ‖‖1 is the L1 distance; λ1 and λ2 are weight coefficients.

[0020] Furthermore, the affinity network calculates affinity values ​​only for coordinate pairs within a radius of γ, and the set S of coordinate pairs for calculating affinity is:

[0021]

[0022] In the formula, d((x i ,y i ),(x j ,y j )) represents coordinates (x) i ,y i ) and coordinates (x) j ,y j The distance between them;

[0023] Divide S into three subsets and aggregate the losses obtained from each subset:

[0024] First, divide S into two subsets: positive and negative pairs.

[0025]

[0026]

[0027] In the formula, Affinity value A ij The result of binarization;

[0028] Based on the category attributes of the coordinate pairs corresponding to the segmentation results from the semantic segmentation head, further... Decomposed into background-related and related to prospects The formula for calculating the cross-entropy loss for each subset is as follows:

[0029]

[0030] The affinity network is trained under supervision using a loss function, enabling it to extract and generate affinity information from the encoder process, forming an affinity matrix.

[0031] Furthermore, the random walk probability matrix T is calculated using the affinity matrix A, as shown in the following formula:

[0032]

[0033] In the formula, A° β The affinity matrix is ​​the Hadamard power, β is a hyperparameter greater than 1; D is a diagonal matrix, and the focal element is D. ii The value is calculated using the formula above;

[0034] One-to-one propagation is achieved by multiplying the random walk probability matrix T by the CAM; the semantic segmentation model iteratively performs this propagation until a predefined number of iterations t; the formula is as follows:

[0035]

[0036] In the formula, M c The output of the segmentation head is vectorized and optimized using the diffused matrix T. The mechanism of learning semantic affinity and semantic propagation is achieved through an affinity network.

[0037] Furthermore, the lightweight multi-scale segmentation head includes multiple depthwise separable convolutions at different scales. By using depthwise separable convolution kernels at different scales, the receptive field of the model is increased while reducing the network architecture and the number of parameters. The feature extractions at different scales are then fused together through summation. A 1*1 convolution reduces the number of channels to extract deep semantic information. The deep semantic information is then fused with the shallow semantic information extracted from only a 3*3 convolution. Finally, the segmentation result is extracted from the rich semantic knowledge obtained by this fusion.

[0038] After obtaining a coarse new category segmentation result from the segmentation head, this coarse data is refined using the affinity matrix obtained through the affinity network, and then concatenated with the old category segmentation result obtained from the original encoder-decoder network to finally obtain the complete output result.

[0039] Furthermore, the affinity network optimizes the segmentation head results as follows:

[0040] The segmentation head produces a coarse prediction result M. c By using the affinity information A obtained through the affinity network, the random walk probability matrix T is calculated. Then, through iterative diffusion, the prediction results are optimized based on the diffusion results, ultimately resulting in a more accurate output from the segmentation head.

[0041] The second technical solution adopted in this invention is:

[0042] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a weakly supervised incremental learning semantic segmentation method as described above.

[0043] The third technical solution adopted in this invention is:

[0044] A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a weakly supervised incremental learning semantic segmentation method as described above.

[0045] The fourth technical solution adopted in this invention is:

[0046] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned weakly supervised incremental learning semantic segmentation method.

[0047] The beneficial effects of this invention are as follows: This invention introduces an affinity network, which refines the coarse segmentation results using an affinity matrix obtained from affinity values, resulting in better performance. Furthermore, the affinity network is directly designed as a module within the framework, achieving end-to-end training and prediction with superior results. Additionally, decoding deep semantic information using a lightweight multi-scale segmentation head is more efficient and accurate than using a single class activation layer. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of an existing weakly supervised incremental learning framework;

[0050] Figure 2 This is a schematic diagram of the weakly supervised incremental learning framework provided in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the affinity network in an embodiment of the present invention;

[0052] Figure 4 This is a structural diagram of the lightweight semantic segmentation head in an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram illustrating the relationship between the affinity network and the segmentation head within the framework of this invention. Detailed Implementation

[0054] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0055] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0056] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0057] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0058] Example 1

[0059] This embodiment provides a weakly supervised incremental learning semantic segmentation method, including the following steps:

[0060] S1. Obtain the image to be segmented;

[0061] S2. Input the image to be segmented into the trained semantic segmentation model and output the segmentation result;

[0062] The semantic segmentation model includes an encoder, a decoder, an affinity network, and a lightweight multi-scale segmentation head. The encoder is used to extract features from the input image, and the decoder is used to predict the category based on the extracted features. The affinity network is used to obtain affinity information based on the features extracted by the encoder, which serves as the affinity matrix for optimizing the segmentation head result. The lightweight multi-scale segmentation head is used to obtain a coarse new category segmentation result based on the encoder output.

[0063] See Figure 1 Existing frameworks learn new categories using only simple class activation layers, resulting in poor learning performance and low efficiency. (See also...) Figure 2 In this embodiment, the affinity network and lightweight multi-scale segmentation head are introduced into the weakly supervised incremental learning framework. The affinity matrix obtained by the affinity value is used to refine the coarse segmentation results, which has a better effect. Moreover, the affinity network is directly designed into the network as a module in the framework to achieve end-to-end training and prediction, which has a good effect.

[0064] Specifically, see Figure 2 The working principle of the framework provided in this embodiment is as follows: the weights obtained from training the encoder-decoder structure in step t-1 are applied to step t; during the training in step t, the affinity network and the multi-scale segmentation head are first trained using labeled data of the old categories. The weights of the encoder-decoder need to be frozen during this training stage; then, unlabeled data is input into the network, and the prediction results of the lightweight multi-scale segmentation head are enhanced by the affinity matrix obtained by the affinity network. The result obtained is the framework's prediction result for the new category in the unlabeled data. At the same time, it is spliced ​​with the previous prediction results of the encoder-decoder for the old category to obtain the recognition effect of the entire framework for all categories of the unlabeled data. The prediction result is supervised so that the entire framework incrementally and weakly learns the new category.

[0065] The framework proposed in this embodiment will be further explained below with reference to the accompanying drawings and specific implementation methods.

[0066] (a) Affinity Network

[0067] This framework extracts feature information from different levels during the encoder encoding process, mixes this information and inputs it into the affinity network. The affinity information is obtained through the prediction of the affinity network, and this affinity information is used as the optimization matrix to optimize the segmentation head result.

[0068] Affinity networks are required to extract affinity information at relevant locations from feature information input at different levels. The accuracy of this prediction is guaranteed by effective training of the affinity network.

[0069] See Figure 3 The affinity network is trained by using input feature maps and affinity labels to learn how to predict the relationship between pixels. The input feature maps selected by the affinity network in the framework of this invention are feature information at different depth levels generated during the decoder process, and the affinity labels are the affinity matrix composed of affinity values ​​between different coordinates in the result of the segmentation head in the framework.

[0070] The affinity label designed in this framework is obtained by calculating the affinity between a pair of feature vectors using a weighted average of their L1 distance and spatial cosine similarity.

[0071] A ij L1 =exp{-||f aff (x i ,y i )-f aff (x j ,y j )||1}

[0072]

[0073] Among them, A ij L1 A is the affinity value calculated based on the L1 distance. ij cos A is the cosine similarity calculated based on cosine similarity. ij The affinity value is the weighted average of the two values.

[0074] Because the correlation between coordinate pairs at long distances weakens, and a trade-off needs to be made between computational cost and the effect gain brought by semantic information, the affinity network only calculates affinity values ​​for coordinate pairs within a radius of γ. This embodiment adopts... Let this represent the set of coordinate pairs we use to calculate affinity:

[0075]

[0076] Due to class imbalance, directly using information from these sets to train affinity networks is not advisable. Typically, The class distribution in the image is clearly biased towards positive classes because negative pairs typically only exist at the edges of objects. Simultaneously, within the subset of positive pairs, the number of background pairs is significantly greater than the number of object pairs, because in many images the background area is larger than the object area. To address this issue, this embodiment will... Divide into three subsets and aggregate the losses obtained from each subset. First, Divided into two subsets: positive and negative pairs:

[0077]

[0078] And further based on the category attributes of the coordinate pairs corresponding to the segmentation results of the lightweight semantic segmentation head, Decomposed into background-related and related to prospects Therefore, the cross-entropy loss for each subset is calculated as follows:

[0079]

[0080] This framework uses the above loss function to supervise the training of the affinity network, enabling the affinity network to extract and generate affinity information from the encoder process to form an affinity matrix.

[0081] In previous weakly supervised semantic segmentation models, affinity networks were often auxiliary networks independent of the segmentation network. However, in the framework of this invention, the affinity module is directly embedded into the framework to achieve end-to-end training and prediction, rather than being treated as a separate network. This allows us to directly apply affinity results to the model's prediction results. The local semantic affinity predicted by the affinity network is transformed into an affinity matrix, which enables random walks to recognize image boundaries and encourages them to spread activation scores within these boundaries. We use the affinity matrix A to calculate the random walk probability matrix T, as shown in the following formula:

[0082]

[0083] Among them, A° β The affinity matrix is ​​the Hadamard power, and the diagonal matrix is ​​a row normalized version of it. One-to-one propagation is achieved by multiplying the random walk matrix by the CAM. This model iteratively performs this propagation until a predefined number of iterations t. The formula is as follows:

[0084]

[0085] Where M c The output of the segmentation head is vectorized and optimized using the diffused matrix T. The mechanism of learning semantic affinity and propagating semantics through affinity network not only significantly improves the performance of weakly supervised semantic segmentation without additional data or stronger supervision, but also enables the final segmentation model to more accurately capture the complete shape of the target object, thus improving the weakly supervised incremental learning framework's ability to understand new knowledge.

[0086] (II) Lightweight Multi-Scale Segmentation Head

[0087] See Figure 4 The lightweight multi-scale segmentation head designed in this framework is composed of multiple separable convolutions at different scales. By using separable convolution kernels at different scales, the receptive field of the model is increased while reducing the network architecture and the number of parameters. The feature extractions at different scales are then fused together through summation. A 1*1 convolution reduces the number of channels to extract deep semantic information. The deep semantic information is then fused with the shallow semantic information extracted from a single 3*3 convolution. Finally, the segmentation result is extracted from the rich semantic knowledge obtained by this fusion.

[0088] A rough new category segmentation result is obtained from this segmentation head. Then, this rough data is refined through the affinity matrix obtained by the affinity network, and then concatenated with the old category segmentation result obtained by the original encoder-decoder network to finally obtain the complete output result. The entire network structure is supervised learning through this final result, so that the framework has the ability to segment new category data.

[0089] (III) The Influence of Affinity Network on Segmentation Results

[0090] See Figure 5 The segmentation head produces a coarse prediction result M. c By using the affinity information A obtained through the affinity network, the random walk probability matrix T is calculated. Then, through iterative diffusion, the prediction results are optimized using the diffusion results, ultimately leading to a more accurate output from the segmentation head. In this weakly supervised incremental learning framework, the segmentation head predicts new categories within this incremental framework. Improved performance helps generate more accurate pseudo-labels in subsequent processes. Finally, supervision of the entire encoder-decoder model enables the framework to perform more effective incremental learning on unlabeled data.

[0091] Example 2

[0092] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to achieve the following: Figure 2 This illustrates a weakly supervised incremental learning semantic segmentation method.

[0093] It is understood that the memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the stored data area may store data created according to the use of the server, etc.

[0094] A processor may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU) and Modem. The CPU primarily handles the operating system and applications; the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0095] Since this electronic device is an electronic device corresponding to a weakly supervised incremental learning semantic segmentation method in this embodiment of the invention, and the principle of solving the problem by this electronic device is similar to that of this method, the implementation of this electronic device can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0096] Example 3

[0097] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to achieve the following: Figure 2 This illustrates a weakly supervised incremental learning semantic segmentation method.

[0098] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0099] Since this storage medium is the storage medium corresponding to a weakly supervised incremental learning semantic segmentation method in this embodiment of the invention, and the principle of this storage medium in solving the problem is similar to that of this method, the implementation of this storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0100] Example 4

[0101] In some possible implementations, various aspects of the methods of the embodiments of the present invention can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of a weakly supervised incremental learning semantic segmentation method according to various exemplary embodiments of this application as described above. The executable computer program code or "code" for performing the various embodiments can be written in high-level programming languages ​​such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0102] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0103] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0104] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A weakly supervised incremental learning semantic segmentation method, characterized in that, Includes the following steps: Obtain the image to be segmented; Input the image to be segmented into the trained semantic segmentation model, and output the segmentation result; The semantic segmentation model includes an encoder, a decoder, an affinity network, and a lightweight multi-scale segmentation head. The encoder is used to extract features from the input image, and the decoder is used to predict the category based on the extracted features. The affinity network is used to obtain affinity information based on the features extracted by the encoder, which serves as an affinity matrix to optimize the segmentation head result. The lightweight multi-scale segmentation head is used to obtain a coarse new category segmentation result based on the encoder output. The semantic segmentation model is trained in the following way: Apply the weights obtained from training the encoder-decoder structure in step t-1 to step t; During training in step t, the weights of the encoder-decoder are frozen, and the affinity network and lightweight multi-scale segmentation head are trained using labeled data of the old categories. Unlabeled data is input, and the prediction results of the lightweight multi-scale segmentation head are enhanced by the affinity matrix obtained by the affinity network. The result is the prediction result of the semantic segmentation model for the new category in the unlabeled data. The prediction results of the new category are concatenated with the prediction results of the encoder-decoder for the old category to obtain the prediction results of the semantic segmentation model for all categories of unlabeled data. The prediction results are then supervised so that the semantic segmentation model can incrementally and weakly learn new categories. The input feature map during the training of the affinity network consists of feature information at different depth levels generated during the decoder process, and the affinity label is the affinity matrix composed of affinity values ​​between different coordinates in the results of the segmentation head in the frame.

2. The weakly supervised incremental learning semantic segmentation method according to claim 1, characterized in that, The affinity tag is obtained by weighted average of the affinity value calculated based on L1 distance and the cosine similarity calculated based on cosine similarity. The formula for calculating the affinity label is as follows: In the formula, The affinity value is calculated based on the L1 distance. The cosine similarity is calculated based on the cosine similarity. The affinity value is the weighted average of the two. The position coordinates are Affinity value The position coordinates are Affinity value; The number of channels in the feature map to be calculated, with subscripts. For the calculated feature channels; The distance is L1; These are the weighting coefficients.

3. The weakly supervised incremental learning semantic segmentation method according to claim 2, characterized in that, The affinity network is only for radii of [missing value]. The set of coordinate pairs within the range is used to calculate the affinity value. for: In the formula, Representing coordinates and coordinates The distance between them; Will Divide into three subsets and aggregate the losses obtained from each subset: First of all Divided into two subsets: positive and negative pairs: In the formula, Affinity value The result of binarization; Based on the category attributes of the coordinate pairs corresponding to the segmentation results from the semantic segmentation head, further... Decomposed into background-related and related to prospects The formula for calculating the cross-entropy loss for each subset is as follows: The affinity network is trained under supervision using a loss function, enabling it to extract and generate affinity information from the encoder process, forming an affinity matrix.

4. The weakly supervised incremental learning semantic segmentation method according to claim 1, characterized in that, Using affinity matrix Calculate the random walk probability matrix The formula is as follows: In the formula, The Hadamard power of the affinity matrix. For hyperparameters greater than 1; It is a diagonal matrix; Through the random walk probability matrix Multiplying by CAM achieves one-to-one propagation; the semantic segmentation model iteratively performs this propagation until a predefined number of iterations is reached. The formula is as follows: In the formula, The output of the segmentation head is vectorized and then processed by the diffusion matrix. We optimize the mechanism of learning semantic affinity and propagating semantics through affinity networks.

5. The weakly supervised incremental learning semantic segmentation method according to claim 1, characterized in that, The lightweight multi-scale segmentation head includes multiple separable convolutions at different scales. By using separable convolution kernels at different scales, the receptive field of the model is increased while reducing the network architecture and the number of parameters. The feature extractions at different scales are then fused together through summation. A 1*1 convolution reduces the number of channels to extract deep semantic information. The deep semantic information is then fused with the shallow semantic information extracted from only a 3*3 convolution. Finally, the segmentation result is extracted from the rich semantic knowledge obtained by this fusion. After obtaining a coarse new category segmentation result from the segmentation head, this coarse data is refined using the affinity matrix obtained through the affinity network, and then concatenated with the old category segmentation result obtained from the original encoder-decoder network to finally obtain the complete output result.

6. The weakly supervised incremental learning semantic segmentation method according to claim 1, characterized in that, The affinity network optimizes the segmentation head results as follows: The segmentation head produces coarse prediction results. Affinity information obtained through affinity networks The random walk probability matrix is ​​calculated. Then, through iterative diffusion, the prediction results are optimized using the diffusion results, ultimately resulting in a more accurate output from the segmentation head.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method as described in any one of claims 1 to 6.

Images