City scene image CTTA segmentation model, segmentation method, equipment and medium

Abstract

The invention provides a city scene image CTTA segmentation model, a segmentation method, equipment and a medium. The segmentation model comprises a backbone network and a segmentation head, wherein the backbone network is formed by a plurality of network blocks which are connected in series. The front end of each network block is provided with an instance-level prompt generator, and the instance-level prompt generator generates an instance-level prompt for the network block corresponding to the instance-level prompt generator; each transformer layer in each network block is connected in parallel with a dynamic adapter, and the dynamic adapter learns domain feature invariance and domain specific knowledge. The method comprises the following steps: pre-training an initial CTTA segmentation model; in the process of predicting each target image in the city scene image, the instance-level prompt generator with unfrozen parameters and the dynamic adapter are continuously trained, and the trained dynamic adapter, the backbone network and the segmentation head are used to obtain the segmentation result of the last target image. According to the model and the method, the problems of error accumulation and disastrous forgetting can be solved, and the precision of the CCTA process is improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and relates to urban scene image segmentation technology, and specifically relates to an urban scene image CTTA (continual test-time adaptation) segmentation model, segmentation method, device and medium. Background Art

[0002] The purpose of urban scene image (USI) segmentation is to predict the pixel-level category of urban scene images according to the predefined semantic set of ground objects. The USI segmentation task has always been an indispensable step in obtaining urban observation information, and can provide important support for regional planning, land cover survey, path navigation, autonomous driving, etc.

[0003] Benefiting from the rapid development of deep learning, USIs segmentation has presented a new data-driven research paradigm. Deep learning models can obtain more accurate and robust segmentation results, thereby effectively expanding the practical application scope of USIs segmentation. Sufficient labeled samples are the prerequisite for deep models to achieve excellent performance. However, in practice, it is usually very difficult and time-consuming to obtain high-quality labeled images in some urban areas. Therefore, when the pre-trained deep model is directly applied to new urban scenes without labeled images, serious prediction errors will inevitably occur.

[0004] In recent years, in order to enhance the adaptability of deployed models to changing environments, preliminary research has been conducted on the continuous test-time adaptation (CTTA) method for urban scene images (USIs) segmentation. The CTTA method unifies the training and testing phases into a complete online learning process without accessing the source USIs, effectively reducing the computational and storage costs. Online learning means that unlabeled target images arrive in the form of a data stream, and each target image can only be accessed once. The data stream contains multiple different USIs domains to simulate the dynamic changes of real urban scenes.

[0005] The learning goal of the CTTA method is to improve the overall adaptation performance to different target USIs domains. However, in the process of continuous adaptation to multiple target domains, due to the lack of effective solutions to error accumulation and catastrophic forgetting, the accuracy of USIs segmentation using existing CTTA methods is low, making it difficult to achieve satisfactory performance. Summary of the invention

[0006] In order to solve the technical problem of low segmentation accuracy caused by lack of error accumulation and catastrophic forgetting in the existing CTTA segmentation method when segmenting urban scene images, the present invention discloses a CTTA segmentation model for urban scene images, wherein the segmentation model comprises a backbone network and a segmentation head, wherein the backbone network comprises a plurality of network blocks connected in series, and the segmentation head is connected to the last of the network blocks.

[0007] Wherein, a front end of each network block is provided with an instance-level prompt generator, and the instance-level prompt generator is used to generate an instance-level prompt for the network block corresponding thereto;

[0008] Each transformer layer in each of the network blocks is connected in parallel with a dynamic adapter, and the dynamic adapter is used to learn domain feature invariant and domain specific knowledge.

[0009] The embodiment of the present invention further provides a method for CTTA segmentation of urban scene images, which uses the urban scene image CTTA segmentation model provided in the above embodiment to perform CTTA segmentation on the urban scene image. The method includes the following steps:

[0010] S1. Pre-training the constructed initial CTTA segmentation model through source urban scene image samples to obtain a pre-trained segmentation model, wherein the pre-trained segmentation model includes a backbone network and a segmentation head with frozen parameters, and an instance-level prompt generator and a dynamic adapter with unfrozen parameters, wherein in the process of segmenting each target image in the urban scene image, the instance-level prompt generator and the dynamic adapter with unfrozen parameters are continuously trained and updated;

[0011] S2, performing semantic segmentation prediction on the current target image through the backbone network, the segmentation head and the dynamic adapter obtained after training with the previous target image in the urban scene image to obtain a predicted segmentation result;

[0012] S3, according to the current target image and its corresponding depth map, using the backbone network and the segmentation head, as well as the instance-level prompt generator and the dynamic adapter obtained after training with the previous target image, to perform semantic segmentation on the current target image to obtain an adapted segmentation result;

[0013] S4, using the predicted segmentation result and the adapted segmentation result to calculate consistency loss, and obtaining an instance-level prompt generator and a dynamic adapter used for the next target image in the urban scene image according to the goal of minimizing the consistency loss;

[0014] S5. Repeat steps S2 to S4 until a predicted segmentation result of the last target image in the urban scene image is obtained, wherein the predicted segmentation result and the adapted segmentation result both include a category corresponding to each pixel in the target image.

[0015] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned urban scene image CTTA segmentation methods when executing the computer program, so as to solve the technical problem of low segmentation accuracy caused by the lack of error accumulation and catastrophic forgetting when segmenting urban scene images in the existing CTTA segmentation method.

[0016] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program for executing any of the above-mentioned urban scene image CTTA segmentation methods to solve the technical problem of low segmentation accuracy caused by the lack of error accumulation and catastrophic forgetting when segmenting urban scene images in the existing CTTA segmentation method.

[0017] Compared with the prior art, at least one of the above-mentioned technical solutions adopted in the embodiments of this specification can achieve at least the following beneficial effects: the urban scene image CTTA segmentation model of the present invention, on the one hand, adds an instance-level prompt generator to the front end of each network block in the backbone network, and generates instance-level prompts through the instance-level prompt generator, so as to introduce more accurate prompt information into different network block levels, thereby effectively alleviating the error accumulation problem; on the other hand, a dynamic adapter is set in parallel at each transformer layer of each network block, and by learning the domain feature invariance and domain-specific knowledge of the input image or feature map, it can quickly adapt to new tasks while avoiding catastrophic forgetting problems.

[0018] Furthermore, the segmentation model of the present invention also introduces a plurality of different feature enhancement modules and adopts an uncertain estimation strategy to further improve the accuracy of the segmentation result.

[0019] A large number of experiments were conducted on drone images and street view images, and the results showed that the segmentation model of the present invention performs better than the traditional segmentation model without instance-level prompt generator and dynamic adapter, and the accuracy of the segmentation results is improved by at least 1.8% to 2.1%. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a structural diagram of the CTTA segmentation model of urban scene images disclosed in an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of generating instance-level prompts by an instance-level prompt generator of a first network block disclosed in an embodiment of the present invention;

[0023] Figure 3 Schematic diagram of generating instance-level prompts for the instance-level prompt generator of the remaining network blocks disclosed in the embodiment of the present invention

[0024] Figure 4 A schematic diagram of a dynamic adapter disclosed in an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of a short-term adapter disclosed in an embodiment of the present invention;

[0026] Figure 6 A schematic diagram of a long-term adapter write operation and a read operation disclosed in an embodiment of the present invention;

[0027] Figure 7 It is a flow chart of the CTTA segmentation method of urban scene images disclosed in an embodiment of the present invention;

[0028] Figure 8 A comparison diagram of the segmentation of drone images by two traditional methods, CoTTA and BECoTTA, disclosed in the embodiments of the present invention and the CTTA segmentation method provided by the present invention;

[0029] Fig. 9 A comparison diagram of the segmentation of street view images by two traditional methods, CoTTA and BECoTTA, disclosed in an embodiment of the present invention and the CTTA segmentation method provided by the present invention;

[0030] Fig.10 A schematic diagram of a computer device disclosed in an embodiment of the present invention;

[0031] Among them, 101 is a memory; 102 is a processor. DETAILED DESCRIPTION

[0032] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0033] The following describes the implementation methods of the present application through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and the features of the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0034] The embodiment of the present invention provides a CTTA segmentation model for urban scene images. The term "unit" or "module" used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiment is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0035] See also Figure 1 FIG. 4 is a structural block diagram of a CTTA segmentation model for urban scene images according to an embodiment of the present invention. Figure 1 As shown, the urban scene image CTTA segmentation model includes a backbone network and a segmentation head. The backbone network includes multiple network blocks connected in series, and the segmentation head is connected to the last network block. An instance-level prompt generator is provided at the front end of each network block, and the instance-level prompt generator is used to generate instance-level prompts for the corresponding network block. A dynamic adapter is connected in parallel to each transformer layer in each network block, and the dynamic adapter is used to learn domain feature invariant and domain-specific knowledge. The structure is described below.

[0036] In the embodiment of the present invention, the selection of the backbone network can be selected according to the requirements, for example, see Figure 1 As described above, the Segformer model can be selected as the backbone network, which is formed by four network blocks, each of which contains multiple transformer layers. It should be noted that the backbone network, the number of network blocks contained therein, and the number of transformer layers in each network block are not limited in the embodiments of the present invention.

[0037] In the example of the present invention, all the instance-level hint generators include a batch normalization layer and a ReLU activation function, the instance-level hint generator of the first network block in the backbone network is constructed based on a convolutional layer, and the instance-level hint generators of the remaining network blocks are constructed based on a convolutional layer and a channel attention mechanism. Figure 2 and Figure 3 As shown, Figure 2 Schematic diagram of generating instance-level prompts for the instance-level prompt generator of the first network block in the backbone network. Figure 3 Schematic diagram of the instance-level hint generator generating instance-level hints for the remaining network blocks in the backbone network.

[0038] In the examples of the present invention, see Figure 4 As shown, the dynamic adapter includes a long-term adapter and multiple short-term adapters with different ranks, wherein the long-term adapter is used to learn domain-invariant knowledge, and the short-term adapter is used to learn domain-specific knowledge.

[0039] In an improved embodiment of the present invention, see Figure 1 As shown, the segmentation model also includes a feature enhancement module, which includes multiple different feature enhancement submodules, and the feature enhancement submodule is used to enhance the features of each target image in the input urban scene image. For example, the feature enhancement module includes submodules such as magnification, reduction, angle transformation, folding, and rotation. By enhancing the target image in different ways, multiple enhancement bodies can be obtained. Each enhancement body can output a result (also called a pseudo label) when it passes through the backbone network and the segmentation head for semantic learning segmentation prediction. A more accurate prediction segmentation result can be obtained through the uncertain estimation strategy.

[0040] In the process of generating instance-level prompts, the depth map of each target image needs to be used. Therefore, in an improved embodiment of the present invention, see Figure 1 As shown, the segmentation model also includes a deep everything model DAM, which is used to generate a depth map of each target image in the urban scene image, and input the depth map into each of the instance-level prompt generators.

[0041] The embodiment of the present invention further provides a method for CTTA segmentation of urban scene images, which uses the urban scene image CTTA segmentation model provided in the above embodiment to perform CTTA segmentation on the urban scene image. Figure 7 As shown, the following steps are included:

[0042] S1. Pre-training the constructed initial CTTA segmentation model through source urban scene image samples to obtain a pre-trained segmentation model, wherein the pre-trained segmentation model includes a backbone network and a segmentation head with frozen parameters, and an instance-level prompt generator and a dynamic adapter with unfrozen parameters, wherein in the process of segmenting each target image in the urban scene image, the instance-level prompt generator and the dynamic adapter with unfrozen parameters are continuously trained and updated;

[0043] S2, performing semantic segmentation prediction on the current target image through the backbone network, the segmentation head and the dynamic adapter obtained after training with the previous target image in the urban scene image to obtain a predicted segmentation result;

[0044] S3, according to the current target image and its corresponding depth map, using the backbone network and the segmentation head, as well as the instance-level prompt generator and the dynamic adapter obtained after training with the previous target image, to perform semantic segmentation on the current target image to obtain an adapted segmentation result;

[0045] S4, using the predicted segmentation result and the adapted segmentation result to calculate consistency loss, and obtaining an instance-level prompt generator and a dynamic adapter used for the next target image in the urban scene image according to the goal of minimizing the consistency loss;

[0046] S5. Repeat steps S2 to S4 until a predicted segmentation result of the last target image in the urban scene image is obtained, wherein the predicted segmentation result and the adapted segmentation result both include a category corresponding to each pixel in the target image.

[0047] In an improved embodiment, in the above step S2, the semantic segmentation prediction of the current target image of the urban scene image is performed through the backbone network, the segmentation head and the dynamic adapter obtained after training the previous target image in the urban scene image to obtain the predicted segmentation result, including:

[0048] S21, enhancing each target image in the urban scene image in terms of scale and direction to obtain an enhanced volume sample of each target image;

[0049] S22, performing semantic segmentation prediction on each of the enhanced body samples in the current target image through the backbone network, the segmentation head and the dynamic adapter obtained after training the previous target image, to obtain a segmentation result for each enhanced body sample;

[0050] S23. By using an uncertainty estimation strategy, the segmentation results of all the enhanced volume samples are calculated to obtain a predicted segmentation result.

[0051] In an improved embodiment, in the above step S3, based on the current target image and its corresponding depth map, using the backbone network and the segmentation head, as well as the instance-level prompt generator and the dynamic adapter obtained after training with the previous target image, semantically segmenting the current target image to obtain an adapted segmentation result includes:

[0052] S31, obtaining a depth map of the current target image based on a deep everything model DAM, using an instance-level prompt generator corresponding to the first network block in the backbone network obtained after training with the previous target image, performing convolution operations, feature concatenation and feature dimensionality reduction on the current target image and its depth map, respectively, to obtain an instance-level prompt input to the first network block;

[0053] S32. For the remaining network blocks in the backbone network, perform feature transformation on the depth map to obtain a depth feature map with the same dimension as the semantic feature map output by the previous network block, use the instance-level prompt generator corresponding to the remaining network blocks in the backbone network obtained after training with the previous target image, fuse the depth feature map and the semantic feature map through the channel attention mechanism, and obtain instance-level prompts input to the remaining network blocks.

[0054] S33, based on the instance-level prompt and the current target image, learning domain-invariant knowledge and domain-specific knowledge through the backbone network and the dynamic adapter trained with the previous target image, and segmenting the semantic feature map obtained after learning through the segmentation head to obtain an adaptive segmentation result

[0055] The urban scene image CTTA segmentation model and method of the present invention, on the one hand, adds an instance-level prompt generator at the front end of each network block of the backbone network, and generates instance-level prompts through the instance-level prompt generator, so as to introduce more accurate prompt information into different network block levels, thereby effectively alleviating the error accumulation problem; on the other hand, a dynamic adapter is set in parallel at each transformer layer of each network block, and by learning the domain feature invariance and domain-specific knowledge of the input image or feature map, it can quickly adapt to new tasks while avoiding catastrophic forgetting problems.

[0056] Furthermore, the segmentation model of the present invention also introduces a plurality of different feature enhancement modules and adopts an uncertain estimation strategy to further improve the accuracy of the segmentation result.

[0057] A large number of experiments were conducted on drone images and street view images, and the results showed that the segmentation model of the present invention performs better than the traditional segmentation model without instance-level prompt generator and dynamic adapter, and the accuracy of the segmentation results is improved by at least 1.8% to 2.1%.

[0058] The following is a detailed description of the generation of the above instance-level prompts, the learning of domain-invariant knowledge and domain-specific knowledge, the optimization process of the instance-level prompt generator and dynamic adapter, and the CTTA segmentation of urban scene images through specific examples:

[0059] Step 1: The initial CTTA segmentation model can be selected with the Segformer model used as the backbone network, and consists of four network blocks.

[0060] The constructed initial CTTA segmentation model is pre-trained on the source domain (i.e., the source urban scene image sample) so that the pre-trained model can adapt to the ever-changing urban scene images and obtain more accurate segmentation results. In the process of CTTA segmentation of the collected urban scene images, each target image in the urban scene image can only be accessed once, and the segmentation result of each target image is output. In the segmentation model obtained after pre-training the initial CTTA segmentation model, the parameters of the backbone network and the segmentation head are frozen, and the parameters of the instance-level prompt generator and the dynamic adapter are not frozen. The instance-level prompt generator and the dynamic adapter can be trained and optimized in the process of accessing each target image, and the trained dynamic adapter is used when accessing the next target image.

[0061] Step 2: Use the pre-trained CTTA segmentation model to perform CTTA segmentation pre-test on urban scene images. The workflow is as follows: Figure 1 and Figure 7 As shown, the learning of each target image includes two branches: “prediction” and “adaptation”.

[0062] In the “prediction” branch, firstly, each target USI (i.e., target image, denoted as ) Perform different types of enhancements in scale and direction, input multiple enhanced samples into the frozen pre-trained model backbone network and segmentation head, and perform segmentation prediction based on the dynamic adapter optimized under the previous target US I to obtain multiple predicted segmentation results, which can also be called pseudo labels, including the category of each pixel in the target image. The final pseudo labels are recalibrated through the uncertainty estimation strategy to improve the quality of pseudo labels.

[0063] In the “adaptation” branch, the deep everything model DAM is introduced to generate the depth map of the target US I (denoted as ), the depth map of each US I is unique and contains globally coherent contextual information, which can be used as a hint of the pixel area occupied by different semantic targets. The target US I (or the feature map output by each network block) and its depth map are input into the corresponding instance-level hint generator to fuse US I / features and depth information at different levels. Unlike the existing addition operation, the instance-level hint generator designed in the embodiment of the present invention can perform nonlinear mapping of depth information and US I / features, providing more accurate knowledge guidance at the instance level. At the same time, the dynamic adapter is connected in parallel to each transformer layer of each network layer. Each dynamic adapter module includes multiple short-term adapters based on adaptive routing and a long-term adapter based on knowledge memory, which can better utilize the learned knowledge in the parameter space.

[0064] Step 3: Calculate the consistency loss based on the output of the "adaptation" branch and the pseudo-label results in the "prediction" branch, optimize the instance-level hint generator and dynamic adapter according to the goal of minimizing the consistency loss, and obtain the trained instance-level hint generator and dynamic adapter used in the next target image. By training the instance-level hint generator and dynamic adapter in the process of predicting each target image, the error accumulation problem can be effectively alleviated, and catastrophic forgetting problems can be avoided while quickly adapting to new tasks, and more accurate segmentation results can be output.

[0065] In step 2, the instance-level hint generator can implement instance-level knowledge guidance to mitigate error accumulation. The dynamic adapter can balance domain-invariant knowledge and domain-specific knowledge to mitigate catastrophic forgetting. The uncertainty estimation strategy can optimize the segmentation results output for each target image. The instance-level hint generator, dynamic adapter, and uncertainty estimation strategy are described below.

[0066] Step 21: The depth map of the target image contains contextual information about the pixel areas occupied by different semantic objects, which can guide the pre-trained model to segment more complete objects. Figure 2 As shown, the instance-level prompt generator of the present invention takes a depth map and a target image (or a feature map output by each network block) as input and outputs an instance-level prompt.

[0067] The instance-level cue generator at the initial level (i.e., the first network block) is mainly built based on convolutional layers. Figure 2 As shown, first, the target image x T and its depth map h TAfter performing different convolution operations, they are concatenated in the channel dimension. The resulting feature map with a dimension of H×W×6 is reduced in dimension through another convolution layer including 3 convolution kernels to generate an instance-level prompt P1 with a dimension of H×W×3, which can flexibly fuse channel information according to the characteristics of each target image to capture better prompt knowledge. Add prompt P1 to the target image x T and input into the first network block.

[0068] At the intermediate levels, i.e. before the second, third, fourth, etc. and other network blocks, instance-level hint-level generators are built based on convolutional layers and channel-level attention mechanisms. Figure 3 As shown, first transform the depth map into h×w×n c The feature map of is consistent with the feature dimension of the feature map output by the previous network block. The two feature maps are added by matrix, and an efficient channel attention mechanism is used to better focus on the key discriminant information. The channel attention mechanism can be expressed as formula (1):

[0069]

[0070] In formula (1), z block represents the eigenvector obtained after matrix addition, w att represents the weight of each channel, σ is the sigmoid activation function. GAP represents the global average pooling layer, and C1D represents the one-dimensional convolution layer. The kernel size k of C1D is determined by the channel dimension n. c Adaptively determine, as shown in the following formula (2):

[0071]

[0072] Among them, in formula (2), the function ||odd outputs the nearest odd number, and γ and b are hyperparameters. Through the local cross-channel interaction defined by k, the adopted attention mechanism can better realize the fusion of deep information and feature vectors. Finally, the feature z block Multiply by weight w att , to form the instance level prompt p at the intermediate level i (i=2, 3, 4).

[0073] It is worth noting that a batch normalization layer and a ReLU activation function are added after each convolutional layer to further improve the efficiency and stability of parameter optimization.

[0074] Step 22, see Figure 4As shown, it is an overall schematic diagram of the dynamic adapter, which is connected in parallel to each transformer layer in each network block. Each dynamic adapter includes multiple short-term adapters with different ranks and one long-term adapter. Compared with the practice of adding only one adapter or multiple identical adapters, the short-term adapter and long-term adapter designed in the present invention can dynamically and explicitly learn domain-specific and domain-invariant feature knowledge. The output features of each layer are calculated as the addition of the output of the frozen transformer layer, the short-term adapter output, and the long-term adapter output to achieve rapid adaptation to the new USIs domain (target image) while mitigating catastrophic forgetting.

[0075] Step 221, learning domain-specific knowledge, which can be considered as short-term knowledge, can facilitate faster adaptation to new target USIs domains. In practice, achieving accurate adaptation to each target domain requires diverse knowledge, not just one type of knowledge. In most cases, the same knowledge contributes to different target domains to different degrees. Rather than using only fixed-structure adapters, the present invention first establishes n a short-term adapters with different ranks, such as Figure 5 Each short-term adapter actually consists of a and The rank of the two linear projection layers is d r Usually smaller than the input feature dimension d. Considering that different ranks can increase the degree of learning of different features, it is n a The adapters are assigned different ranks, i.e. To increase the diversity and richness of the learned knowledge. Then, according to the specific characteristics of each target domain, the learned diverse knowledge is combined with the adaptive routing process to make it more discriminative and robust.

[0076] like Figure 5 As shown in Figure 1, the adaptive routing process applied to different adapters consists of two stages: “dispatch” and “combination”. Formally, the input token sequence is recorded as where n t is the number of tokens, and d is the feature dimension. Before each network block, an embedding function is used to generate u based on the feature map z, and the dispatch matrix w is learned. dis , converting the tokens sequence into a shorter slot sequence For details, please refer to the following formula (3) and formula (4):

[0077]

[0078] Among them, in formula (3) and formula (4), is a learnable parameter. The matrix w disIt is actually the result of applying the softmax function to the columns of uφ, which is responsible for converting the length n t The u of the fusion is n s of n s The value is greater than n t Much smaller, calculated as n s =n a ×n p , where n p is the number of slots processed by each short-term adapter. This not only adaptively combines the features between different tokens and enriches the representation capability, but also shortens the length of the sequence to be processed and optimizes the computational efficiency.

[0079] Apply each short-term adapter to n p corresponding slots, and get the output slot sequence For details, please refer to the following formula (5):

[0080]

[0081] In formula (5), i indexes the slot sequence. Index the short-term adapters. So far, short-term adapters with different ranks have learned different knowledge. Finally, the diverse knowledge is adaptively combined to convert the slot sequence Restore to the output token sequence v, see the following formula (6) and formula (7) for details:

[0082]

[0083] Among them, in formula (6) and formula (7), the learnable matrix It is actually the result of applying the softmax function to the rows of uφ. The parameter φ plays an important role in the whole adaptive routing process. In the experiment, it is implemented by a trainable fully connected layer, which can be continuously optimized during the CTTA process to improve the selection and integration of diverse knowledge.

[0084] Step 222, learning long-term, domain-invariant knowledge can effectively alleviate the catastrophic forgetting problem in the CTTA process. The present invention captures common knowledge across multiple target USIs domains through a long-term adapter based on knowledge memory. Similar to the short-term adapter, the two linear projection layers in the long-term adapter can be expressed as and Write and read operations are the basic means for long-term adapters to interact with knowledge memory, such as Figure 6 shown.

[0085] The writing operation occurs in the “adaptation” branch, which aims to write knowledge from different USIs domains into the knowledge memory during the CTTA process. Specifically, the long-term adapter parameters corresponding to the given time step t = 1 are and The knowledge memory is initialized to Subsequently, at any time step t, the knowledge memory is updated as shown in equation (8):

[0086]

[0087] In formula (8), α controls the update momentum. As the time step t increases, the proposed method can adjust the parameters based on the USIs of different target domains. and By optimizing, the knowledge in different USIs domains can be written into the knowledge memory, making the learned knowledge memory more and more universal.

[0088] Read operations occur on the "prediction" branch, which directly uses the knowledge memory The output of the long-term adapter replaces the output of the original long-term adapter, which is more robust through the learned domain-invariant knowledge and can provide more accurate predictions when the target USIs domain appears again.

[0089] Step 23: In the self-training based CTTA process, the segmentation accuracy can be improved by using the uncertainty estimation strategy based on multiple predictions. Figure 1 As shown, in the “prediction” branch, each target USI can be enhanced in spatial scale and direction to generate n aug By generating multiple enhanced samples from a set of enhanced samples, we can get many pseudo labels, and use the uncertainty estimation strategy to get a final pseudo label result, which is recorded as The frozen segmentation head outputs the logit prediction corresponding to the above sample, i.e. Meanwhile, the mean of the logit predictions is calculated as The prediction uncertainty estimation is shown in the following formula (9):

[0090]

[0091] Among them, in formula (9), d unc is the uncertainty matrix obtained, d unc Each uncertainty value in is the sum of the KL divergence between the average logit and the logit of each enhanced sample at the same pixel location. If the predicted logits of different enhanced samples differ greatly in a certain pixel area, then d unc The corresponding position in the uncertainty matrix is ​​larger, indicating that the prediction in this area has higher uncertainty. uncConverted to pixel-by-pixel weights, the pseudo-labels are recalibrated as shown in equation (10):

[0092] w pl =exp(-μd unc ), (10)

[0093] In formula (10), μ represents the degree of decrease of the soft weight. Formula (10) can be used to assign low weight values ​​to pixels with high uncertainty, thereby reducing the negative impact of incorrect prediction on CTTA.

[0094] In step 3, the instance-level hint generator and dynamic adapter can be trained and optimized through consistency loss, so that the segmentation result of the next target image is more accurate. The loss between the output result of the "adaptation" branch and the pseudo-label result in the "prediction" branch is It can be expressed by the following formula (11):

[0095]

[0096] Among them, in formula (11), Reason Generated pseudo-label, y c is the segmentation result of the “adaptation” branch, and C represents the number of categories.

[0097] In the embodiment of the present invention, two traditional methods, CoTTA and BECoTTA, are used to segment UAV images (UAV) and street view images (SV-Easy) with the CTTA segmentation method provided by the present invention. The segmentation results of the UAV images are shown in Figure 2. Figure 8 As shown in Figure 2, the segmentation results of the SV-Easy image are as follows: Fig. 9 shown.

[0098] See also Figure 8 It can be seen from the results that the method provided by the present invention can generate more accurate segmentation maps and effectively reduce noise and false predictions. At the same time, it is worth noting that the introduction of depth information as an instance-level hint significantly improves the segmentation results of objects located at different depths. Relying only on shape or texture information, some ground objects that are similar to the surrounding environment, such as Building, Tree, and Clutter, are prone to classification confusion problems. In contrast, the introduction of depth information can provide contextual information of the pixel areas occupied by different objects, thereby guiding the model to perform more accurate and complete segmentation of objects.

[0099] See also Fig. 9As shown, the method of the present invention can obtain more accurate segmentation maps, which benefits from the ability to learn domain-invariant knowledge and domain-specific knowledge. At the same time, the instance-level hints based on the depth map further optimize the segmentation results of the foreground objects. For example, in the first row, the proposed method recognizes a more complete fence object than other methods. In the third row, the edge of the traffic sign object is smoother and closer to the real shape. In addition, the segmentation results of objects that are not in the foreground position are also significantly improved, such as the rod-shaped objects in the second row.

[0100] In this embodiment, a computer device is provided, such as Fig.10 As shown, it includes a memory 101, a processor 102 and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned urban scene image CTTA segmentation methods is implemented.

[0101] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0102] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any of the above-mentioned urban scene image CTTA segmentation methods.

[0103] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include transitory media such as modulated data signals and carrier waves.

[0104] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0105] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A CTTA segmentation model for urban scene images, characterized in that: The segmentation model includes a backbone network and a segmentation head, the backbone network includes a plurality of network blocks connected in series, and the segmentation head is connected to the last network block; An instance-level prompt generator is provided at the front end of each network block, and the instance-level prompt generator is used to generate an instance-level prompt for the network block corresponding thereto; Each transformer layer in each of the network blocks is connected in parallel with a dynamic adapter, and the dynamic adapter is used to learn domain feature invariant and domain specific knowledge.

2. The urban scene image CTTA segmentation model according to claim 1 is characterized in that: All of the instance-level prompt generators include a batch normalization layer and a ReLU activation function. The instance-level prompt generator of the first network block in the backbone network is constructed based on a convolutional layer, and the instance-level prompt generators of the remaining network blocks are constructed based on a convolutional layer and a channel attention mechanism.

3. The urban scene image CTTA segmentation model according to claim 1, characterized in that: The dynamic adapter includes a long-term adapter and multiple short-term adapters with different ranks, the long-term adapter is used to learn domain-invariant knowledge, and the short-term adapter is used to learn domain-specific knowledge.

4. The urban scene image CTTA segmentation model according to claim 1, characterized in that: The segmentation model also includes a feature enhancement module, which includes a plurality of different feature enhancement sub-modules, and the feature enhancement sub-modules are used to perform feature enhancement on each target image in the input urban scene image.

5. The urban scene image CTTA segmentation model according to claim 1, characterized in that: The segmentation model also includes a deep everything model DAM, which is used to generate a depth map of each target image in the urban scene image, and input the depth map into each of the instance-level prompt generators.

6. A CTTA segmentation method for urban scene images, characterized in that: Using the urban scene image CTTA segmentation model according to any one of claims 1 to 5 to perform CTTA segmentation on the urban scene image, the method comprising: Pre-training the constructed initial CTTA segmentation model through source city scene image samples to obtain a pre-trained segmentation model, wherein the pre-trained segmentation model includes a backbone network and a segmentation head with frozen parameters, and an instance-level prompt generator and a dynamic adapter with unfrozen parameters; Performing semantic segmentation prediction on the current target image to obtain a predicted segmentation result through the backbone network, the segmentation head and the dynamic adapter obtained after training with the previous target image in the urban scene image; According to the current target image and its corresponding depth map, using the backbone network and the segmentation head, as well as the instance-level prompt generator and the dynamic adapter obtained after training with the previous target image, semantically segment the current target image to obtain an adapted segmentation result; The predicted segmentation result and the adapted segmentation result are used to calculate the consistency loss, and the instance-level prompt generator and the dynamic adapter used for the next target image in the urban scene image are obtained according to the goal of minimizing the consistency loss.

7. The urban scene image CTTA segmentation method according to claim 6, characterized in that: The method of performing semantic segmentation prediction on the current target image of the urban scene image through the backbone network, the segmentation head and the dynamic adapter obtained after training the previous target image in the urban scene image to obtain the predicted segmentation result includes: Enhance each target image in the urban scene image in terms of scale and direction to obtain an enhanced volume sample of each target image; Performing semantic segmentation prediction on each of the enhanced body samples in the current target image through the backbone network, the segmentation head and the dynamic adapter obtained after training the previous target image, to obtain a segmentation result of each enhanced body sample; Through the uncertainty estimation strategy, the predicted segmentation results are calculated for all the enhanced volume sample segmentation results.

8. The urban scene image CTTA segmentation method according to claim 6, characterized in that: The method of performing semantic segmentation on the current target image to obtain an adapted segmentation result by using the backbone network and the segmentation head, and the instance-level prompt generator and the dynamic adapter obtained after training with the previous target image according to the current target image and its corresponding depth map includes: Based on the deep everything model DAM, a depth map of the current target image is obtained, and an instance-level prompt generator corresponding to the first network block in the backbone network obtained after training with the previous target image is used to perform convolution operations, feature concatenation and feature dimension reduction on the current target image and its depth map to obtain an instance-level prompt input to the first network block; For the remaining network blocks in the backbone network, the depth map is feature transformed to obtain a depth feature map with the same dimension as the semantic feature map output by the previous network block, and the instance-level prompt generator corresponding to the remaining network blocks in the backbone network obtained after training with the previous target image is used to fuse the depth feature map and the semantic feature map through a channel attention mechanism to obtain instance-level prompts input to the remaining network blocks; According to the instance-level prompt and the current target image, domain-invariant knowledge and domain-specific knowledge are learned through the backbone network and the dynamic adapter trained with the previous target image, and the semantic feature map obtained after learning is segmented by the segmentation head to obtain an adaptive segmentation result.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the urban scene image CTTA segmentation method according to any one of claims 6 to 8 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the urban scene image CTTA segmentation method according to any one of claims 6 to 8.

Images