A domain generalization semantic segmentation method and system, electronic device and storage medium

Abstract

The application discloses a domain generalization semantic segmentation method and system, electronic equipment and a storage medium, the method comprising: loading pre-training parameters of a visual basic large model to generate a virtual simulation dataset; extracting intermediate features of an input image, performing cross-attention fusion with word vectors, inputting a segmentation head for semantic segmentation prediction to obtain a class prediction result corresponding to each pixel; randomly assigning a class label not to participate in loss calculation at each small batch iteration training; calculating the loss value between the predicted class and the real class; setting the corresponding word vector gradient to zero in the index mode during gradient back propagation; iteratively training to obtain a target model; and performing domain generalization semantic segmentation processing on target data in a real-world target scene to be segmented to obtain a segmentation result. The application can reduce data labeling costs, achieve generalization segmentation performance of the model on real-world scenes, and can be widely applied in the technical field of computer vision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a domain generalization semantic segmentation method and system, an electronic device and a storage medium. BACKGROUND

[0002] Deep Neural Networks (DNNs) have achieved remarkable success in autonomous vehicles and related tasks in recent years. This is mainly due to the powerful feature learning ability of DNNs, which enables them to automatically extract meaningful features from massive amounts of data. However, the training of DNNs usually relies on the Independent and Identically Distributed (IID) assumption, i.e., the training data and test data come from the same distribution. Once the test data distribution changes, the model performance may drop significantly, which is known as domain shift. The existence of domain shift limits the generalization ability of DNNs, making them perform poorly in different scenarios or data distributions, especially in practical applications, where this problem is more prominent.

[0003] In the semantic segmentation task, the problem of domain shift is particularly serious. The goal of semantic segmentation is to accurately divide each pixel in the input image into a specific semantic category, in order to achieve high-level visual understanding. This task not only requires the model to have good feature extraction ability, but also requires rich labeled data to support. However, the labeling cost of semantic segmentation tasks is extremely high. Unlike image classification or object detection tasks, semantic segmentation requires each pixel to be labeled one by one, which is a complex and time-consuming task, especially when dealing with high-resolution images and complex scenes, the labeling process may take hours or even days. With the development of 3D modeling and simulation technology, this problem has been significantly alleviated. Modern 3D engines (such as Unreal Engine, Unity, etc.) can realistically simulate real-world scenes while supporting automatic generation of pixel-level semantic labels. For example, by setting different materials and object categories in a virtual scene, each pixel can be automatically assigned a corresponding semantic label without human intervention. This approach not only greatly reduces the labeling cost, but also flexibly generates diverse data, covering different environments, lighting conditions, weather changes, etc., thereby enhancing the robustness and generalization ability of the model.

[0004] However, there are obvious differences between virtual data and real-world scenes, mainly reflected in the fact that virtual engine simulated data is not realistic enough, and has different lighting environments and shooting conditions from the real world; therefore, in order to train the model only using virtual data and achieve excellent semantic segmentation performance in real-world scenes, it is necessary to obtain image features that are robust and domain-independent.

[0005] With the development of large model technology, many visual foundation large models can obtain more robust image representations. Explicit visual pre-training and fine-tuning strategies in various downstream tasks have shown strong image feature extraction capabilities. At the same time, efficient fine-tuning methods based on visual foundation large models have shown obvious superior performance in out-of-distribution (OOD) generalization tasks. At present, most solutions focus on general domain generalization, rather than specifically targeting out-of-distribution semantic segmentation tasks. Therefore, how to utilize visual foundation large models in out-of-distribution semantic segmentation tasks and combine efficient fine-tuning strategies to ensure fewer parameters and better generalization performance is the key to the problem. SUMMARY

[0006] The main purpose of the embodiment of the present application is to provide an efficient and high-precision domain generalization semantic segmentation method, system, electronic device and storage medium, which can reduce the data labeling cost and achieve the generalization segmentation performance of the model to the real world scene.

[0007] To achieve the above-mentioned purpose, one aspect of an embodiment of the present application provides a domain generalization semantic segmentation method, comprising the following steps:

[0008] Load the pre-training parameters of the visual foundation large model, and the virtual engine generates a virtual simulation dataset;

[0009] Freeze the parameters of the visual foundation large model, input the simulation dataset into the visual foundation large model, and extract the intermediate features of the input image through the encoder of the visual foundation large model;

[0010] Cross-attention fusion is performed on the extracted intermediate features and the word vector;

[0011] The fused features are input into the segmentation head for semantic segmentation prediction to obtain the class prediction result corresponding to each pixel;

[0012] In each small-batch iterative training, randomly assign the class label not to participate in loss calculation; according to the specified class label, calculate the loss value between the predicted class of each pixel and the real class;

[0013] When gradient backpropagation, obtain the class label that does not participate in loss calculation, and set the corresponding word vector gradient to zero in the index mode;

[0014] Iteratively train the visual foundation large model until the loss converges, save the corresponding word vector and segmentation head, and obtain the trained target model;

[0015] According to the target model, the target data in a real-world target scene to be segmented is subjected to domain generalization semantic segmentation processing, and segmentation results of different semantics are obtained.

[0016] In some embodiments, the step of cross-attention fusion of the extracted intermediate features and the word vector comprises the following steps:

[0017] The cross-attention fusion mode is adopted to complementarily fuse the deep semantic information of the visual features and the class expression information guided by the word vector; wherein a query matrix Q is obtained by linear transformation of the word vector; a key matrix K is obtained by linear transformation of the feature; and a value matrix V is obtained by linear transformation of the feature;

[0018] The expression of the cross-attention fusion is: wherein d k is a learnable temperature coefficient; and A is the obtained output result.

[0019] In some embodiments, in the step of not participating in loss calculation, when randomly selecting the label, the label set of the small batch is set as {label i | i = 1, …, N}, wherein N≤19 is the total number of small batch label categories, and each time n class labels are randomly specified in the set, and the specified label is set as not participating in loss calculation.

[0020] In some embodiments, the number of the specified n class labels is limited by the condition that:

[0021]

[0022] wherein n max represents the maximum number of class labels, and 1≤n≤n max .

[0023] In some embodiments, in the step of inputting the fused features into a segmentation head for semantic segmentation prediction to obtain a class prediction result corresponding to each pixel, mask2forme is used as the segmentation head.

[0024] The calculation expression of the total loss of the visual basic large model is: loss = h_loss_cls + h_loss_dice + h_loss_focal, wherein h_loss_cls is the classification loss of the prediction result, h_loss_dice is the overlap loss between the predicted mask and the real mask, and h_loss_focal is the loss of sample proportion imbalance.

[0025] In some embodiments, the method further comprises: evaluating the processing result of the domain generalization semantic segmentation by pixel average intersection over union as an evaluation index of semantic segmentation.

[0026] wherein the calculation formula of the pixel average intersection over union is:

[0027]

[0028] wherein mIoU represents the pixel average intersection over union; N represents the total number of classes; TP i is the number of pixels correctly predicted as class i; FP i is the number of pixels incorrectly predicted as class i; FN i is the number of missed pixels of class i.

[0029] Another aspect of the embodiment of the application also provides a domain generalization semantic segmentation system, comprising:

[0030] A first module is configured to load pre-training parameters of a visual base large model, and a virtual engine generates a virtual simulation dataset;

[0031] A second module is configured to freeze parameters of the visual base large model, and extract intermediate features of an input image through an encoder of the visual base large model after inputting the simulation dataset into the visual base large model;

[0032] A third module is configured to cross attention fusion of the extracted intermediate features and word vectors;

[0033] A fourth module is configured to input the fused features into a segmentation head for semantic segmentation prediction to obtain a class prediction result corresponding to each pixel;

[0034] A fifth module is configured to randomly specify a class label not participating in loss calculation at each small batch iteration training, and calculate a loss value between a predicted class of each pixel and a real class according to the specified class label;

[0035] A sixth module is configured to obtain a class label not participating in loss calculation when gradient is back propagated, and set a corresponding word vector gradient to zero in an index manner;

[0036] A seventh module is configured to cyclically and iteratively train the visual base large model until the loss converges, save the corresponding word vector and segmentation head, and obtain a trained target model;

[0037] An eighth module is configured to perform domain generalization semantic segmentation processing on target data in a target scene of a real world to be segmented according to the target model, and obtain a segmentation result of different semantics.

[0038] To achieve the above object, another aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method described above when executing the computer program.

[0039] To achieve the above object, another aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method described above.

[0040] The embodiment of the present application also discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the method described above.

[0041] The embodiment of the present application at least has the following beneficial effects: the present application provides a domain generalization semantic segmentation method and system, an electronic device and a storage medium, the scheme loads the pre-training parameters of a visual basic large model, a virtual engine generates a virtual simulation dataset, the parameters of the visual basic large model are frozen, the intermediate features of an input image are extracted by an encoder of the visual basic large model after the simulation dataset is input into the visual basic large model, the extracted intermediate features are cross-attention fused with word vectors, the fused features are input into a segmentation head for semantic segmentation prediction to obtain a class prediction result corresponding to each pixel, a class label is randomly specified not to participate in loss calculation at each small batch iteration training, the loss value between the predicted class of each pixel and the real class is calculated according to the specified class label, the class label not participating in loss calculation is obtained when gradient is back propagated, and the corresponding word vector gradient is set to zero in an index manner, the visual basic large model is iteratively trained in a loop until the loss converges, the corresponding word vector and segmentation head are saved to obtain a trained target model, and the target data in a target scene in a real world to be segmented is subjected to domain generalization semantic segmentation processing according to the target model to obtain segmentation results of different semantics. The embodiment of the present application is efficient and high in precision, can reduce data labeling cost, and achieves generalization segmentation performance of a model to a real world scene. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is an implementation environment schematic diagram provided by the embodiment of the present application;

[0043] Figure 2 is a flowchart of the overall steps provided by the embodiment of the present application;

[0044] Figure 3is a step flow chart of a specific implementation process provided by an embodiment of the present application;

[0045] Figure 4 is an example diagram of an end-to-end domain generalization semantic segmentation model provided by an embodiment of the present application;

[0046] Figure 5 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the objects, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. When the following description relates to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the embodiments of the present application, but is only an example of devices and methods consistent with some aspects of the embodiments of the present application as described in the appended claims.

[0048] It can be understood that the terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above drawings are used to distinguish similar objects and do not necessarily have to describe a particular order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0049] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or", used to describe the association between the associated objects, means that there can be three relationships, for example, "A and / or B" can represent three cases: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0051] The domain generalization semantic segmentation method, system, electronic device and storage medium provided by the embodiments of the present application relate to the technical field of computer vision. The domain generalization semantic segmentation method provided by the embodiments of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto; the server end can be configured as a stand-alone physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and the server can also be a node server in a blockchain network; the software can be an application that implements the domain generalization semantic segmentation method, and the like, but is not limited to the above forms.

[0052] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as a program module. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0053] As shown in Figure 1 , it is an implementation environment schematic diagram provided by the embodiments of the present application. Referring to Figure 1 , the implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected by wireless or wired means for network connection to complete data transmission and exchange.

[0054] The server 101 can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0055] In addition, the server 101 can also be a node server in a blockchain network. The blockchain is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0056] The terminal 102 can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like. The terminal 102 can also be a vehicle-mounted terminal of various device types as exemplified above, but is not limited thereto. The terminal 102 and the server 101 can be directly or indirectly connected through wired or wireless communication, and the present embodiment is not limited thereto.

[0057] Exemplarily based on the implementation environment shown, Figure 1 The present embodiment provides a domain generalization semantic segmentation method. The following is an example of the domain generalization semantic segmentation method applied to the server 101. It should be understood that the method can also be applied to the terminal 102.

[0058] Referring to Figure 2 , Figure 2 The flowchart of the domain generalization semantic segmentation method applied to the server provided by the present embodiment can be executed by any one of the computer devices (including servers or terminals) described above. Referring to Figure 2 , the method can include the following steps:

[0059] Load the pre-training parameters of the visual base large model, and the virtual engine generates a virtual simulation dataset;

[0060] Freeze the parameters of the visual base large model, and input the simulation dataset into the visual base large model, and then extract the intermediate features of the input image through the encoder of the visual base large model;

[0061] Cross-attention fusion is performed between the extracted intermediate features and the word vectors;

[0062] The fused features are input into the segmentation head for semantic segmentation prediction to obtain the class prediction result corresponding to each pixel;

[0063] In each iteration of the mini-batch training, randomly assign the class label not to participate in the loss calculation; according to the specified class label, calculate the loss value between the predicted class of each pixel and the real class;

[0064] When the gradient is back-propagated, the class label not participating in the loss calculation is obtained, and the corresponding word vector gradient is set to zero in the index mode;

[0065] Iteratively train the visual base model until the loss converges, save the corresponding word vector and segmentation head, and obtain the trained target model;

[0066] According to the target model, the target data in the real world target scene to be segmented is processed for domain generalization semantic segmentation, and the segmentation results of different semantics are obtained.

[0067] In some embodiments, the step of cross-attention fusion of the extracted intermediate feature and the word vector comprises the following steps:

[0068] The cross-attention fusion mode is adopted to fuse the complementary information of the deep semantic information of the visual feature and the class expression information guided by the word vector; wherein, the query matrix Q is obtained by linear transformation of the word vector; the key matrix K is obtained by linear transformation of the feature; and the value matrix V is obtained by linear transformation of the feature;

[0069] The expression of cross-attention fusion is: Wherein, d k is a learnable temperature coefficient; A is the output result obtained.

[0070] In some embodiments, in the step of randomly assigning the class label not to participate in the loss calculation, when randomly selecting the label, let the small batch label set be {label i |i=1,...,N}, wherein N≤19 is the total number of small batch label categories, and n class labels are randomly assigned in the set each time, and the assigned label is set not to participate in the loss calculation.

[0071] In some embodiments, the number of n class labels specified is limited by:

[0072]

[0073] Wherein, n max represents the maximum number of class labels, 1≤n≤n max .

[0074] In some embodiments, in the step of inputting the fused feature into the segmentation head for semantic segmentation prediction to obtain the class prediction result corresponding to each pixel, mask2forme is used as the segmentation head;

[0075] The calculation expression of the model total loss of the visual base large model is: loss = h_loss_cls + h_loss_dice + h_loss_focal, wherein h_loss_cls is the classification loss of the prediction result, h_loss_dice is the overlap loss between the predicted mask and the real mask, and h_loss_focal is the loss of sample proportion imbalance.

[0076] In some embodiments, the method further comprises: evaluating the processing result of the domain generalization semantic segmentation by taking the pixel average intersection over union as an evaluation index of semantic segmentation;

[0077] The calculation formula of the pixel average intersection over union is:

[0078]

[0079] Wherein, mIoU represents the pixel average intersection over union; N represents the total number of classes; TP i is the number of pixels correctly predicted as class i; FP i is the number of pixels incorrectly predicted as class i; FN i is the number of missed pixels of class i.

[0080] The specific implementation process of the method of the application will be described in detail below with a specific application scenario as an example:

[0081] In view of the problems existing in the prior art, the application provides a domain generalization semantic segmentation method for efficiently fine-tuning a visual base large model, which is suitable for end-to-end training in actual driving scenarios. In order to reduce the data labeling cost, the application proposes to train completely relying on the data generated by the virtual engine simulation scene, extract discriminative features through the visual base large model, and combine an efficient fine-tuning strategy to enable the model to quickly adapt to the real driving environment. Specifically, the application fuses the features of each layer of the visual large model with randomly initialized word vectors to guide feature expression, thereby promoting the differentiation of class information. Since the class information is domain-independent, it can improve the generalization ability of the model and enable it to maintain good segmentation performance in different driving scenarios. To ensure the effective binding of the word vector and the class information, the application proposes to randomly assign the class label not to participate in the loss calculation at each training time, and set the gradient of the corresponding word vector to zero according to the index, to ensure the relevance of the word vector and the specific class. Through the virtual simulation data, the robust features are extracted by combining the visual base large model, and the class information expression is further guided by the word vector, so that the application can significantly improve the precision of domain generalization semantic segmentation and realize excellent generalization performance of the model in the real driving scenario.

[0082] As Figure 3As shown, the specific implementation process of the method of the present application includes the following steps:

[0083] Step 1: Prepare a virtual simulation dataset d; load the pre-training parameters of the visual base large model m. The simulation dataset is generated by a virtual engine, simulating different driving scenarios to reduce data labeling costs.

[0084] Step 2: Freeze the parameters of the base visual large model m, and use the encoder of m to extract the intermediate features f of the input image;

[0085] 2、Step 3: Cross-attention fusion of the extracted intermediate features f and the word vector t; In this embodiment, the intermediate features f and the word vector t are fused by using cross-attention (Cross-attention) to fuse and complement the deep semantic information of the visual features and the class expression information guided by the word vector. Wherein Q is the query matrix, which is obtained by linear transformation from the word vector t; K is the key matrix, which is obtained by linear transformation from the feature f; V is the value matrix, which is obtained by linear transformation from the feature f; d k is a learnable temperature coefficient; A is the obtained output result; Specifically as shown in formula (1).

[0086]

[0087] Step 4: Input the fused features F into the segmentation head h to perform semantic segmentation prediction, and obtain the corresponding class prediction result for each pixel;

[0088] Step 5: Randomly specify the class labels not to participate in loss calculation during each mini-batch iteration training; Wherein, when randomly selecting labels in step 5, the label set of the mini-batch is {label i | i = 1,...,N}, where N≤19 is the total number of mini-batch label categories, and n class labels are randomly specified in the set each time, and the specified labels are set not to participate in loss calculation. Each time n class labels are randomly specified not to participate in loss calculation, n is an integer and 1≤n≤n max , wherein n max is defined as formula (2):

[0089]

[0090] Step 6: According to the selection in step 5, calculate the loss value l between the predicted class of each pixel and the real class;

[0091] Step 7: During gradient backpropagation, obtain the category labels that are not involved in the loss calculation, and set the gradient of the corresponding word vector to zero according to the index. In Step 7, labels that are not involved in the loss calculation are selected. Each label corresponds to a portion of the pixels of the sample, and the features corresponding to these pixels will not be used for object category prediction. Therefore, the gradient obtained by the network during backpropagation will not be applied to the corresponding word vector. Assuming that the category label is 1 and does not participate in the loss calculation, according to the index, the first word vector will not be updated by the gradient. This invention utilizes virtual simulation data to efficiently fine-tune the basic visual model. Combining robust image features and using word vectors to guide the expression of category information, the model can maintain high generalization ability in real driving scenarios, effectively reduce the data annotation cost of semantic segmentation, and significantly improve the accuracy of domain generalization semantic segmentation tasks.

[0092] Step 8: Repeat steps 2-7 until the loss l converges, and save the word vector t and the segmentation head h. The driving scene domain generalization model trained in this invention is an end-to-end semantic segmentation model, which takes objects in real-world driving scenes (roads, pedestrians, cars, plants, sky, etc.) as categories to be segmented, and identifies different semantically meaningful objects in the content of images taken in driving scenes through semantic segmentation methods.

[0093] The semantic segmentation framework, combined with a basic large-scale visual model, allows for the selection of any suitable end-to-end basic large-scale visual model based on actual project requirements (inference speed, segmentation accuracy, stability, etc.). For example... Figure 4 The DINOV2 basic visual model shown can be used as a feature encoding model for semantic segmentation. It can extract image features with strong discriminative and generalizable properties. By combining word vectors, it achieves the fusion and complementarity of high-level visual features and category boundary information, enabling efficient fine-tuning in semantic segmentation tasks. End-to-end refers to the fact that a semantic segmentation model is generally composed of multiple parts, each performing different functions. This embodiment involves three parts: the basic visual model as the image feature extraction part, whose network parameters are fixed and do not participate in gradient updates; a series of learnable word vectors fused with intermediate layer image features, whose parameters need to be updated for efficient fine-tuning of the large model to adapt to the semantic segmentation task; and the segmentation head decoding the fused features and implementing end-to-end loss training, whose parameters need to be updated to ensure the accuracy of the segmentation function. These parts are sequentially processed to achieve the overall predictive function of the network. During training, this training method, which updates the parameters that the model needs to update simultaneously with the total loss using gradient descent, is called end-to-end training. Figure 4The mask2forme is used as the segmentation head in the embodiment, and the module loss thereof includes the classification loss h_loss_cls of the prediction result, the overlap loss h_loss_dice between the predicted mask and the real mask, and the loss h_loss_focal for solving the uneven sample ratio. The total loss is shown in formula (1):

[0094] loss = h_loss_cls + h_loss_dice + h_loss_focal (1)

[0095] The loss obtained in step 6 is the total loss of the model.

[0096] The intermediate feature f in step 3 is fused with the word vector t by using the cross attention method to fuse and complement the deep semantic information of the visual feature and the category expression information guided by the word vector. Q is the query matrix obtained by linear transformation of the word vector t; K is the key matrix obtained by linear transformation of the feature f; V is the value matrix obtained by linear transformation of the feature f; d k is the temperature coefficient obtained by learning; A is the output result obtained; and the specific formula is shown in formula (2).

[0097]

[0098] In step 5, when randomly selecting labels, the label set of a mini-batch is {label i ,..., N}, wherein N≤19 is the total number of label categories of the mini-batch, n classes of labels are randomly specified in the set each time, and the specified class labels are set not to participate in loss calculation, n is an integer and 1≤n≤N max , wherein n max is defined in formula (3):

[0099]

[0100] In addition, the labels not participating in loss calculation in step 7 each correspond to part of the pixel points of the sample, and the features corresponding to these pixel points will not be used for object category prediction. Therefore, the gradient obtained by the network during back propagation will not act on the corresponding word vector. Assuming that the class label is 1 and does not participate in loss calculation, according to the index mode, the first word vector will not be updated by the gradient.

[0101] By using virtual simulation data to efficiently fine-tune the visual basic model, combining robust image features and using word vectors to guide the expression of category information, the model can maintain high generalization ability in real driving scenarios, effectively reduce the data labeling cost of the semantic segmentation task, and significantly improve the accuracy of the domain generalization semantic segmentation task.

[0102] The commonly used evaluation index of semantic segmentation is the pixel average intersection over union (Mean Intersection over Union, mIoU). The mIoU is an index for measuring the overlap between the predicted result and the true label, and the specific definition is given in formula (4):

[0103]

[0104] N represents the total number of classes in the whole, wherein TP i is the number of pixels correctly predicted as class i; FP i is the number of pixels incorrectly predicted as class i; FN i is the number of missed pixels of class i.

[0105] In summary, the present application can efficiently fine-tune the visual basic large model, and can perform good semantic segmentation on real-world driving scenarios using only virtual data training; the visual basic large model extracts robust image features, and a series of learnable word vectors are combined with the image features through cross-attention, and the word vectors are further bound to the category information by erasing the specified category label and word vector gradient zero method. Since the category information has the characteristics of domain independence, the image features guided by the word vectors also have the characteristics of domain independence, which significantly reduces the data labeling cost while achieving the generalization segmentation performance of the model on real-world scenarios, and can be widely applied to the fields of image processing and computer vision technology.

[0106] Another aspect of the embodiment of the present application also provides a domain generalization semantic segmentation system, comprising:

[0107] The first module is used for loading the pre-training parameters of the visual basic large model, and a virtual engine generates a virtual simulation data set;

[0108] The second module is used for freezing the parameters of the visual basic large model, and after inputting the simulation data set into the visual basic large model, the intermediate features of the input image are extracted through the encoder of the visual basic large model;

[0109] The third module is used for cross-attention fusion of the extracted intermediate features and word vectors;

[0110] The fourth module is used for inputting the fused features into a segmentation head for semantic segmentation prediction to obtain the category prediction result corresponding to each pixel.

[0111] The fifth module is configured to randomly assign the class label not to participate in loss calculation at each iteration of small batch training, and calculate the loss value between the predicted class of each pixel and the real class according to the assigned class label;

[0112] The sixth module is configured to obtain the class label not participating in loss calculation when gradient is back propagated, and set the corresponding word vector gradient to zero in an index manner;

[0113] The seventh module is configured to iteratively train the visual base model until the loss converges, save the corresponding word vector and segmentation head, and obtain the trained target model;

[0114] The eighth module is configured to perform domain generalization semantic segmentation processing on target data in a target scene in a real world to be segmented according to the target model, and obtain a segmentation result of different semantics.

[0115] It can be understood that the content in the above method embodiments is applicable to the present system embodiment, the present system embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0116] The embodiment of the present application also provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above domain generalization semantic segmentation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0117] It can be understood that the content in the above method embodiments is applicable to the present device embodiment, the present device embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0118] Please refer to Figure 5 , Figure 5 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device comprises:

[0119] The processor 501 can be implemented in the form of a general CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0120] The memory 502 can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM), etc. The memory 502 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 502 and are called and executed by the processor 501 to implement the domain generalization semantic segmentation method of the embodiments of the present application;

[0121] The input / output interface 503 is configured to realize information input and output.

[0122] The communication interface 504 is configured to realize the communication interaction between the device and other devices, and the communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0123] The bus 505 is configured to transmit information between various components (for example, the processor 501, the memory 502, the input / output interface 503 and the communication interface 504) of the device.

[0124] The processor 501, the memory 502, the input / output interface 503 and the communication interface 504 are connected to each other through the bus 505 to realize the communication connection between the device.

[0125] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the above-mentioned domain generalization semantic segmentation method.

[0126] It can be understood that the contents in the above method embodiments are applicable to the present storage medium embodiments, the functions realized by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved by the present storage medium embodiments are also the same as those of the above method embodiments.

[0127] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0128] It should be noted that in various specific embodiments of the present application, when relevant processing needs to be performed on data related to the identity or characteristics of the user, such as user information, user behavior data, user history data, and user location information, the user's permission or consent is obtained first, and the collection, use, and processing of such data comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the user's separate permission or separate consent is obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the user's separate permission or separate consent, the necessary user-related data for enabling the embodiments of the present application to normally operate is obtained.

[0129] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0130] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0131] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0132] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and device can be implemented as software, firmware, hardware and their appropriate combinations.

[0133] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, and the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0134] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0135] In addition, each functional unit in each embodiment of the application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0136] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the application, the essential part or the whole or part of the contribution to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0137] The preferred embodiments of the embodiments of the application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the application shall be within the scope of the embodiments of the application.

Claims

1. A method of domain generalization semantic segmentation, the method comprising: The method comprises the following steps: loading pre-training parameters of a visual base large model, and generating a virtual simulation dataset by a virtual engine; freezing parameters of the visual base large model, and extracting intermediate features of an input image by an encoder of the visual base large model after inputting the simulation dataset into the visual base large model; cross-attention fusion of the extracted intermediate features and word vectors; inputting the fused features into a segmentation head for semantic segmentation prediction to obtain a class prediction result corresponding to each pixel; in each small-batch iterative training, randomly assigning a class label not to participate in loss calculation; calculating a loss value between a predicted class of each pixel and a real class according to the assigned class label; in gradient backpropagation, obtaining a class label not participating in loss calculation, and setting corresponding word vector gradients to zero in an index manner; iterative training of the visual base large model until loss convergence, saving corresponding word vectors and the segmentation head to obtain a trained target model; performing domain generalization semantic segmentation processing on target data in a target scene of a real world to be segmented according to the target model to obtain segmentation results of different semantics; the cross-attention fusion of the extracted intermediate features and word vectors comprises the following steps: The cross attention is used to fuse the deep semantic information of the visual features and the category expression information guided by the word vector complementarily The key matrix is obtained by linear transformation of the features The value matrix is obtained by linear transformation of the features ​ The expression of cross-attention fusion is: wherein, is a learnable dimension coefficient; is the output result obtained.

2. The domain generalization semantic segmentation method of claim 1, wherein, In the step of not participating in the loss calculation, the randomly selected label is set as the small batch label set wherein, is the total number of small batch label categories, and category labels are randomly selected in the set each time, and the selected label is set as not participating in the loss calculation.

3. The domain generalization semantic segmentation method of claim 2, wherein, designated The number of the category tags is limited by the following condition: wherein a maximum number of representative category labels, .

4. The domain generalization semantic segmentation method of claim 1, wherein, in the step of inputting the fused features into a segmentation head for semantic segmentation prediction to obtain a class prediction result corresponding to each pixel, mask2forme is used as the segmentation head; A calculation expression of a model total loss of the visual base model is as follows: wherein, is a classification loss of a prediction result, is an overlap loss between a predicted mask and a real mask, the method further comprises: evaluating the processing result of the domain generalization semantic segmentation by taking pixel mean intersection over union as an evaluation index of semantic segmentation; a loss of sample proportion imbalance.

5. The domain generalization semantic segmentation method of claim 1, wherein, wherein the calculation formula of the pixel mean intersection over union is: comprising: wherein, represents the average intersection over union of the pixels; represents the total number of classes overall; is the number of pixels that were correctly predicted as class ; is the number of pixels that were incorrectly predicted as class ; is the number of pixels that were missed for class .

6. A system for implementing the domain generalization semantic segmentation method according to any one of claims 1-5, characterized in that, a first module for loading pre-training parameters of a visual base large model, and generating a virtual simulation dataset by a virtual engine; a second module for freezing parameters of the visual base large model, and extracting intermediate features of an input image by an encoder of the visual base large model after inputting the simulation dataset into the visual base large model; a third module for cross-attention fusion of the extracted intermediate features and word vectors; a fourth module for inputting the fused features into a segmentation head for semantic segmentation prediction to obtain a class prediction result corresponding to each pixel; a fifth module for randomly assigning a class label not to participate in loss calculation in each small-batch iterative training; calculating a loss value between a predicted class of each pixel and a real class according to the assigned class label; a sixth module for obtaining a class label not participating in loss calculation in gradient backpropagation, and setting corresponding word vector gradients to zero in an index manner; a seventh module for iterative training of the visual base large model until loss convergence, saving corresponding word vectors and the segmentation head to obtain a trained target model; an eighth module for performing domain generalization semantic segmentation processing on target data in a target scene of a real world to be segmented according to the target model to obtain segmentation results of different semantics. comprising a processor and a memory; 7. An electronic device, comprising: the memory is used to store a program; ​ The processor executes the program to implement the method in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program, and the program is executed by a processor to implement the method in any one of claims 1-5.

9. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the method in any one of claims 1-5.

Images