Alignment module, training of decoder, image segmentation method, device and medium

Abstract

The application discloses an alignment module, training of a decoder, an image segmentation method, equipment and a medium. The method comprises the following steps: obtaining a first training sample; performing feature extraction on a source domain image to obtain a source domain image feature; performing feature extraction on prompt text of each target domain to obtain text features of each target domain; inputting the source domain image feature and the text features of each target domain into an alignment module for processing, and outputting image features of each target domain; fusing the image features of each target domain to obtain visual fusion features; fusing the text features of each target domain to obtain text fusion features; aligning the visual fusion features and the text fusion features according to an object type, and calculating a loss value of the alignment module; and adjusting parameters of the alignment module according to the loss value. By adopting the embodiment of the application, the semantic segmentation implementation efficiency can be improved, and the semantic segmentation accuracy in a general field can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an alignment module, decoder training, image segmentation method, device and medium. Background Technology

[0002] In image segmentation tasks, image segmentation models often experience performance degradation when trained on untrained domains. Domain generalization can be used to generalize models trained in the source domain to unknown target domains.

[0003] However, for image segmentation tasks with different target domains, it is necessary to train specific segmentation models separately for each target domain, resulting in high training costs and low training efficiency. Summary of the Invention

[0004] This application provides an alignment module, a decoder training method, an image segmentation method, an apparatus, and a medium, which can improve the efficiency of semantic segmentation and the accuracy of semantic segmentation in general domains.

[0005] In a first aspect, embodiments of this application provide a method for training an alignment module, comprising:

[0006] Obtain the first training sample, which includes the source domain image, the prompt text of the target domain, and the ground truth segmentation value of the source domain image. There are multiple target domains.

[0007] Feature extraction is performed on the source domain image to obtain source domain image features; feature extraction is performed on the prompt text of each target domain to obtain text features of each target domain;

[0008] The source domain image features and the text features of each target domain are input into the alignment module for processing, and the image features of each target domain are output.

[0009] Based on the source domain image segmentation ground truth, the image features of each target domain are fused according to the type of the object to be identified, resulting in visual fusion features; the text features of each target domain are fused according to the type of the object to be identified, resulting in text fusion features.

[0010] Based on the type of the object being identified, the visual fusion features and text fusion features are aligned, and the loss value of the alignment module is calculated; based on the loss value, the parameters of the alignment module are adjusted.

[0011] The alignment module is used to train the target domain image decoder in the target domain image segmentation model; the target domain image segmentation model is used to perform semantic segmentation on images in the target domain.

[0012] Secondly, embodiments of this application provide a training method for a target domain image decoder, comprising:

[0013] Obtain a second training sample, which includes the source domain image, the prompt text of the first domain, and the ground truth segmentation value of the source domain image; the first domain is the specified requirement domain.

[0014] Feature extraction is performed on the source domain image to obtain the source domain image features; feature extraction is performed on the prompt text of the first domain to obtain the text features of the first domain;

[0015] The source domain image features and the text features of the first domain are input into the alignment module for processing, and the target domain image features are output; wherein, the alignment module is trained by the alignment module training method as described in any embodiment of this application;

[0016] The target domain image features and the first domain text features are input into the corrector for processing, and the corrected target domain image features are output.

[0017] The corrected target domain image features are input into the target domain image decoder for processing, and the target domain image segmentation result is output.

[0018] The parameters of the corrector and the target domain image decoder are adjusted based on the difference between the target domain image segmentation result and the source domain image segmentation ground truth.

[0019] Thirdly, embodiments of this application provide an image segmentation method, including:

[0020] The acquired target domain image is input into a trained target domain image segmentation model; the target domain image segmentation model includes an image encoder and a target domain image decoder, and the target domain image decoder is trained by the training method of the target domain image decoder as described in any embodiment of this application;

[0021] The target domain image features are obtained through an image encoder;

[0022] The target domain image features are input into the target domain image decoder for processing, and the target domain image segmentation result is output.

[0023] The target object is determined based on the segmentation results of the target domain image.

[0024] Fourthly, this application provides a computer device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps in the above-described alignment module training method, target domain image decoder training method, or image segmentation method.

[0025] Fifthly, this application provides a computer program product comprising a computer program that, when executed by a processor, implements the steps in the above-described alignment module training method, target domain image decoder training method, or image segmentation method.

[0026] The aforementioned training methods for the alignment module, the target domain image decoder, the image segmentation method, the apparatus, the computer device, the computer-readable storage medium, and the computer program product acquire image features and text features from multiple target domains, fuse these features across multiple target domains to obtain visual fusion features and text fusion features, and align these visual fusion features and text fusion features to train the alignment module. This enables the alignment module to perform image-text alignment across multiple target domains, and the target domain image decoder in the target domain image segmentation model is trained based on this alignment module. This solves the problem in existing technologies that require targeted training of the image segmentation model for the target domain, improves the generalization ability of the image segmentation model, increases the efficiency of image segmentation, and enhances the accuracy of image segmentation for general target domains. Attached Figure Description

[0027] Figure 1 An application environment diagram for a training method for an alignment module, a training method for a target domain image decoder, or an image segmentation method provided in the embodiments of this application;

[0028] Figure 2 A flowchart illustrating a training method for an alignment module provided in an embodiment of this application;

[0029] Figure 3 A flowchart illustrating a training method for a target domain image decoder provided in an embodiment of this application;

[0030] Figure 4 A flowchart illustrating an image segmentation method provided in an embodiment of this application;

[0031] Figure 5 A structural block diagram of a hierarchical, multimodal graph-text semantic information alignment system provided in this application embodiment;

[0032] Figure 6 This application provides an illustration of an target domain image segmentation model.

[0033] Figure 7 This application provides an example of a domain consistency alignment application scenario diagram.

[0034] Figure 8 An application scenario diagram for training an alignment module provided in an embodiment of this application;

[0035] Figure 9 An application scenario diagram of a domain gap provided in an embodiment of this application;

[0036] Figure 10 An application scenario diagram for training a corrector provided in an embodiment of this application;

[0037] Figure 11 A structural block diagram of a training device for an alignment module provided in an embodiment of this application;

[0038] Figure 12 A structural block diagram of a training device for a target domain image decoder provided in an embodiment of this application;

[0039] Figure 13 A structural block diagram of an image segmentation apparatus provided in an embodiment of this application;

[0040] Figure 14 An internal structural diagram of a computer device provided in an embodiment of this application;

[0041] Figure 15 An internal structural diagram of another computer device provided in an embodiment of this application;

[0042] Figure 16 This is an internal structural diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] The alignment module training method, target domain image decoder training method, or image segmentation method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a communication network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0045] like Figure 2 As shown, this application embodiment provides a training method for an alignment module, which is applied to... Figure 1 The method will be illustrated using terminal 102 or server 104 as examples. It is understood that the computer device may include at least one of a terminal and a server. The method includes the following steps:

[0046] S201. Obtain the first training sample, which includes the source domain image, the prompt text of the target domain, and the ground truth of the source domain image segmentation. There are multiple target domains.

[0047] The first training sample is used to train the alignment module. The source domain image can refer to an image within an application scenario in the source domain. The source domain is relative to the target domain; it can be understood as an application scenario where data can be collected, while the target domain is an application scenario where data is difficult to collect. For example, the source domain could be a sunny day, and the target domain a sandstorm. Or, the source domain could be outdoors, and the target domain indoors. In fact, the image segmentation in this application performs semantic segmentation on images of the application scenario in the target domain. Since images in the source domain are relatively easy to collect, while images in the target domain are difficult to collect, the image segmentation model trained based on source domain images has low segmentation accuracy in the target domain. Therefore, the final training objective in this embodiment is to improve the image segmentation accuracy of the image segmentation model in the target domain.

[0048] The prompt text for the target domain refers to the text that describes the target domain. The prompt text can be a prompt word, specifically referring to the input of a model or algorithm, usually a piece of text used to guide the model to generate the corresponding output. Different target domains describe different scenarios. For example, the prompt text for target domain A is: "Rainy." The prompt text for target domain B is: "Sandstorm." The prompt text for target domain C is: "Typhoon." The prompt text for target domain D is: "Snowing."

[0049] The ground truth for source domain image segmentation refers to the correct result of image segmentation in the source domain image. This ground truth can be obtained through pre-annotation or by processing with a trained source domain image segmentation model. In reality, the ground truth for source domain image segmentation and the ground truth for target domain image segmentation are the same. The difference between the source and target domains lies in the scene; the scene represents the background in image segmentation, while the object being identified is the foreground. Therefore, the same image content will result in the same segmentation in both the source and any target domain.

[0050] S202. Extract features from the source domain image to obtain source domain image features; extract features from the prompt text of each target domain to obtain text features of each target domain.

[0051] A trained image encoder can be used to extract features from the source domain image, thus obtaining the source domain image features. The image encoder may include at least one convolutional layer. Furthermore, the image encoder can also have other structures; there are no specific limitations on this.

[0052] A trained text feature extraction model can be used to extract features from the prompt text in the target domain. The resulting text features are the text features of that target domain. Each prompt text in the target domain corresponds one-to-one with a text feature of that target domain.

[0053] S203. Input the source domain image features and the text features of each target domain into the alignment module for processing, and output the image features of each target domain.

[0054] There is a one-to-one correspondence between the target domain and image features. Target domain image features can refer to the characteristics of the image within the target domain. Specifically, the cue text for the target domain includes the overall cue text for that target domain and at least one unit cue text identifying the object type. The overall cue text for the target domain and the source domain image features are input into the alignment module for processing, and the output is the image features of the target domain.

[0055] S204. Based on the source domain image segmentation ground value, fuse the image features of each target domain according to the type of the object to be identified to obtain visual fusion features; fuse the text features of each target domain according to the type of the object to be identified to obtain text fusion features.

[0056] The object type to be identified can refer to the type of object segmented in the image. Visual fusion features include image features from multiple target domains, and can further distinguish the target domain to which each image feature belongs. Additionally, visual fusion features include at least one image feature that identifies an object type, and can further distinguish pixels of each object type within the image features.

[0057] Specifically, based on the ground truth of the source domain image segmentation, the image regions corresponding to each object type can be determined and mapped to the image features of each target domain to determine the image features corresponding to each object type. The image features of each target domain are then fused according to the image features corresponding to each object type, forming multi-dimensional image features, i.e., visual fusion features. One dimension distinguishes different target domains, another distinguishes different object types, and other dimensions can also exist; for example, other dimensions can also be used to distinguish features corresponding to different pixels of the same object type within the same target domain.

[0058] Text fusion features include text features from multiple target domains, and can further distinguish the target domain to which each text feature belongs. Simultaneously, text fusion features include at least one text feature representing an object type. Based on the text features corresponding to each target domain and each object type, the text features of each target domain are fused to form multi-dimensional text features, i.e., text fusion features. One dimension distinguishes different target domains, another distinguishes different object types, and other dimensions may also exist, such as those used to distinguish text content.

[0059] S205. Align the visual fusion features and text fusion features according to the type of the object being identified, and calculate the loss value of the alignment module; adjust the parameters of the alignment module according to the loss value.

[0060] The alignment module is used to train the target domain image decoder in the target domain image segmentation model; the target domain image segmentation model is used to perform semantic segmentation on images in the target domain.

[0061] Aligning visual and textual fusion features based on the object type can refer to aligning image features and textual features of the same object type. Furthermore, for each object type, image features and textual features of the same object type can be aligned. Alignment can be understood as image features and textual features expressing the same semantic information. For example, image feature A describes the image content of a cat, image feature B describes the image content of a dog, text feature a describes the text content of a cat, and text feature b describes the text content of a dog. Aligning image feature A with text feature a means aligning the image feature of the cat object type with the text feature; aligning image feature B with text feature b means aligning the image feature of the dog object type with the text feature. Specifically, the alignment module calculates its loss value with the goal of making image features and textual features of the same object type as similar as possible. The loss value of the alignment module describes the similarity between image features and textual features of the same object type.

[0062] Furthermore, alignment can also be represented by the similarity of image features and text features of different object types. For example, the loss value of the alignment module can be calculated with the goal of making the image features and text features of different object types as different as possible. The loss value of the alignment module describes the similarity between image features and text features of different object types.

[0063] Specifically, the alignment module's parameters can be adjusted to minimize its loss value. Training is considered complete when the alignment module meets training completion conditions. For example, these conditions might include a loss value less than a preset threshold, loss value convergence, or the number of iterations being greater than or equal to a preset threshold. The alignment module converts source domain image features into target domain image features. The target domain image decoder processes these features to obtain the semantic segmentation result of the target domain image. The target domain image segmentation model performs semantic segmentation on the target domain image.

[0064] This application's embodiments transform source domain image features to obtain target domain image features, and use these features to train a segmentation model. This eliminates the need to train the segmentation model with the target domain image itself, yet the trained model still possesses semantic segmentation capabilities for the target domain image. This represents zero-shot training for the target domain image segmentation model. Zero-shot training refers to a model's ability to effectively classify or process specific categories without having encountered training samples of those categories before. Zero-shot learning means the model performs well when faced with previously unseen categories or tasks. The resulting image segmentation model exhibits good segmentation performance across domains other than the source domain, demonstrating domain adaptability. Domain adaptation improves the model's performance and generalization ability in the target domain by adapting it to the data distribution and characteristics of different domains.

[0065] In some embodiments, image features of each target domain are fused according to the object type based on the source domain image segmentation ground truth to obtain visual fusion features, including:

[0066] Based on at least one object type, the source domain image segmentation ground value is transformed and max-pooled to obtain the category ground value features;

[0067] Based on the image features of each target domain, and according to the true value features of the categories, the features corresponding to each object type are extracted from the image features of the target domain to obtain the regional features of the target domain;

[0068] The regional features of each target domain are fused to obtain visual fusion features.

[0069] The source domain image segmentation ground truth includes at least one segmentation ground truth corresponding to a recognized object type. Typically, different pixel values ​​in the source domain image segmentation ground truth represent objects of different recognized object types. The source domain image segmentation ground truth can refer to a pixel matrix of the same size as the source domain image. For example, if there are two recognized object types, with pixel values ​​of 1 for object type 1 and 2 for object type 2, then pixels with a value of 1 represent the segmentation ground truth for object type 1, pixels with a value of 2 represent the segmentation ground truth for object type 2, and pixels with a value of 0 represent the background. The segmentation ground truth corresponding to each recognized object type is used to filter image features that need alignment. Specifically, image features of the same recognized object type are aligned with text features, while image features of different recognized object types are not aligned with text features.

[0070] Specifically, based on the type of each object to be identified, the ground truth of the source domain image segmentation is transformed to form the ground truth of the segmentation for each object type; the ground truth of the segmentation for each object type is then fused to obtain the category ground truth features.

[0071] Based on the pixel values ​​corresponding to each object type in the source domain image segmentation ground value, segmentation ground values ​​corresponding to each object type can be generated. The pixel values ​​of a pixel in the segmentation ground value corresponding to an object type include both the pixel value corresponding to the object type and the pixel value corresponding to the background. The segmentation ground values ​​corresponding to each object type are then fused to obtain category ground value features. The number of channels in the category ground value features is the same as the number of object types, and the image features of the same channel in the category ground value features include the segmentation ground value corresponding to the same object type for that channel.

[0072] In practice, the size of the category ground truth features needs to be consistent with the size of the target domain image features. In addition to splitting and fusing the source domain image segmentation ground truths, size adjustment is also required. For example, max pooling can be performed on the source domain image segmentation ground truths, and the pooled source domain image segmentation ground truths can be split to obtain segmentation ground truths corresponding to each object type. At this point, the size of the segmentation ground truths corresponding to each object type is the same as the size of the target domain image features. Fusing the segmentation ground truths of each object type yields category ground truth features with the same size as the target domain image features. Alternatively, segmentation ground truths corresponding to each object type can be generated based on the source domain image segmentation ground truths, and the segmentation ground truths of each object type can be fused. Max pooling of the fused results yields category ground truth features with the same size as the target domain image features.

[0073] The regional features of the target domain can be image features corresponding to the identified object type selected from the image features of the target domain. Based on the category ground truth features, image features corresponding to the identified object type are extracted from the image features of the target domain. These extracted image features are then fused to obtain the regional features of the target domain. In this embodiment, regional features are obtained according to the identified object type. The extracted regional features contain information about the existence of the same identified object type. Based on the consistency of the same identified object type, the image features of the same identified object type in the regional features and the text features of the same identified object type in the text features of the target domain are aligned. This is done with the constraint that features of the same identified object type remain unchanged when converting from source domain image features to target domain image features, thereby improving the accuracy of the target domain image features generated by the alignment module.

[0074] The true feature of the category is y s * ∈{0,1} n×h×w The target domain image features are f s→t Where n is the number of object types to be identified, the region features are calculated based on the following formula.

[0075]

[0076] The regional features of the target domain are obtained and the regional features of different target domains are fused. Specifically, a dimension is added, with the target domain as the variable of that dimension.

[0077] In a specific example, there are m target domains. The values ​​C of the m target domains are fused to obtain the visual fusion feature.

[0078] By extracting image features corresponding to the object type based on the segmentation ground value corresponding to the image features of each target domain, the region features corresponding to the target domain are formed. The region features of each target domain are then fused to obtain visual fusion features. This results in visual fusion features that include image features of each target domain and image features of each object type. This allows for alignment based on the object type and increases the content of the applicable target domains. It enables the image features of the target domains to be adapted to various application scenarios, improves the versatility of the image segmentation model across multiple domains, enhances the generalization ability of the image segmentation model, and ultimately improves the accuracy of image segmentation.

[0079] In some embodiments, the target domain text features include at least one category text feature that identifies the object type; the text features of each target domain are fused according to the identified object type to obtain text fusion features, including:

[0080] Text features of each object type in each target domain are fused to obtain text fusion features.

[0081] The prompt text includes at least one unit prompt text identifying an object type. The unit prompt text for the target domain describes the object type being identified within that target domain. There is a one-to-one correspondence between the unit prompt text for the target domain and the object type being identified. For each target domain, the number of object types being identified is the same as the number of unit prompt texts for that target domain. For example, object types include: tree, sign, vehicle, and pedestrian; correspondingly, the unit prompt texts for the target domain include: tree in the rain, sign in the rain, vehicle in the rain, and pedestrian in the rain, etc. The number and content of object types in the unit prompt texts existing in different target domains are the same. For example, the unit prompt text for target domain 1 includes prompt text for object type a and prompt text for object type b. The unit prompt text for target domain 2 includes prompt text for object type a and prompt text for object type b.

[0082] Text features are extracted from a single unit prompt text within a target domain to obtain a category text feature for that target domain. The unit prompt text and the category text feature correspond to the same object type. Category text features from different object types within the same target domain are fused to obtain an intermediate fusion result. These intermediate fusion results from different target domains are then fused to obtain a fused text feature. Fusion is used to add dimensions; the first fusion adds the object type, and the second fusion adds the target domain.

[0083] In a specific example, there are m target domains. Unit cue text for each target domain can be obtained by describing the objects within those m domains. For instance, for the "rainy day scene" domain, the number of object types to be identified is n. Specifically, objects can be described as "bus in the rain" and "rider in the rain," etc. Through text feature extraction, category text features are obtained. Using the n identified object types as dimensions, the category text features of each object type are fused to obtain the text features for the target domain. This refers to the aforementioned intermediate fusion result. For the text features of m target domains, with ym target domains as the dimension, the text features of the m target domains are fused to obtain the fused text features.

[0084] By fusing the category text features of each object type in each target domain, a text fusion feature is obtained. This text fusion feature includes text features from each target domain and text features from each object type, thereby enabling alignment for object types and increasing the content of applicable target domains. This allows the image features of the target domains to be adapted to various application scenarios, improves the versatility of the image segmentation model across multiple domains, enhances the generalization ability of the image segmentation model, and ultimately improves the accuracy of image segmentation.

[0085] In some embodiments, visual fusion features and text fusion features are aligned according to the type of the object being identified, and the loss value of the alignment module is calculated, including:

[0086] Calculate the mean squared error loss values ​​of visual fusion features and text fusion features based on the type of the object being identified;

[0087] Calculate the loss value of the alignment module based on the mean square error loss value.

[0088] The visual and text fusion features are aligned according to the object type. Specifically, the visual and text fusion features are adjusted into matrices of the same dimension. For the target domain dimension, the order and content of each target domain in both matrices are identical (e.g., target domain 1, target domain 2, and target domain 3). Similarly, for the object type dimension, the order of each object type in both matrices is identical (e.g., object type a and object type b). A domain consistency loss value is calculated based on the mean squared error loss value. Based on this domain consistency loss value, the loss value of the alignment module is determined, thereby forcing domain consistency alignment by aligning the correlation between the visual features of objects in different scenarios and the corresponding cue word features.

[0089] In a specific example, there are m target domains, and visual fusion features Text fusion features The domain alignment loss value is calculated based on the following formula.

[0090]

[0091] By calculating the mean squared error loss values ​​of visual fusion features and text fusion features, and then calculating the loss value of the alignment module, the alignment of image features and text features of the same object category in different target domains can be achieved, realizing domain consistency. From the perspective of consistent object recognition in different target domains, the feature alignment of the objects recognized in multiple domains constrains the output of the alignment module, further improving the accuracy of the target domain image features output by the alignment module, thereby improving the accuracy of target domain image segmentation.

[0092] In some embodiments, the target domain includes a first domain and at least one second domain; the target domain image segmentation model is used to perform semantic segmentation on the image of the first domain; the first domain and the second domain are different, and the first domain includes a specified requirement domain.

[0093] Image segmentation models can perform image segmentation tasks based on a specific target domain. Multiple target domains include a first domain and at least one second domain; the first and second domains are different, and each second domain is also different. The specified requirement domain can refer to the target domain specified by the task requirements; the first domain is usually related to business requirements or user specifications. The second domain is used to improve the domain versatility of the image segmentation model.

[0094] By adding at least one second domain to the first domain of the required segmentation, and combining the first domain with each second domain as training content, the representativeness of features in the samples is increased, the content of the applicable target domain is increased, the image features of the target domain can be adapted to application scenarios, the versatility of the image segmentation model for multiple domains is improved, and the generalization ability of the image segmentation model is improved, thereby improving the accuracy of image segmentation.

[0095] In some embodiments, the text features of the first domain further include the overall text features of the first domain; according to the type of the object being identified, the visual fusion features and the text fusion features are aligned, and the loss value of the alignment module is calculated, including:

[0096] Calculate the domain alignment loss value based on the object type, visual fusion features, and text fusion features;

[0097] Calculate the overall feature alignment loss value based on the image features of the first domain and the overall text features of the first domain;

[0098] The loss value of the alignment module is obtained by weighted summing of the domain alignment loss value and the overall feature alignment loss value.

[0099] The domain alignment loss is calculated by aligning the visual fusion features and text fusion features according to the type of the object being identified, and then calculating the mean squared error loss of the visual fusion features and text fusion features as the domain alignment loss.

[0100] The prompt text for the first domain also includes the overall prompt text for the first domain, and the text features of the first domain also include the overall text features of the first domain. The overall prompt text for the first domain can refer to the global prompt text describing the first domain. The overall prompt text for the first domain is the opposite of the unit prompt text for the first domain; the overall prompt text for the first domain can be understood as the global prompt text, and the unit prompt text for the first domain can be understood as the local prompt text. For example, the overall prompt text for the first domain is: "In the rain." The identified object types include: tree, sign, vehicle, and pedestrian. Correspondingly, the unit prompt text for the first domain includes: tree in the rain, sign in the rain, vehicle in the rain, and pedestrian in the rain, etc.

[0101] Both the image features and the overall text features of the first domain contain descriptions of the target domain. Therefore, the image features and the overall text features of the first domain can be aligned as a whole according to the first domain. This constrains the alignment module to generate accurate image features of the first domain, thereby adjusting the alignment module parameters.

[0102] The image features of the first domain are f s→t The overall textual features of the first domain Overall feature alignment loss value It can be calculated using the following formula:

[0103]

[0104] in, By f s→t It is obtained through attention-based pooling operations.

[0105] Domain alignment loss value Overall feature alignment loss value Loss value of alignment module Calculated based on the following formula:

[0106]

[0107] Where, λ g and λ DC It is the weighting coefficient, an adjustable parameter that can be set to 1.

[0108] By globally aligning the image features and overall text features of the first domain, the overall feature alignment loss is calculated. The overall feature alignment loss and the domain alignment loss are then weighted and summed to obtain the loss value of the alignment module. The loss value of the alignment module can be calculated from the perspective of combining a specific domain with multiple general domains. This allows for targeted and general feature alignment constraints on the output of the alignment module, further improving the accuracy of the target domain image features output by the alignment module, thereby improving the accuracy of target domain image segmentation.

[0109] The technical solution of this application embodiment obtains image features and text features of multiple target domains, fuses them for multiple target domains to obtain visual fusion features and text fusion features, and aligns the visual fusion features and text fusion features to train an alignment module. This enables the alignment module to align images and text in multiple target domains, and the target domain image decoder in the target domain image segmentation model is trained based on the alignment module. This solves the problem in the prior art that the image segmentation model needs to be trained specifically for the target domain. It can improve the generalization ability of the image segmentation model, improve the efficiency of image segmentation, and improve the accuracy of image segmentation for general target domains.

[0110] like Figure 3 As shown, this application provides a training method for a target domain image decoder, which is applied to... Figure 1 The method will be illustrated using terminal 102 or server 104 as examples. It is understood that the computer device may include at least one of a terminal and a server. The method includes the following steps:

[0111] It should be noted that for parts not described in detail in the embodiments of this application, please refer to the descriptions in other embodiments.

[0112] S301. Obtain the second training sample, which includes the source domain image, the prompt text of the first domain, and the ground truth of the source domain image segmentation; the first domain is the specified requirement domain.

[0113] The second training sample is used to train the target domain image decoder. The source domain image and source domain image segmentation ground truth in the second training sample can be the same as those in the first training sample. The prompt text for the first domain in the second training sample includes the complete prompt text for the first domain. The second training sample does not include content from the second domain, only content from the first domain. The image decoder is actually a decoder that processes the image features of the first domain; correspondingly, the training samples for training this image decoder may not include content from the second domain.

[0114] S302. Extract features from the source domain image to obtain source domain image features; extract features from the prompt text of the first domain to obtain text features of the first domain.

[0115] S303. The source domain image features and the text features of the first domain are input into the alignment module for processing, and the target domain image features are output; wherein, the alignment module is trained by the alignment module training method of any embodiment of this application.

[0116] When the alignment module training is complete, the parameters of the alignment module are frozen, and the target domain image decoder is trained using the second training sample. The target domain image features are the image features of the first domain.

[0117] S304. Input the target domain image features and the text features of the first domain into the corrector for processing, and output the corrected target domain image features.

[0118] The corrector is used to correct the image features of the first domain, making the corrected image features of the target domain closer to the real image features. In fact, after the alignment module is trained, there is still a domain gap between the image features of the first domain and the image features extracted from the real first domain image, which needs to be corrected. By correcting the image features of the first domain using the corrector, the prediction results output by the alignment module of the first domain can be made closer to the real results, further improving the prediction accuracy of the image features of the target domain.

[0119] S305. Input the corrected target domain image features into the target domain image decoder for processing, and output the target domain image segmentation result.

[0120] The image decoder processes image features to obtain the semantic segmentation result of the image corresponding to those features. The target domain image decoder processes target domain image features to obtain the semantic segmentation result of the target domain image corresponding to those features. The target domain image segmentation result is the prediction result of the target domain image decoder. For example, the image decoder can be DeepLabv3+.

[0121] S306. Adjust the parameters of the corrector and the target domain image decoder based on the difference between the target domain image segmentation result and the source domain image segmentation true value.

[0122] The corrector and the target domain image decoder are trained simultaneously. The difference between the target domain image segmentation result and the source domain image segmentation ground truth is used to train both the corrector and the target domain image decoder.

[0123] In the ground truth segmentation values ​​for both the source and target domains, the source domain is the background. Therefore, the ground truth segmentation values ​​for the source domain can be directly used as the ground truth segmentation values ​​for the target domain. The difference between the target domain segmentation result and the ground truth segmentation value for the source domain can be calculated as the discrepancy. Based on this difference, the loss value of the target domain image decoder can be calculated. The parameters of the target domain image decoder are then adjusted according to the loss value.

[0124] The technical solution of this application embodiment corrects the image features output by the alignment module through a corrector, making the output image features closer to the real image features, thereby improving the accuracy of the predicted image features. This trains the image decoder and further improves the prediction accuracy of the image decoder. At the same time, by generating a target domain image decoder based on the source domain image, that is, simulated image features applied to the target domain, the target domain image decoder can perform semantic segmentation based on the target domain image features. This enables semantic segmentation of the target domain image, and can achieve zero-sample domain adaptive image segmentation, reducing the sample generation cost of target domain image segmentation and improving the training efficiency of the target domain image segmentation model.

[0125] like Figure 4 As shown, this application provides an image segmentation method, which is applied to... Figure 1 The method will be illustrated using terminal 102 or server 104 as examples. It is understood that the computer device may include at least one of a terminal and a server. The method includes the following steps:

[0126] S401. The acquired target domain image is input into the trained target domain image segmentation model; the target domain image segmentation model includes an image encoder and a target domain image decoder, and the target domain image decoder is trained by the training method of the target domain image decoder as described in any embodiment of this application.

[0127] Target domain image can refer to an image in the application scenario of the target domain.

[0128] The image encoder can be any of the image encoders described in any embodiment of this application. In the foregoing embodiments, the image encoder for extracting features from the source domain image can be the same as the image encoder for extracting features from the target domain image. The image encoder can be CLIP-ResNet-50.

[0129] S402. Obtain the target domain image features through the image encoder.

[0130] S403. Input the target domain image features into the target domain image decoder for processing, and output the target domain image segmentation result.

[0131] S404. Determine the target object based on the target domain image segmentation results.

[0132] The target domain image segmentation result includes at least one object of a recognized object type. Different objects correspond to image regions containing pixels with different pixel values. At least one target object is extracted from the target domain image segmentation result. Specifically, pixels with the same pixel value in the target domain image segmentation result are identified as belonging to the same type of target object. Pixels with the same pixel value in the same enclosed region are defined as pixels included in a single target object. The target object is then identified among the objects identified in the target domain image segmentation result. For example, the target domain is "rain." The target domain image is an image captured during a rainy scene. The target objects to be identified are a car and a person in the rain.

[0133] The technical solution of this application embodiment extracts features from the target domain image through an image encoder to obtain target domain image features, processes the target domain image features through a target domain image decoder to obtain target domain segmentation results, realizes object recognition in the target domain image, realizes semantic segmentation of the target domain image, can realize zero-sample domain adaptive image segmentation, reduces the sample generation cost of target domain image segmentation, improves the training efficiency of the target domain image segmentation model, and thus improves the implementation speed of target domain image segmentation.

[0134] In a specific application scenario, the training method of the alignment module provided in this application embodiment uses domain consistency loss to achieve multimodal image-text domain consistency alignment, and uses a text prompt word-driven visual feature corrector to achieve alignment between simulated features of the target domain image and real target domain image features, thereby enabling the training of a general cross-domain semantic segmentation model.

[0135] First, the structure of the entire training system is as follows: Figure 5As shown. The entire training process is as follows: Training is divided into two stages. In the first stage, two learnable parameters μ and σ are trained mainly for the alignment module of multimodal image-text domain consistency. In the second stage, the values ​​of parameters μ and σ of the alignment module in the first stage are frozen, and a target domain image decoder is trained as a general cross-domain semantic segmentation model. At the same time, a parameter β and a simple linear mapping layer are trained for the proposed text cue word-driven visual feature corrector. and As a domain correction parameter.

[0136] The target domain prompt text includes the overall target domain prompt text and at least one target domain unit prompt text indicating that the identified object type is within the target domain. The text features of the target domain include: overall text features and category text features.

[0137] First, let's introduce the second stage. In the second stage, the source domain image is already available. and its segmentation real mask Use target domain overall prompt text p t The image describes the features of the target domain; for example, "driving under rain" describes the image features of the target domain of driving in the rain. The source domain image is then processed by a fixed-parameter image encoder E. img Obtain image features The entire target domain prompt text p t Text encoder E with fixed parameters text Obtain overall text features Next, the obtained image features and overall text features are passed through an alignment module and a corrector to obtain corrected target domain image features. Through a learnable, general cross-domain semantic segmentation model, the corrected target domain image features f are segmented... t The image is decoded into a target domain image segmentation result. By calculating the segmentation loss between the target domain image segmentation result and the known source domain image segmentation truth mask, a general image decoder is obtained as a general cross-domain semantic segmentation model. At the same time, the linear mapping layer of the learnable parameter β and the text prompt word-driven visual feature corrector is optimized.

[0138] The reasoning process of the image segmentation model is as follows: Figure 6 As shown: After training the image decoder for the target domain, the parameters of the image decoder are fixed. The target domain image is then processed by the image encoder to obtain the target domain image features, and finally by the image decoder to obtain the final target domain image segmentation result. At this point, there is no alignment module or corrector involved.

[0139] The core of this method is described below: the multimodal graph-text consistency alignment module and the text cue word-driven visual feature corrector in the first stage of the training process.

[0140] The alignment module aims to consistently align the inherent correlations between visual features of recognized objects in images from different domains with the inherent correlations between textual features describing those objects in different domains. For example... Figure 7 As shown, for example, street view images of rainy days and snowy days are considered as two different domains. In each domain, each type of object can be abstracted into a feature as its visual characteristic, and text can be used to describe each type of object in each domain. For example, in the domain of a rainy day street view image, the objects can be described as "bus in the rain" or "rider in the rain"; in the domain of a snowy day street view image, the objects can be described as "bus in the snow" or "rider in the snow". The goal is to achieve consistent alignment between the inherent correlation between the visual features of the objects in the images of different domains and the inherent correlation between the textual features describing the objects in different domains. Figure 8 As shown, the training steps for the alignment module are as follows:

[0141] (1) Source domain image and the overall hint text p of the target domain t Image encoder E with fixed parameters img and text encoder E text Obtain source domain image features and the overall text features of the target domain To map source domain image features to the target domain, two learnable parameters μσ are introduced to represent the parameters of the alignment module. These two learnable parameters are updated based on the guidance of target domain prompts. Specifically, the alignment module performs the following calculations:

[0142] The above formula can be understood as: mapping the source domain image features to the target domain through the target domain image feature parameters, μ(f s ) and σ(f s ) are f s The mean and variance of f. s→t This can be understood as image features of an image that possesses both the content of the original source domain image and the stylistic features of the target domain. Utilizing overall text features... To guide the update of the alignment module, the loss function is used to calculate the overall feature alignment loss of the image, and the overall feature alignment loss value is...

[0143]

[0144] in By f s→tIt is obtained through attention-based pooling operations.

[0145] (2) Obtain the visual features of objects in different domain images by utilizing the semantic information in the ground truth of the image, i.e., the regional features of each target domain. For example, suppose the source domain image is a sunny street scene in an autonomous driving scenario, and the objects that can be identified in the image include n types of objects such as buses and roads; the ground truth of this image, i.e., the source domain image segmentation ground truth, is a pixel-level category map. It can be resized and converted into a categorical truth feature y. s * ∈{0,1} n×h×w First, the regional features of the target domain are obtained through max pooling.

[0146]

[0147] Assuming there are m target domains, for each target domain, we can obtain the visual features of the objects in that domain, i.e., the region features C of that target domain. Therefore, combining the region features of all target domains yields the visual fusion feature. Similarly, we can obtain the unit text prompt text for each target domain by describing each recognition object in these m target domains (for example, for the rain scene domain, we can describe the recognition object as "bus in the rain", "rider in the rain", etc.), and obtain the category text features of each target domain by using a text encoder. We can then fuse the category text features of each target domain to obtain the text fusion feature. The following domain consistency loss is used to force domain consistency alignment by aligning the correlation between visual features of objects and corresponding cue word features in different scenes:

[0148]

[0149] Finally, in the first stage, the loss value of the alignment module is calculated. The parameters are updated using the gradients of the learnable parameters μ and σ of the alignment module, where λ g and λ DC It is an adjustable weighting coefficient.

[0150]

[0151] Finally, after the first stage of training, the image features f of the target domain are obtained by calculating the learned parameters μ and σ. s→t .

[0152] Next, a text-based cue-driven visual feature corrector is trained. For example... Figure 9As shown, the target domain image simulates features f s→t A domain gap still exists between the distribution of features in the target domain image and the distribution in the actual target domain image. Therefore, a learnable parameter β and a simple linear mapping layer l are used for correction. Training the corrector is essentially adjusting the learnable parameter β and the linear mapping layer l (domain correction parameter). and The linear mapping layer l takes the overall prompt text of the target domain as input and outputs the domain correction parameters corresponding to the features of the target domain. and

[0153]

[0154] The following is the formula for obtaining the features of the corrected target domain image, where f s→t These are the simulated features of the target domain image before correction:

[0155]

[0156] like Figure 10 As shown, the specific application process of the text cue word-driven visual feature corrector is as follows: During the second stage of training the general cross-domain semantic segmentation model, the following steps are taken: As input to a general image decoder, the learnable parameters β and the parameters of the linear mapping layer l are updated simultaneously while updating the cross-domain semantic segmentation model.

[0157] In a specific example, CLIP-ResNet-50 is used as a pre-trained image encoder and text encoder with fixed parameters, and DeepLabv3+ is used as the target domain image decoder.

[0158] This application's embodiments achieve more efficient multimodal image-text domain consistency alignment. Simultaneously, by utilizing a text-driven visual feature corrector, a general cross-domain semantic segmentation model can be trained. The following are the quantitative experimental comparison results: using the Cityscapes dataset as the source domain (training dataset) and the ACDC (Automatic Cardiac Diagnosis Challenge) dataset as the target domain (test dataset), the mean Intersection of Union (mIoU) is used as the quantitative evaluation metric. The experimental results using the multimodal image-text semantic information alignment method proposed in this application's embodiments are shown in Table 1 below.

[0159] Table 1

[0160] experiment mIoU Image and text alignment, without the method provided in the embodiments of this application. 40.52 Image and text alignment + alignment module 41.49 Image and text alignment + corrector 41.35 Image and text alignment + alignment module + corrector 42.47

[0161] Using the multimodal graph-text consistency alignment and text-driven visual feature corrector proposed in this application, experiments were conducted on street view images in driving scenarios, comparing results from a source domain of clear street view images to a target domain of street view images with varying degrees of adverse weather conditions. The Cityscapes dataset was selected as the source domain, and the ACDC dataset as the target domain. mIoU and mean accuracy (mAcc) were used as quantitative evaluation metrics. The experimental results are shown in Table 2 below:

[0162] Table 2

[0163]

[0164] Here, the Source method can refer to a method that uses a segmentation model trained on a source domain image to process a target domain image. CLIPStyler can refer to image style transfer with a single text condition. (Prompt-driven Zero-shot Domain Adaptation) can refer to a prompt-driven zero-shot domain adaptation method. It can be Improved methods.

[0165] In the table above, the multimodal graph-text consistency alignment and text cue word-driven visual feature corrector proposed in this application embodiment are combined with other improvements, but the conclusion that the embodiments of this application are practical and effective can still be drawn from them.

[0166] Using the proposed multimodal graph-text consistency alignment and text-driven visual feature corrector, experiments were conducted on street view images in driving scenarios, comparing the source domain from game-generated images to the target domain from real street view images. The GTA5 (Grand Theft Auto 5, a synthetic image dataset with pixel-level semantic annotation) dataset was selected as the source domain, and the Cityscapes and ACDC datasets as the target domains. mIoU and mAcc were used as quantitative evaluation metrics. The experimental results are shown in Table 3 below.

[0167] Table 3

[0168]

[0169] In Table 3 above, the multimodal graph-domain consistency alignment and text cue word-driven visual feature corrector proposed in this application are combined with other improvements, but it is still possible to conclude that the method is effective.

[0170] In this embodiment, CLIP-ResNet-50 is used as a pre-trained image encoder and text encoder with fixed parameters, and DeepLabv3+ is used as the image segmentation decoder. Theoretically, these pre-trained models can be replaced with any effective pre-trained encoder model.

[0171] In the practice of the embodiments of this application, for the proposed loss weight λ g and λ DC It can be adjusted as a hyperparameter.

[0172] In the practice of this application embodiment, the learnable parameter β is initialized to 0.1 by default, but its learnability and its initial value can be effectively adjusted.

[0173] In the practice of this application's embodiments, hyperparameters such as optimizer and learning rate during the training process can be adjusted arbitrarily and effectively.

[0174] Existing methods only utilize global image context information, which ignores the consistency between multimodal domains. This may cause the model to fail to learn the intrinsic correlation of image feature semantic information, resulting in the offset and loss of feature semantic information. At the same time, a specific segmentation model needs to be trained for each possible target domain, which makes it impossible to train a general cross-domain semantic segmentation model, thus resulting in poor flexibility.

[0175] This application's embodiments extract object features from images as semantic information of local image regions by introducing ground truth from image semantic segmentation tasks, and achieve consistent alignment of multimodal image-text domains using more refined text prompts. This optimizes the model's learning of the intrinsic correlation of semantic information of image features to a certain extent. By proposing a text prompt-driven visual feature corrector, alignment between simulated features of the target domain image and real target domain image features is achieved, thereby enabling the training of a general cross-domain semantic segmentation model without introducing more learnable parameters. At the same time, this method has strong transferability and can be applied to any scheme that requires consistent alignment of multimodal domains.

[0176] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0177] Based on the same inventive concept, this application also provides a training device for an alignment module. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more alignment module training device embodiments provided below can be found in the limitations of the alignment module training method described above, and will not be repeated here.

[0178] like Figure 11 As shown, this application embodiment provides a training device 1100 for an alignment module, comprising:

[0179] The first training sample acquisition module 1101 is used to acquire the first training sample, which includes a source domain image, a prompt text for the target domain, and ground truth values ​​for source domain image segmentation. The number of target domains is multiple.

[0180] The image and text feature extraction module 1102 is used to extract features from the source domain image to obtain source domain image features; and to extract features from the prompt text of each target domain to obtain text features of each target domain.

[0181] The target domain image feature conversion module 1103 is used to input the source domain image features and the text features of each target domain into the alignment module for processing, and output the image features of each target domain.

[0182] The target domain image-text feature fusion module 1104 is used to fuse the image features of each target domain according to the object type based on the source domain image segmentation ground value to obtain visual fusion features; and to fuse the text features of each target domain according to the object type to obtain text fusion features.

[0183] The alignment module parameter adjustment module 1105 is used to align visual fusion features and text fusion features according to the type of the object being identified, calculate the loss value of the alignment module, and adjust the parameters of the alignment module according to the loss value. The alignment module is used to train the target domain image decoder in the target domain image segmentation model. The target domain image segmentation model is used to perform semantic segmentation on the target domain image.

[0184] In some embodiments, the target domain image-text feature fusion module 1104 is specifically used for:

[0185] Based on at least one object type, the source domain image segmentation ground value is transformed and max-pooled to obtain the category ground value features;

[0186] Based on the image features of each target domain, and according to the true value features of the categories, the features corresponding to each object type are extracted from the image features of the target domain to obtain the regional features of the target domain;

[0187] The regional features of each target domain are fused to obtain visual fusion features.

[0188] In some embodiments, the text features of the target domain include at least one category text feature that identifies the type of object; the target domain text-image feature fusion module 1104 is specifically used to: fuse the category text features of each type of object in each target domain to obtain text fusion features.

[0189] In some embodiments, the alignment module parameter adjustment module 1105 is specifically used for:

[0190] Calculate the mean squared error loss values ​​of visual fusion features and text fusion features based on the type of the object being identified;

[0191] Calculate the loss value of the alignment module based on the mean square error loss value.

[0192] In some embodiments, the target domain includes a first domain and at least one second domain; the target domain image segmentation model is used to perform semantic segmentation on the image of the first domain; the first domain and the second domain are different, and the first domain includes a specified requirement domain.

[0193] In some embodiments, the text features of the first field further include the overall text features of the first field; the alignment module parameter adjustment module 1105 is specifically used for:

[0194] Calculate the domain alignment loss value based on the object type, visual fusion features, and text fusion features;

[0195] Calculate the overall feature alignment loss value based on the image features of the first domain and the overall text features of the first domain;

[0196] The loss value of the alignment module is obtained by weighted summing of the domain alignment loss value and the overall feature alignment loss value.

[0197] Each module in the training device for the alignment module described above can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0198] Based on the same inventive concept, this application also provides a training apparatus for a target domain image decoder. The solution provided by this apparatus is similar to the solution described in the above method. Therefore, the specific limitations of one or more training apparatus embodiments for target domain image decoders provided below can be found in the limitations of the training method for target domain image decoders described above, and will not be repeated here.

[0199] like Figure 12 As shown, this application embodiment provides a training device 1200 for a target domain image decoder, including:

[0200] The second training sample acquisition module 1201 is used to acquire the second training sample, which includes the source domain image, the prompt text of the first domain, and the source domain image segmentation ground truth; the first domain is the specified requirement domain.

[0201] The image and text feature extraction module 1202 is used to extract features from the source domain image to obtain source domain image features; and to extract features from the prompt text of the first domain to obtain text features of the first domain.

[0202] The target domain image feature conversion module 1203 is used to input the source domain image features and the text features of the first domain into the alignment module for processing, and output the target domain image features; wherein, the alignment module is trained by the alignment module training method of any embodiment of this application;

[0203] The image feature correction module 1204 is used to input the target domain image features and the text features of the first domain into the corrector for processing, and output the corrected target domain image features.

[0204] The segmentation prediction module 1205 is used to input the corrected target domain image features into the target domain image decoder for processing and output the target domain image segmentation result.

[0205] The decoder parameter adjustment module 1206 is used to adjust the parameters of the corrector and the target domain image decoder based on the difference between the target domain image segmentation result and the source domain image segmentation ground truth.

[0206] Each module in the training device of the aforementioned target domain image decoder can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0207] Based on the same inventive concept, this application also provides an image segmentation apparatus. The solution provided by this apparatus is similar to the solution described in the above method. Therefore, the specific limitations of one or more image segmentation apparatus embodiments provided below can be found in the limitations of the image segmentation method above, and will not be repeated here.

[0208] like Figure 13 As shown, this application embodiment provides an image segmentation apparatus 1300, including:

[0209] The target domain image acquisition module 1301 is used to input the acquired target domain image into a trained target domain image segmentation model; the target domain image segmentation model includes an image encoder and a target domain image decoder, and the target domain image decoder is trained by the training method of the target domain image decoder as described in any embodiment of this application;

[0210] The target domain image feature extraction module 1302 is used to obtain target domain image features through an image encoder;

[0211] The target domain image segmentation module 1303 is used to input the target domain image features into the target domain image decoder for processing and output the target domain image segmentation result.

[0212] The target object determination module 1304 is used to determine the target object based on the target domain image segmentation results.

[0213] Each module in the aforementioned image segmentation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0214] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 14As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to the training methods for alignment modules, target domain image decoders, or image segmentation methods. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the aforementioned training methods for alignment modules, target domain image decoders, or image segmentation methods.

[0215] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 15 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements the steps in the training methods for the alignment module, the target domain image decoder, or the image segmentation method described above. The display unit of the computer device is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen; the input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs or touchpads set on the casing of the computer device, or external keyboards, touchpads or mice, etc.

[0216] Those skilled in the art will understand that Figure 14or Figure 15 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0217] In some embodiments, a computer device is provided, the computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0218] In some embodiments, such as Figure 16 The diagram shows the internal structure of a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the above-described method embodiments.

[0219] In some embodiments, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0220] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0221] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0222] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0223] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A training method for an alignment module, characterized in that, include: Obtain a first training sample, which includes a source domain image, a prompt text for the target domain, and ground truth values ​​for segmentation of the source domain image. The number of target domains is multiple. Feature extraction is performed on the source domain image to obtain source domain image features; feature extraction is performed on the prompt text of each target domain to obtain text features of each target domain; The source domain image features and the text features of each target domain are input into the alignment module for processing, and the image features of each target domain are output. Based on the source domain image segmentation ground truth, the image features of each target domain are fused according to the object type to obtain visual fusion features; The text features of each target domain are fused according to the type of the object being identified to obtain the text fusion features; Based on the type of the object being identified, the visual fusion features and the text fusion features are aligned, and the loss value of the alignment module is calculated; based on the loss value, the parameters of the alignment module are adjusted. The alignment module is used to train the target domain image decoder in the target domain image segmentation model; the target domain image segmentation model is used to perform semantic segmentation on the target domain image.

2. The method according to claim 1, characterized in that, The step of fusing image features of each target domain according to the object type based on the source domain image segmentation ground truth to obtain visual fusion features includes: Based on at least one object type, the source domain image segmentation ground value is transformed and max-pooled to obtain category ground value features; For the image features of each target domain, based on the category ground value features, features corresponding to each recognition object type are extracted from the image features of the target domain to obtain the region features of the target domain; The regional features of each target domain are fused to obtain visual fusion features.

3. The method according to claim 1, characterized in that, The text features of the target domain include at least one category text feature that identifies the object type; The process of fusing the text features of each target domain according to the type of the identified object to obtain text fusion features includes: The text features of each object type in each target domain are fused to obtain text fusion features.

4. The method according to claim 1, characterized in that, The step of aligning the visual fusion features and the text fusion features according to the type of the identified object, and calculating the loss value of the alignment module, includes: Based on the type of the object being identified, calculate the mean squared error loss values ​​of the visual fusion features and the text fusion features; The loss value of the alignment module is calculated based on the mean square error loss value.

5. The method according to claim 1, characterized in that, The target domain includes a first domain and at least one second domain; the target domain image segmentation model is used to perform semantic segmentation on the image of the first domain; the first domain and the second domain are different, and the first domain includes a specified requirement domain.

6. The method according to claim 5, characterized in that, The text features of the first domain also include the overall text features of the first domain; The step of aligning the visual fusion features and the text fusion features according to the type of the identified object, and calculating the loss value of the alignment module, includes: Calculate the domain alignment loss value based on the object type, the visual fusion feature, and the text fusion feature; Calculate the overall feature alignment loss value based on the image features of the first domain and the overall text features of the first domain; The loss value of the alignment module is obtained by weighted summing of the domain alignment loss value and the overall feature alignment loss value.

7. A training method for a target domain image decoder, characterized in that, include: Obtain a second training sample, which includes a source domain image, a prompt text in the first domain, and ground truth values ​​for the segmentation of the source domain image; The first domain is the specified requirement domain; Feature extraction is performed on the source domain image to obtain source domain image features; feature extraction is performed on the prompt text of the first domain to obtain text features of the first domain; The source domain image features and the text features of the first domain are input into the alignment module for processing, and the target domain image features are output; wherein, the alignment module is trained by the method described in any one of claims 1-6; The target domain image features and the text features of the first domain are input into the corrector for processing, and the corrected target domain image features are output. The corrected target domain image features are input into the target domain image decoder for processing, and the target domain image segmentation result is output. The parameters of the corrector and the target domain image decoder are adjusted based on the difference between the target domain image segmentation result and the source domain image segmentation ground truth.

8. An image segmentation method, characterized in that, include: The acquired target domain image is input into a trained target domain image segmentation model; the target domain image segmentation model includes an image encoder and a target domain image decoder, and the target domain image decoder is trained by the method described in claim 7. The target domain image features are obtained through the image encoder. The target domain image features are input into the target domain image decoder for processing, and the target domain image segmentation result is output. The target object is determined based on the target domain image segmentation results.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the training method of the alignment module according to any one of claims 1-6, the training method of the target domain image decoder according to claim 7, or the image segmentation method according to claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the alignment module according to any one of claims 1-6, the training method of the target domain image decoder according to claim 7, or the image segmentation method according to claim 8.

Images