An image segmentation method, device, apparatus and storage medium

Abstract

Embodiments of the present application provide an image segmentation method, device and equipment, and a storage medium, relating to the technical field of artificial intelligence, which comprises: extracting target image features of a target domain image to be processed through a trained target domain image segmentation network, and obtaining a target domain predicted image based on the target image features, the target domain predicted image comprising target objects with target labels; the trained target domain image segmentation network is obtained by jointly and iteratively training a target domain image segmentation network to be trained and a source domain image segmentation network to be trained based on a domain transfer learning manner, and the source domain sample image comprises source domain sample objects with source labels. Through the domain transfer learning manner, the target domain image segmentation network used for segmenting the target domain image is obtained without the need for fine labeling of each pixel in the target domain sample image, thereby reducing the manpower and cost, and improving the image segmentation efficiency and accuracy.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of artificial intelligence, and particularly relate to an image segmentation method and device, equipment and a storage medium. BACKGROUND

[0002] Image segmentation is a basic problem in video and image understanding. The image segmentation process is as follows: after an image x is input into an image segmentation model, the image segmentation model predicts the class of each pixel in the image x, and then the image x is segmented according to the class of each pixel to obtain a predicted image y.

[0003] Since the image segmentation model realizes image segmentation by predicting the class of each pixel in the image, when training the image segmentation model, each pixel point in the sample image needs to be finely labeled by a person. However, the real scene is complex and diverse, such as weather, light, etc., which can cause differences in images of the same object. Collecting images in all scenes and giving corresponding fine labels requires too high manpower and cost, and is low in efficiency. SUMMARY

[0004] Embodiments of the present application provide an image segmentation method, device, equipment and storage medium, which are used to reduce the manpower and cost of image segmentation and improve the efficiency of image segmentation.

[0005] In one aspect, the present application provides an image segmentation method, which comprises:

[0006] Obtaining a target domain image to be processed including a target object;

[0007] Extracting a target image feature of the target domain image to be processed through a trained target domain image segmentation network, and obtaining a target domain predicted image based on the target image feature, the target domain predicted image including a target object with a target label;

[0008] The trained target domain image segmentation network is obtained by jointly and iteratively training a target domain image segmentation network to be trained and a source domain image segmentation network to be trained, and the training samples used include source domain sample images and target domain sample images, wherein the source domain sample images include source domain sample objects with source labels, the target domain image segmentation network to be trained and the source domain image segmentation network to be trained share one encoder to be trained, the target domain image segmentation network to be trained further includes a target domain classifier to be trained, and the source domain image segmentation network to be trained further includes a source domain classifier to be trained.

[0009] In one aspect, the present application provides a method for training a target domain image segmentation network, which comprises:

[0010] jointly iteratively train the to-be-trained target domain image segmentation network and the to-be-trained source domain image segmentation network to obtain a trained target domain image segmentation network, wherein the to-be-trained source domain image segmentation network and the to-be-trained target domain image segmentation network share one to-be-trained encoder, the to-be-trained target domain image segmentation network further comprises a to-be-trained target domain classifier, and the to-be-trained source domain image segmentation network further comprises a to-be-trained source domain classifier;

[0011] In each iteration process, the following operations are performed:

[0012] extract a source domain sample feature of a source domain sample image and a target domain sample feature of a target domain sample image through the to-be-trained encoder, wherein the source domain sample image comprises a source domain sample object with a source label;

[0013] obtain a source domain predicted sample image based on the source domain sample feature through the to-be-trained source domain classifier, wherein the source domain predicted sample image comprises a source domain sample object with a source predicted label;

[0014] obtain a target domain predicted sample image based on the target domain image feature through the to-be-trained target domain classifier, wherein the target domain predicted sample image comprises a target domain sample object with a target predicted label;

[0015] obtain a target loss value based on the source domain predicted sample image and the target domain predicted sample image;

[0016] use the target loss value to respectively adjust parameters of the to-be-trained source domain image segmentation network and the to-be-trained target domain image segmentation network.

[0017] In one aspect, an embodiment of the present application provides an image segmentation device, which comprises:

[0018] an acquisition module configured to acquire a target domain image to be processed comprising a target object;

[0019] an image segmentation module configured to extract a target image feature of the target domain image to be processed through a trained target domain image segmentation network, and obtain a target domain predicted image based on the target image feature, wherein the target domain predicted image comprises the target object with a target label;

[0020] The trained target domain image segmentation network is obtained by jointly and iteratively training a to-be-trained target domain image segmentation network and a to-be-trained source domain image segmentation network, and training samples used in the training include source domain sample images and target domain sample images, wherein the source domain sample images include source domain sample objects with source labels, the to-be-trained target domain image segmentation network and the to-be-trained source domain image segmentation network share a to-be-trained encoder, the to-be-trained target domain image segmentation network further includes a to-be-trained target domain classifier, and the to-be-trained source domain image segmentation network further includes a to-be-trained source domain classifier.

[0021] Optionally, the trained target domain image segmentation network and a trained source domain image segmentation network share a trained encoder, and the trained target domain image segmentation network further includes a trained target domain classifier.

[0022] The image segmentation module is specifically configured to:

[0023] extract target image features of the target domain to-be-processed image through the trained encoder;

[0024] obtain a target domain predicted image based on the target image features through the trained target domain classifier.

[0025] Optionally, the image segmentation apparatus further includes a model training module.

[0026] The model training module is specifically configured to:

[0027] In each iteration process, the following operations are performed:

[0028] extract source domain sample features of the source domain sample images and target domain sample features of the target domain sample images through the to-be-trained encoder respectively;

[0029] obtain a source domain predicted sample image based on the source domain sample features through the to-be-trained source domain classifier, wherein the source domain predicted sample image includes source domain sample objects with source prediction labels;

[0030] obtain a target domain predicted sample image based on the target domain image features through the to-be-trained target domain classifier, wherein the target domain predicted sample image includes target domain sample objects with target prediction labels;

[0031] obtain a target loss value based on the source domain predicted sample image and the target domain predicted sample image;

[0032] use the target loss value to respectively perform parameter adjustment on the to-be-trained source domain image segmentation network and the to-be-trained target domain image segmentation network.

[0033] Optionally, the model training module is specifically configured to:

[0034] determine, by the discriminator, a first probability that the source domain prediction sample image is a source domain image and a second probability that the target domain prediction sample image is a target domain image;

[0035] determine, based on the first probability and the second probability, an adversarial learning loss value between the to-be-trained source domain image segmentation network and the to-be-trained target domain image segmentation network and the discriminator;

[0036] determine, based on the source label and the source prediction label, a segmentation loss value of the to-be-trained source domain image segmentation network;

[0037] obtain the target loss value based on the adversarial learning loss value and the segmentation loss value.

[0038] Optionally, the model training module is specifically configured to:

[0039] determine, based on the network parameters of the to-be-trained source domain classifier and the network parameters of the to-be-trained target domain classifier, a distance loss value between the to-be-trained source domain classifier and the to-be-trained target domain classifier;

[0040] obtain the target loss value based on the adversarial learning loss value, the segmentation loss value, and the distance loss value.

[0041] Optionally, the model training module is specifically configured to:

[0042] determine a maximum mean discrepancy between the source domain prediction sample image and the target domain prediction sample image;

[0043] determine, based on the source label and the source prediction label, a segmentation loss value of the to-be-trained source domain image segmentation network;

[0044] obtain the target loss value based on the maximum mean discrepancy and the segmentation loss value.

[0045] Optionally, the model training module is specifically configured to:

[0046] determine, based on the network parameters of the to-be-trained source domain classifier and the network parameters of the to-be-trained target domain classifier, a distance loss value between the to-be-trained source domain classifier and the to-be-trained target domain classifier;

[0047] obtain the target loss value based on the maximum mean discrepancy, the segmentation loss value, and the distance loss value.

[0048] Optionally, the target domain classifier to be trained and the source domain classifier to be trained include partially shared network parameters.

[0049] In one aspect, an embodiment of the present application provides a device for training a target domain image segmentation network, the device comprising:

[0050] a model training module configured to jointly and iteratively train a target domain image segmentation network to be trained and a source domain image segmentation network to be trained, to obtain a trained target domain image segmentation network, wherein the source domain image segmentation network to be trained and the target domain image segmentation network to be trained share one encoder to be trained, the target domain image segmentation network to be trained further comprises a target domain classifier to be trained, and the source domain image segmentation network to be trained further comprises a source domain classifier to be trained;

[0051] the model training module comprises a feature extraction module, a first prediction module, a second prediction module and a parameter adjustment module;

[0052] the feature extraction module is configured to extract a source domain sample feature of a source domain sample image and a target domain sample feature of a target domain sample image respectively by using the encoder to be trained, wherein the source domain sample image includes a source domain sample object with a source label;

[0053] the first prediction module is configured to obtain a source domain predicted sample image based on the source domain sample feature by using the source domain classifier to be trained, wherein the source domain predicted sample image includes a source domain sample object with a source predicted label;

[0054] the second prediction module is configured to obtain a target domain predicted sample image based on the target domain sample feature by using the target domain classifier to be trained, wherein the target domain predicted sample image includes a target domain sample object with a target predicted label;

[0055] the parameter adjustment module is configured to obtain a target loss value based on the source domain predicted sample image and the target domain predicted sample image, and to adjust parameters of the source domain image segmentation network to be trained and the target domain image segmentation network to be trained respectively by using the target loss value.

[0056] Optionally, the parameter adjustment module is specifically configured to:

[0057] determine, by using a discriminator, a first probability that the source domain predicted sample image is a source domain image and a second probability that the target domain predicted sample image is a target domain image;

[0058] determine, based on the first probability and the second probability, an adversarial learning loss value between the source domain image segmentation network to be trained, the target domain image segmentation network to be trained, and the discriminator;

[0059] determine, based on the source label and the source predicted label, a segmentation loss value of the source domain image segmentation network to be trained;

[0060] obtain the target loss value based on the adversarial learning loss value and the segmentation loss value.

[0061] Optionally, the parameter adjustment module is specifically configured to:

[0062] determine a distance loss value between the source domain classifier to be trained and the target domain classifier to be trained based on the network parameters of the source domain classifier to be trained and the network parameters of the target domain classifier to be trained;

[0063] obtain the target loss value based on the adversarial learning loss value, the segmentation loss value, and the distance loss value.

[0064] In an aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes steps of the image segmentation method or steps of the method for training the target domain image segmentation network.

[0065] In an aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program executable by a computer device, and when the program is executed on the computer device, the computer device executes steps of the image segmentation method or steps of the method for training the target domain image segmentation network.

[0066] In the embodiment of the present application, the source domain image segmentation network is trained in a supervised learning manner, and meanwhile, the segmentation mode of the source domain image segmentation network is migrated to the target domain image segmentation network, so that the target domain image segmentation network for segmenting target domain images is trained without the need of fine labeling of each pixel in the target domain sample images, thereby reducing the manpower and cost and improving the image segmentation efficiency and accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0068] Figure 1 A system architecture schematic diagram applicable to an embodiment of the present application;

[0069] Figure 2 A flowchart of an image segmentation method provided by an embodiment of the present application;

[0070] Figure 3a A schematic diagram of a target domain image to be processed provided by an embodiment of the present application;

[0071] Figure 3b A schematic diagram of a target domain predicted image provided by an embodiment of the present application;

[0072] Figure 4 A structural schematic diagram of a target domain image segmentation network provided by an embodiment of the present application;

[0073] Figure 5 A structural schematic diagram of a neural network architecture provided by an embodiment of the present application;

[0074] Figure 6 A flowchart of a method for training a target domain image segmentation network provided by an embodiment of the present application;

[0075] Figure 7 A structural schematic diagram of a neural network architecture provided by an embodiment of the present application;

[0076] Figure 8 A structural schematic diagram of a neural network architecture provided by an embodiment of the present application;

[0077] Figure 9 A structural schematic diagram of an image segmentation device provided by an embodiment of the present application;

[0078] Figure 10 A structural schematic diagram of a device for training a target domain image segmentation network provided by an embodiment of the present application;

[0079] Figure 11 A structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0080] In order to make the objectives, technical solutions and beneficial effects of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0081] For the convenience of understanding, the terms involved in the embodiments of the present application are explained below.

[0082] Artificial Intelligence (AI) is the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0083] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0084] Machine Learning (ML) is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and example-based learning technologies. For example, in the embodiments of the present application, the domain transfer learning technology is used to train the target domain image segmentation network, and then the target domain image segmentation network is used to segment the target domain image.

[0085] Deep neural network refers to a modeling method of mapping relationship y=G(x) in artificial intelligence. Neural networks with strong generalization ability can accurately predict the corresponding labels y for different inputs x. Deep neural networks generally contain a large number of parameters, and in order to obtain good generalization ability, a large amount of labeled data (x, y) is needed to train the parameters in the neural network.

[0086] Number of neural network layers: a neural network is generally composed of multiple layers of different types of network layers, such as convolutional layers, fully connected layers, activation layers, etc. For example, neural network AlexNet includes 8 network layers, of which 5 network layers are convolutional layers and 3 network layers are fully connected layers; neural network ResNet-50 includes 50 network layers, and neural network ResNet-101 includes 101 network layers.

[0087] Domain transfer learning: a learning method. The training sample includes source domain images and target domain images, wherein the object labels in the source domain images are pre-labeled, and the objects in the target domain images lack labels. The image textures, illumination, etc. of the source domain images and the target domain images are different, such as the source domain images are computer synthesized and the target domain images are collected from real scenes; or the source domain images and the target domain images are collected from different real scenes. Using the above training sample, the neural network is trained to have good generalization ability on the target domain, that is, to accurately predict the labels of the objects in the target domain images.

[0088] Adversarial learning: a learning method. In domain transfer, given two domains of images, a main network that solves a specific problem (classification or segmentation) and a new discriminator, the predicted images or feature images output by the main network are input into the discriminator in an adversarial learning manner, so that the main network focuses on the information shared between the domains and suppresses the information unique to each domain. The learning method is to optimize the discriminator to correctly determine which domain its input belongs to, and at the same time, optimize the main network to produce output that confuses the discriminator. The adversarial method is to calculate the loss function and the corresponding gradient generated by the discriminator, and use gradient ascent method to increase the value of the loss function for the discriminator, and use gradient descent method to reduce the value of the loss function for the main network, so as to confuse the discriminator. When the discriminator cannot successfully determine which domain its input (i.e. the output of the main network) comes from, it is considered that the output of the main network does not contain domain-specific information, that is, the image information unique to each domain is eliminated, and the image information common to the domains is retained.

[0089] The design idea of the embodiments of the present application will be introduced below.

[0090] Image segmentation is a basic problem in video and image understanding. Since the image segmentation model is implemented by predicting the class of each pixel in the image, when training the image segmentation model, each pixel in the sample image also needs to be finely labeled by humans. However, real scenes are complex and diverse, such as weather, lighting, etc. which can cause differences in images of the same object. Collecting images in all scenes and providing corresponding fine labels requires too much manpower and cost, and is low in efficiency.

[0091] Through analysis, it is found that although the source domain image and the target domain image correspond to different scenes, such as street images collected on sunny and rainy days, the source domain image and the target domain image include the same semantic elements, such as cars, people, trees, roads, etc. Therefore, it can be considered to migrate the segmentation mode of the source domain image to the target domain image by domain migration, so that it is not necessary to finely annotate each pixel in the target domain image for training the target domain image segmentation network, thereby reducing manpower and cost and improving image segmentation efficiency.

[0092] In view of this, the embodiment of the present application provides an image segmentation method, which comprises: acquiring a target domain to-be-processed image comprising a target object, then extracting a target image feature of the target domain to-be-processed image through a trained target domain image segmentation network, and obtaining a target domain predicted image based on the target image feature. Wherein, the target domain predicted image comprises the target object with a target label, the trained target domain image segmentation network is obtained by jointly and iteratively training a to-be-trained target domain image segmentation network and a to-be-trained source domain image segmentation network, and the training sample used comprises a source domain sample image and a target domain sample image, wherein the source domain sample image comprises a source domain sample object with a source label, the to-be-trained target domain image segmentation network and the to-be-trained source domain image segmentation network share a to-be-trained encoder, the to-be-trained target domain image segmentation network further comprises a to-be-trained target domain classifier, and the to-be-trained source domain image segmentation network further comprises a to-be-trained source domain classifier.

[0093] In the embodiment of the present application, the source domain image segmentation network is trained in a supervised learning manner, and at the same time, the segmentation mode of the source domain image segmentation network is migrated to the target domain image segmentation network, so as to train the target domain image segmentation network for segmenting the target domain image without the need for finely annotating each pixel in the target domain sample image, thereby reducing manpower and cost, and improving image segmentation efficiency and accuracy.

[0094] Reference Figure 1 It is a system architecture diagram of the image segmentation method provided by the embodiment of the present application. The architecture at least comprises a terminal device 101 and a server 102.

[0095] The image segmentation application is installed in the terminal device 101, wherein the image segmentation application can be a client application, a web application, a program application, etc. The terminal device 101 can comprise one or more processors 1011, a memory 1012, an I / O interface 1013 for interacting with the server 102, a display panel 1014, etc. The terminal device 101 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.

[0096] The server 102 can be a background server of an image segmentation application and provide corresponding services for the image segmentation application. The server 102 can include one or more processors 1021, a memory 1022, an I / O interface 1023 for interacting with the terminal device 101, and the like. In addition, the server 102 can further be configured with a database 1024. The server 102 can be a standalone physical server, a server cluster or a distributed system formed by multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, and the like. The terminal device 101 and the server 102 can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application.

[0097] The image segmentation method in the embodiments of the present application can be executed by the terminal device 101 or the server 102.

[0098] In the first case, the image segmentation method can be executed by the terminal device 101.

[0099] The terminal device 101 stores a trained target domain image segmentation network. The trained target domain image segmentation network is obtained by jointly and iteratively training a to-be-trained target domain image segmentation network and a to-be-trained source domain image segmentation network, and the training samples used include source domain sample images and target domain sample images, wherein the source domain sample images include source domain sample objects with source labels. The to-be-trained target domain image segmentation network and the to-be-trained source domain image segmentation network share a to-be-trained encoder. The to-be-trained target domain image segmentation network further includes a to-be-trained target domain classifier, and the to-be-trained source domain image segmentation network further includes a to-be-trained source domain classifier.

[0100] The terminal device 101 obtains a target domain to-be-processed image including a target object, then extracts target image features of the target domain to-be-processed image through the trained target domain image segmentation network, and obtains a target domain predicted image based on the target image features, wherein the target domain predicted image includes a target object with a target label. The terminal device 101 displays the target domain predicted image in a display interface.

[0101] In the second case, the image segmentation method can be executed by the server 102.

[0102] Server 102 stores a trained target domain image segmentation network. The trained target domain image segmentation network is obtained by jointly iteratively training the target domain image segmentation network to be trained and the source domain image segmentation network to be trained. The training samples used include source domain sample images and target domain sample images. The source domain sample images include source domain sample objects with source labels. The target domain image segmentation network to be trained and the source domain image segmentation network to be trained share a single encoder to be trained. The target domain image segmentation network to be trained also includes a target domain classifier to be trained, and the source domain image segmentation network to be trained also includes a source domain classifier to be trained.

[0103] Terminal device 101 acquires a target domain image to be processed, including the target object, and then sends the target domain image to be processed to server 102. Server 102 extracts target image features from the target domain image to be processed using a trained target domain image segmentation network, and obtains a target domain prediction image based on the target image features. The target domain prediction image includes the target object with the target label. Server 102 sends the target domain prediction image to terminal device 101, and terminal device 101 displays the target domain prediction image on a display interface.

[0104] Furthermore, the process of jointly iteratively training the target domain image segmentation network to be trained and the source domain image segmentation network to be trained to obtain the trained target domain image segmentation network can be performed by the terminal device 101 or by the server 102. This application does not make any specific limitations on this.

[0105] based on Figure 1 The system architecture diagram shown in this application illustrates the flowchart of an image segmentation method. Figure 2 As shown, the process of this method can be... Figure 1 The terminal device 101 or server 102 shown executes the following steps:

[0106] Step S201: Obtain the target domain image to be processed, including the target object.

[0107] Specifically, the target object can be all or part of the objects in the target domain image to be processed, such as people, plants, animals, buildings, roads, etc.

[0108] The source domain and the target domain correspond to different scenes. For example, the source domain corresponds to a sunny day scene, and the target domain corresponds to a rainy day scene. For another example, the source domain corresponds to a daytime scene, and the target domain corresponds to a night scene. For another example, the source domain corresponds to Beijing, and the target domain corresponds to Singapore. For another example, the source domain corresponds to an artificial synthetic scene, and the target domain corresponds to a real collected scene. An image obtained in the source domain scene is a source domain image, and an image obtained in the target domain scene is a target domain image. It should be noted that the scenes corresponding to the source domain and the target domain are not limited to the above-mentioned several examples, but can also be other forms, and the present application does not make a specific limitation in this regard.

[0109] Since the source domain image and the target domain image are images obtained in different scenes, the textures, illuminations, and visibility of the source domain image and the target domain image are different, but the same label space exists between the two, that is, the semantic elements are the same.

[0110] For example, the source domain image and the target domain image are both street images, wherein the source domain image is a daytime collected street image, and the target domain image is a nighttime collected street image. The semantic elements in the source domain image and the target domain image are the same, including cars, pedestrians, trees, roads, and the like.

[0111] In step S202, the target image features of the target domain image to be processed are extracted through the trained target domain image segmentation network, and the target domain predicted image is obtained based on the target image features.

[0112] Specifically, the trained target domain image segmentation network is obtained by jointly and iteratively training the to-be-trained target domain image segmentation network and the to-be-trained source domain image segmentation network, and the training samples used include source domain sample images and target domain sample images, wherein the source domain sample images include source domain sample objects with source labels, the to-be-trained target domain image segmentation network and the to-be-trained source domain image segmentation network share one to-be-trained encoder, the to-be-trained target domain image segmentation network further includes a to-be-trained target domain classifier, and the to-be-trained source domain image segmentation network further includes a to-be-trained source domain classifier. Optionally, in the embodiment of the present application, the trained target domain image segmentation network can be saved on a blockchain.

[0113] In a specific implementation, the target domain image segmentation network to be trained and the source domain image segmentation network to be trained can be jointly iteratively trained based on a domain transfer learning manner to obtain the trained target domain image segmentation network. The domain transfer learning is a learning manner for transferring the image segmentation mode of the source domain image segmentation network to the target domain image segmentation network. In the training process, the source domain sample image input to the source domain image segmentation network to be trained includes a pre-labeled label, that is, the source domain image segmentation network is trained in a supervised learning manner. The target domain sample image input to the target domain image segmentation network to be trained does not have a pre-labeled label, that is, the target domain image segmentation network is trained in an unsupervised domain transfer manner.

[0114] The target domain image to be processed is input into the trained target domain image segmentation network to obtain a target domain prediction image, wherein the target domain prediction image includes a target object with a target label, that is, the segmented target object is displayed in the target domain prediction image.

[0115] For example, as shown in FIG. 3, the target domain image to be processed includes a vehicle 301 and a street lamp 302. The target domain image to be processed is input into the trained target domain image segmentation network to obtain a target domain prediction image as shown in FIG. 4, wherein the target domain prediction image includes the segmented vehicle 301 and street lamp 302. Figure 3a Figure 3b

[0116] In the embodiments of the present application, the source domain image segmentation network is trained in a supervised learning manner, and at the same time, the segmentation mode of the source domain image segmentation network is transferred to the target domain image segmentation network, so that the target domain image segmentation network is adapted to the segmentation of the target domain image, and the target domain image segmentation network for segmenting the target domain image is trained without the need for fine labeling of each pixel in the target domain sample image, thereby reducing the manpower and cost and improving the image segmentation efficiency and accuracy.

[0117] Optionally, in the step S202, the trained target domain image segmentation network and the trained source domain image segmentation network share one trained encoder, and the trained target domain image segmentation network further includes a trained target domain classifier, and the trained target domain image segmentation network specifically includes a trained encoder 401 and a trained target domain classifier 402 as shown in FIG. 5. Figure 4

[0118] The target image features of the target domain image to be processed are extracted through the trained encoder. Then, the target domain prediction image is obtained based on the target image features through the trained target domain classifier.

[0119] ​​​In a specific implementation, the trained encoder includes a convolutional layer, and the target domain image to be processed is subjected to feature extraction through the convolutional layer to obtain target image features of the target domain image to be processed. The trained target domain classifier includes a deconvolutional layer, and the target image features are subjected to up-sampling through the deconvolutional layer to obtain a target domain predicted image of the same size as the target domain image to be processed and a category of each pixel in the target domain predicted image, the category including whether it is a target object. The target domain predicted image is segmented based on the category of each pixel in the target domain predicted image to obtain a target label of the target object.

[0120] In the embodiments of the present application, the target domain image segmentation network and the source domain image segmentation network share one encoder, and therefore, when the trained target domain image segmentation network is obtained by jointly and iteratively training the target domain image segmentation network to be trained and the source domain image segmentation network to be trained based on the domain transfer learning manner, the feature extraction process of the source domain image segmentation network can be transferred to the target domain image segmentation network, thereby improving the image segmentation performance of the target domain image segmentation network.

[0121] Optionally, in the step S202, the neural network architecture involved in the iterative training process includes the source domain image segmentation network 501 to be trained and the target domain image segmentation network 502 to be trained, as shown in the following figure, wherein the source domain image segmentation network 501 to be trained and the target domain image segmentation network 502 to be trained share one encoder 503 to be trained, the target domain image segmentation network 502 to be trained further includes a target domain classifier 504 to be trained, and the source domain image segmentation network 501 to be trained further includes a source domain classifier 505 to be trained. Figure 5

[0122] In each iteration process, the following operations are performed, as shown in the following figure: Figure 6

[0123] In step S601, source domain sample features of the source domain sample image and target domain sample features of the target domain sample image are respectively extracted through the encoder to be trained.

[0124] Specifically, the source domain sample image includes a source domain sample object with a source label, and the source label of the source domain sample object is obtained by pre-labeling. The target domain sample image does not have a label of a target domain sample object.

[0125] For example, the source domain sample image is a street image synthesized by a 3D game engine, and the source domain sample objects such as pedestrians, roads, trees, and street lamps in the source domain sample image are pre-labeled to obtain source labels of the source domain sample objects. The target domain sample image is a street image collected in a real environment, and the target domain sample image includes target domain sample objects such as pedestrians, roads, trees, and street lamps, and the target domain sample image does not have a label of a target domain sample object. ​​

[0126] In step S602, a source domain prediction sample image is obtained based on the source domain sample features by the source domain classifier to be trained.

[0127] Specifically, the source domain prediction sample image includes a source domain sample object with a source prediction label, the source prediction label of the source domain sample object is obtained by the source domain classifier, and the source domain classifier includes one or more neural network layers. In the iterative training process, the source prediction label of the source domain sample object is continuously close to the source label of the source domain sample object.

[0128] In step S603, a target domain prediction sample image is obtained based on the target domain image features by the target domain classifier to be trained.

[0129] Specifically, the target domain prediction sample image includes a target domain sample object with a target prediction label, the target prediction label of the target domain sample object is obtained by the target domain classifier, and the target domain classifier includes one or more neural network layers.

[0130] In step S604, a target loss value is obtained based on the source domain prediction sample image and the target domain prediction sample image.

[0131] In step S605, the target loss value is used to adjust the parameters of the source domain image segmentation network to be trained and the target domain image segmentation network to be trained respectively.

[0132] Specifically, in each iteration training process, it is judged whether the target loss value meets a preset condition or whether the iteration number reaches a preset threshold. If yes, the training is ended. Otherwise, the target loss value is used to adjust the parameters of the source domain image segmentation network to be trained and the target domain image segmentation network to be trained respectively, and then the next iteration training process is entered.

[0133] In the embodiments of the present application, different classifiers are used for the source domain and the target domain respectively, which tolerates the inconsistent edge distribution probability of the source domain and the target domain, does not force the edge distribution probability of the source domain and the target domain to match, thereby giving the classifier better fitting ability and improving the classification accuracy of the classifier. Secondly, different classifiers are used for different domains, which solves the application of complex problems in the field migration on large-scale neural networks, and has good application prospect on more complex tasks other than classification tasks.

[0134] Optionally, in the step S601, since the labels of the target domain sample objects in the target domain prediction sample image are missing, and the source domain image segmentation network and the target domain image segmentation network share an encoder, the encoder may pay too much attention to the distribution consistency, thereby deviating from the learning task, that is, overfitting occurs. In view of this, in the embodiment of the present application, the encoder to be trained includes part of the network parameters that are not shared for the source domain and the target domain, that is, part of the parameters of the encoder to be trained are modeled for different domains respectively, and the network parameters can be the weights of the network layers in the encoder.

[0135] Specifically, in the training process, the best shared structure can be obtained by an automatic search technique or a manual assisted search, that is, the network layers that share the weights and the network layers that do not share the weights in the encoder for the source domain and the target domain are determined. Since part of the network parameters in the encoder are not shared for the source domain and the target domain, the consistency of the source domain image segmentation network and the target domain image segmentation network in the feature extraction part is ensured, and overfitting of the encoder is avoided.

[0136] Optionally, in the steps S602 and S603, part of the network parameters in the target domain classifier to be trained and the source domain classifier to be trained are shared, and the network parameters can be the weights of the network layers in the classifier. Specifically, in the training process, the best shared structure can be obtained by an automatic search technique or a manual assisted search, that is, the network layers that share the weights and the network layers that do not share the weights in the target domain classifier and the source domain classifier are determined.

[0137] In the embodiment of the present application, the source domain and the target domain use different classifiers respectively, and part of the network parameters in the source domain classifier and the target domain classifier are shared, thereby improving the fitting ability of the source domain classifier and the target domain classifier, and improving the prediction accuracy of the target domain classifier obtained based on the domain transfer.

[0138] Optionally, in the step S604, the target loss value is obtained based on the source domain prediction sample image and the target domain prediction sample image, and the present application provides at least the following implementation manners:

[0139] Implementation manner one: the first probability that the source domain prediction sample image is the source domain image and the second probability that the target domain prediction sample image is the target domain image are determined by the discriminator. Then, the adversarial learning loss value between the source domain image segmentation network to be trained and the target domain image segmentation network to be trained and the discriminator is determined based on the first probability and the second probability. The segmentation loss value of the source domain image segmentation network to be trained is determined based on the source label and the source prediction label. The target loss value is obtained based on the adversarial learning loss value and the segmentation loss value.

[0140] Specifically, the training process involves a neural network architecture as shown in Figure 7 The training process involves a neural network architecture as shown in

[0141] In the iterative training process, the trained target domain classifier 504 outputs a target domain predicted sample image, and the trained source domain classifier 505 outputs a source domain predicted sample image. Then, the source domain predicted sample image and the target domain predicted sample image are input into the discriminator 701, which outputs a first probability that the source domain predicted sample image is a source domain image, and a second probability that the target domain predicted sample image is a target domain image. Based on the first probability and the second probability, an adversarial learning loss value is determined.

[0142] The adversarial learning loss value serves to adjust the parameters of the trained encoder 503, the trained target domain classifier 504, and the trained source domain classifier 505, so that the target domain predicted sample image and the source domain predicted sample image output by the trained target domain classifier 504 and the trained source domain classifier 505 cannot be distinguished by the discriminator. After multiple iterations of training, when the discriminator cannot distinguish whether the target domain predicted sample image comes from the source domain or the target domain, and whether the source domain predicted sample image comes from the source domain or the target domain, it can be considered that the trained network successfully eliminates the image information unique to the source domain and the target domain, and retains the image information common to the source domain and the target domain, i.e., the trained source domain classifier 505 and the trained target domain classifier 504 can adapt to their respective feature distributions. At this time, the adversarial learning process ends.

[0143] Based on the difference between the source label and the source predicted label, a segmentation loss value of the trained source domain image segmentation network is determined, and then a weighted sum of the adversarial learning loss value and the segmentation loss value is taken as a target loss value.

[0144] In the embodiments of the present application, the unique image information of the source domain and the target domain is eliminated by means of adversarial learning, and the image information common to the source domain and the target domain is retained, so that different classifiers automatically adapt to each domain, reducing the difficulty of domain transfer tasks. At the same time, the segmentation accuracy of the source domain image segmentation network is ensured by the segmentation loss value, thereby improving the segmentation performance of the target domain classifier.

[0145] In the embodiment two, the discriminator is used to determine a first probability that the source domain prediction sample image is a source domain image and a second probability that the target domain prediction sample image is a target domain image. Then, based on the first probability and the second probability, an adversarial learning loss value between the source domain image segmentation network to be trained and the target domain image segmentation network to be trained and the discriminator is determined. Based on the source label and the source prediction label, a segmentation loss value of the source domain image segmentation network to be trained is determined. Based on the network parameters of the source domain classifier to be trained and the network parameters of the target domain classifier to be trained, a distance loss value between the source domain classifier to be trained and the target domain classifier to be trained is determined. Based on the adversarial learning loss value, the segmentation loss value and the distance loss value, a target loss value is obtained.

[0146] Specifically, the specific process of obtaining the adversarial learning loss value and the segmentation loss value has been described above, and will not be described here. The process of determining the distance loss value is described below.

[0147] Specifically, the network parameters of the source domain classifier can be the weights of each network layer in the source domain classifier, and the network parameters of the target domain classifier can be the weights of each network layer in the target domain classifier. The distance loss value between the source domain classifier to be trained and the target domain classifier to be trained can be the norm, such as L1 norm, L2 norm, etc., between the network parameters in the two classifiers. It can also be a linear transformation matrix between the network parameters in the two classifiers. The weighted sum of the adversarial learning loss value, the segmentation loss value and the distance loss value is taken as the target loss value.

[0148] In the embodiment of the application, the segmentation loss value is used to ensure the segmentation accuracy of the source domain image segmentation network, the adversarial learning loss value is used to eliminate the image information unique to the source domain and the target domain, and the image information common to the source domain and the target domain is retained, so that the target domain classifier can adaptively find the optimal modeling mode for the target domain while being close to the source domain classifier. Since the source domain classifier and the target domain classifier are aimed at the same semantic elements, i.e., contain similar information, the distance loss value can effectively prevent overfitting of the target domain classifier.

[0149] In the embodiment three, the maximum average difference between the source domain prediction sample image and the target domain prediction sample image is determined. Then, based on the source label and the source prediction label, the segmentation loss value of the source domain image segmentation network to be trained is determined. Based on the adversarial learning loss value and the segmentation loss value, the target loss value is obtained.

[0150] Specifically, a maximum mean discrepancy (MMD) algorithm is used to determine a maximum mean discrepancy between the predicted sample image and the target domain predicted sample image, and the maximum mean discrepancy is used to measure the distance between two different but related distributions. Based on the difference between the source label and the source predicted label, a segmentation loss value of the source domain image segmentation network to be trained is determined, and a weighted sum between the maximum mean discrepancy and the segmentation loss value is taken as a target loss value.

[0151] In the embodiments of the present application, the segmentation loss value is used to ensure the segmentation accuracy of the source domain image segmentation network, and the maximum mean discrepancy is used to match the distributions of the source domain and the target domain, so that the target domain classifier is constantly close to the source domain classifier, thereby improving the performance of the target domain classifier.

[0152] In the embodiments of the present application, the segmentation loss value is used to ensure the segmentation accuracy of the source domain image segmentation network, and the maximum mean discrepancy is used to match the distributions of the source domain and the target domain, so that the target domain classifier is constantly close to the source domain classifier, thereby improving the performance of the target domain classifier.

[0153] In the embodiments of the present application, the segmentation loss value is used to ensure the segmentation accuracy of the source domain image segmentation network, and the maximum mean discrepancy is used to match the distributions of the source domain and the target domain, so that the target domain classifier is constantly close to the source domain classifier, thereby improving the performance of the target domain classifier.

[0154] In the embodiments of the present application, the segmentation loss value is used to ensure the segmentation accuracy of the source domain image segmentation network, and the maximum mean discrepancy is used to match the distributions of the source domain and the target domain, so that the target domain classifier is constantly close to the source domain classifier, thereby improving the performance of the target domain classifier.

[0155] It should be noted that, in the embodiments of the present application, the implementation of obtaining the target loss value based on the source domain prediction sample image and the target domain prediction sample image is limited to the above several kinds, and can also be other combinations of the adversarial learning loss value, the segmentation loss value, the distance loss value and the maximum mean difference, or can be any one of the above several loss values. The present application does not make specific limitations in this regard.

[0156] In order to better explain the embodiments of the present application, the following will take street images in different scenes as an example to introduce the process of the image segmentation method provided by the embodiments of the present application. First, the process of obtaining the trained target domain image segmentation network is introduced, which specifically includes the following steps:

[0157] Training sample preparation: the training sample includes a source domain sample image x S and a target domain sample image x T , wherein the source domain sample image is a street image synthesized by a 3D game engine, and the target domain sample image is a street image collected in a real environment. In the embodiments of the present application, the data set used for training and verifying the model includes three image sets of GTA5, Synthia and Cityscape, all of which are street images, wherein GTA5 and Synthia are image sets synthesized by a 3D game engine, and Cityscape is an image set collected in a real environment.

[0158] Model initialization: the source domain image segmentation network to be trained and the target domain image segmentation network to be trained are initialized in a random initialization manner.

[0159] Model training: the neural network architecture is as shown in Figure 8 , which includes a source domain image segmentation network, a target domain image segmentation network and a discriminator D, the source domain image segmentation network and the target domain image segmentation network share an encoder ε, the source domain image segmentation network further includes a source domain classifier C S , and the target domain image segmentation network further includes a target domain classifier C T . The encoder ε, the source domain classifier C S and the target domain classifier C T may be a series of classic neural network structures such as DeepLab V2, and the discriminator D can be composed of five convolution layers and four activation layers in turn.

[0160] The source domain sample image x S is input into the encoder ε to obtain the source domain sample feature f S of the source domain sample image x S = ε(x S ), and the target domain sample image x T is input into the encoder ε to obtain the target domain sample feature f Ttarget domain sample feature f T . T .

[0161] source domain sample feature f S is input into a source domain classifier C S , to obtain a source domain predicted sample image target domain sample feature f T is input into a target domain classifier C T , to obtain a target domain predicted sample image source domain predicted sample image and target domain predicted sample image are then input into a discriminator D.

[0162] An L1 norm is used to determine a distance loss value between source domain classifier C S and target domain classifier C T , and is specifically shown in the following formula (1):

[0163]

[0164] wherein L dis (C S ,C T ) represents the distance loss value, represents a weight of a network layer in source domain classifier C S , represents a weight of a network layer in target domain classifier C T . L dis (C S ,C T ) is used to supervise source domain classifier C S and target domain classifier C T in the network structure.

[0165] Based on cross-entropy, a segmentation loss value of source domain image segmentation network C S is determined, and is specifically shown in the following formula (2):

[0166]

[0167] wherein L seg (ε,C S ) represents the segmentation loss value, k is a pixel index, and source domain sample image and source domain predicted sample image each include K pixels, represents a probability that the kth pixel is of the cth class, and if the kth pixel is of the cth class, then otherwise represents a probability that the kth pixel is of the cth class predicted by source domain classifier, Lseg (ε,C S ) for the encoder ε, the source domain classifier C S .

[0168] The adversarial learning loss value is obtained by using the following formula (3):

[0169]

[0170] wherein, represents the adversarial learning loss value, represents the first probability that the source domain predicted sample image is a source domain image, represents the second probability that the target domain predicted sample image is a target domain image. The encoder ε, the source domain classifier C S , the target domain classifier C T and the discriminator D in the supervised network structure.

[0171] The process of adversarial learning is as follows:

[0172]

[0173] That is, the loss function generated by the discriminator D and the corresponding gradient are calculated, the gradient ascent method is used for the discriminator D to increase the value of the loss function, and the gradient descent method is used for the main network composed of the encoder ε, the source domain classifier C S and the target domain classifier C T to reduce the value of the loss function, so as to confuse the discriminator D. When the discriminator D cannot successfully judge which domain its input (that is, the output of the main network) comes from, it is considered that the source domain classifier C S and the target domain classifier C T can adapt to the feature distribution of each domain respectively, that is, the image information unique to each domain is eliminated, and the image information common to the domains is retained.

[0174] The target loss value is obtained by using the following formula (4):

[0175]

[0176] wherein, λ dis and λ adv are adjustment coefficients for balancing the size of different loss values. According to experiments, the model performance is best when λ adv = 0.01 and λ dis ∈ [10, 70].

[0177] The training process of the entire network is as follows:

[0178]

[0179] Even if the discriminator D is optimized using the gradient ascent method, the encoder ε, the source domain classifier C and the target domain classifier C are optimized using the gradient descent method S and the target domain classifier C T During the optimization process, a small part of training samples can be extracted from all training sets to calculate the loss each time, and the discriminator uses the Adam optimization method.

[0180] In the embodiments of the application, the performance measurement index for verifying the model is mean IoU, abbreviated as mIoU, and the calculation method is shown in formula (5):

[0181]

[0182] Wherein, Y c is the original label region of the c-th class in the pre-labeled target domain sample image, is the predicted label region of the c-th class in the target domain prediction sample image, represents the intersection of the original label region and the predicted label region, represents the union of the original label region and the predicted label region, and C is the total number of classes. mIoU is the average performance of all classes. When the number of incorrect predictions increases, the union will increase, and mIoU will decrease. The larger the mIoU is, the better the performance of the target domain image segmentation network is.

[0183] Specifically, the effect of migrating GTA5 to Cityscape is shown in Table 1:

[0184] Table 1.

[0185]

[0186] The mIoU in Table 1 is the average value of 19 classes. The source domain image segmentation network refers to training the source domain image segmentation network separately using the source domain sample image, and then using the trained source domain image segmentation network as the trained target domain image segmentation network. AdaptSegNet, SIBAN and CLAN are domain migration methods in the prior art. By comparison, it can be seen that the performance of the target domain image segmentation network obtained by the application is better than that of the prior art.

[0187] The effect of migrating Synthia to Cityscape is shown in Table 2:

[0188] Table 2.

[0189]

[0190] The mIoU in Table 2 is the average value of 13 classes. By comparison, it can be seen that the performance of the target domain image segmentation network obtained by the application is better than that of the prior art.

[0191] After obtaining the trained target domain image segmentation network by using the above method, for the street image collected in the real environment, the target image features of the street image can be extracted through the trained target domain image segmentation network, and the predicted street image can be obtained based on the target image features, wherein the predicted street image includes segmented target objects such as pedestrians, roads, trees, and street lamps.

[0192] In the embodiment of the present application, the source domain image segmentation network is trained in a supervised learning manner, and at the same time, the segmentation mode of the source domain image segmentation network is migrated to the target domain image segmentation network through domain transfer learning, so that the target domain image segmentation network for segmenting the target domain image is trained without the need for fine labeling of each pixel in the target domain sample image, thereby reducing manpower and cost and improving image segmentation efficiency and accuracy.

[0193] The embodiment of the present application also provides a flow of a method for training a target domain image segmentation network, which can be executed by the terminal device or the server shown in Figure 1 The flow specifically includes:

[0194] The target domain image segmentation network to be trained and the source domain image segmentation network to be trained are jointly iteratively trained to obtain a trained target domain image segmentation network, wherein the source domain image segmentation network to be trained and the target domain image segmentation network to be trained share one encoder to be trained, the target domain image segmentation network to be trained further includes a target domain classifier to be trained, and the source domain image segmentation network to be trained further includes a source domain classifier to be trained. In each iteration process, the following operations are performed:

[0195] The source domain sample features of the source domain sample image and the target domain sample features of the target domain sample image are extracted through the encoder to be trained, respectively, wherein the source domain sample image includes source domain sample objects with source labels. The source domain prediction sample image is obtained based on the source domain sample features through the source domain classifier to be trained, wherein the source domain prediction sample image includes source domain sample objects with source prediction labels. The target domain prediction sample image is obtained based on the target domain image features through the target domain classifier to be trained, wherein the target domain prediction sample image includes target domain sample objects with target prediction labels. Then, the target loss value is obtained based on the source domain prediction sample image and the target domain prediction sample image. The source domain image segmentation network to be trained and the target domain image segmentation network to be trained are respectively adjusted in parameters by using the target loss value.

[0196] In the embodiments of the present application, different classifiers are used for the source domain and the target domain, which tolerates the inconsistency of the marginal distribution probability of the source domain and the target domain, does not force the marginal distribution probability of the source domain and the target domain to match, thereby giving the classifier better fitting capability and improving the accuracy of the classifier classification. Secondly, different classifiers are used for different domains, which solves the application of complex problems in domain migration on large-scale neural networks and has good application prospect on more complex tasks other than classification tasks.

[0197] Optionally, for the source domain prediction sample image and the target domain prediction sample image, a target loss value is obtained, and the embodiments of the present application at least provide the following implementation manners:

[0198] Implementation manner one, the first probability that the source domain prediction sample image is the source domain image and the second probability that the target domain prediction sample image is the target domain image are determined through the discriminator. Then, based on the first probability and the second probability, the adversarial learning loss value between the to-be-trained source domain image segmentation network, the to-be-trained target domain image segmentation network and the discriminator is determined. Based on the source label and the source prediction label, the segmentation loss value of the to-be-trained source domain image segmentation network is determined. Based on the adversarial learning loss value and the segmentation loss value, the target loss value is obtained.

[0199] Implementation manner two, the first probability that the source domain prediction sample image is the source domain image and the second probability that the target domain prediction sample image is the target domain image are determined through the discriminator. Then, based on the first probability and the second probability, the adversarial learning loss value between the to-be-trained source domain image segmentation network, the to-be-trained target domain image segmentation network and the discriminator is determined. Based on the source label and the source prediction label, the segmentation loss value of the to-be-trained source domain image segmentation network is determined. Based on the network parameters of the to-be-trained source domain classifier and the network parameters of the to-be-trained target domain classifier, the distance loss value between the to-be-trained source domain classifier and the to-be-trained target domain classifier is determined. Based on the adversarial learning loss value, the segmentation loss value and the distance loss value, the target loss value is obtained.

[0200] In the embodiments of the present application, the segmentation accuracy of the source domain image segmentation network is ensured through the segmentation loss value, the image information unique to the source domain and the target domain is eliminated through the adversarial learning loss value, and the image information common to the source domain and the target domain is retained, so that the target domain classifier can adaptively find the optimal modeling mode for the target domain while being close to the source domain classifier. Since the semantic elements targeted by the source domain classifier and the target domain classifier are the same, that is, contain similar information, the distance loss value effectively prevents overfitting of the target domain classifier.

[0201] Based on the same technical concept, the embodiments of the present application provide a structural schematic diagram of an image segmentation device, as shown in Figure 9 The device 900 includes:

[0202] The acquisition module 901 is configured to acquire a target domain to-be-processed image including a target object.

[0203] The image segmentation module 902 is configured to extract a target image feature of the target domain to-be-processed image by using the trained target domain image segmentation network, and obtain a target domain predicted image based on the target image feature, wherein the target domain predicted image includes the target object with a target label.

[0204] The trained target domain image segmentation network is obtained by jointly and iteratively training a to-be-trained target domain image segmentation network and a to-be-trained source domain image segmentation network, and the training samples include source domain sample images and target domain sample images, wherein the source domain sample images include source domain sample objects with source labels, the to-be-trained target domain image segmentation network and the to-be-trained source domain image segmentation network share a to-be-trained encoder, the to-be-trained target domain image segmentation network further includes a to-be-trained target domain classifier, and the to-be-trained source domain image segmentation network further includes a to-be-trained source domain classifier.

[0205] Optionally, the trained target domain image segmentation network and the trained source domain image segmentation network share a trained encoder, and the trained target domain image segmentation network further includes a trained target domain classifier.

[0206] The image segmentation module 902 is specifically configured to:

[0207] extract the target image feature of the target domain to-be-processed image by using the trained encoder;

[0208] obtain the target domain predicted image based on the target image feature by using the trained target domain classifier.

[0209] Optionally, the image segmentation apparatus further includes a model training module 903.

[0210] The model training module 903 is specifically configured to:

[0211] In each iteration process, the following operations are performed:

[0212] extract source domain sample features of the source domain sample images and target domain sample features of the target domain sample images by using the to-be-trained encoder;

[0213] obtain source domain predicted sample images based on the source domain sample features by using the to-be-trained source domain classifier, wherein the source domain predicted sample images include source domain sample objects with source predicted labels;

[0214] obtaining a target domain prediction sample image based on the target domain image feature through the target domain classifier to be trained, wherein the target domain prediction sample image comprises a target domain sample object with a target prediction label;

[0215] obtaining a target loss value based on the source domain prediction sample image and the target domain prediction sample image;

[0216] adopting the target loss value to respectively adjust parameters of the source domain image segmentation network to be trained and the target domain image segmentation network to be trained.

[0217] Optionally, the model training module 903 is specifically configured to:

[0218] determining, through the discriminator, a first probability that the source domain prediction sample image is the source domain image and a second probability that the target domain prediction sample image is the target domain image;

[0219] determining, based on the first probability and the second probability, an adversarial learning loss value between the source domain image segmentation network to be trained and the target domain image segmentation network to be trained and the discriminator;

[0220] determining, based on the source label and the source prediction label, a segmentation loss value of the source domain image segmentation network to be trained;

[0221] obtaining the target loss value based on the adversarial learning loss value and the segmentation loss value.

[0222] Optionally, the model training module 903 is specifically configured to:

[0223] determining, based on the network parameters of the source domain classifier to be trained and the network parameters of the target domain classifier to be trained, a distance loss value between the source domain classifier to be trained and the target domain classifier to be trained;

[0224] obtaining the target loss value based on the adversarial learning loss value, the segmentation loss value and the distance loss value.

[0225] Optionally, the model training module 903 is specifically configured to:

[0226] determining a maximum mean discrepancy between the source domain prediction sample image and the target domain prediction sample image;

[0227] determining, based on the source label and the source prediction label, a segmentation loss value of the source domain image segmentation network to be trained;

[0228] obtaining the target loss value based on the maximum mean discrepancy and the segmentation loss value.

[0229] Optionally, the model training module 903 is specifically configured to:

[0230] determine a distance loss value between the source domain classifier to be trained and the target domain classifier to be trained based on network parameters of the source domain classifier to be trained and network parameters of the target domain classifier to be trained;

[0231] obtain a target loss value based on the maximum mean difference, the segmentation loss value, and the distance loss value.

[0232] Optionally, the target domain classifier to be trained and the source domain classifier to be trained include partially shared network parameters.

[0233] In the embodiment of the application, when the source domain image segmentation network is trained in a supervised learning manner, the segmentation mode of the source domain image segmentation network is migrated to the target domain image segmentation network through domain transfer learning, so that the target domain image segmentation network adapts to the segmentation of the target domain image, and the target domain image segmentation network for segmenting the target domain image is trained without the need for fine labeling of each pixel in the target domain sample image, thereby reducing manpower and cost and improving image segmentation efficiency and accuracy.

[0234] Based on the same technical concept, the embodiment of the application provides a structural diagram of a device for training a target domain image segmentation network, as shown in Figure 10 The device 1000 includes:

[0235] The model training module 1001 is configured to jointly and iteratively train the target domain image segmentation network to be trained and the source domain image segmentation network to be trained to obtain a trained target domain image segmentation network, wherein the target domain image segmentation network to be trained and the source domain image segmentation network to be trained share one encoder to be trained, the target domain image segmentation network to be trained further includes a target domain classifier to be trained, and the source domain image segmentation network to be trained further includes a source domain classifier to be trained.

[0236] The model training module includes a feature extraction module 1002, a first prediction module 1003, a second prediction module 1004, and a parameter adjustment module 1005.

[0237] The feature extraction module 1002 is configured to extract source domain sample features of a source domain sample image and target domain sample features of a target domain sample image through the encoder to be trained, wherein the source domain sample image includes a source domain sample object with a source label.

[0238] The first prediction module 1003 is configured to obtain a source domain predicted sample image based on the source domain sample features through the source domain classifier to be trained, wherein the source domain predicted sample image includes a source domain sample object with a source predicted label.

[0239] The second prediction module 1004 is configured to obtain a target domain prediction sample image based on the target domain image feature by using a target domain classifier to be trained, wherein the target domain prediction sample image includes a target domain sample object with a target prediction label.

[0240] The parameter adjustment module 1005 is configured to obtain a target loss value based on the source domain prediction sample image and the target domain prediction sample image, and adjust parameters of the source domain image segmentation network to be trained and the target domain image segmentation network to be trained respectively by using the target loss value.

[0241] Optionally, the parameter adjustment module 1005 is specifically configured to:

[0242] determine, by using the discriminator, a first probability that the source domain prediction sample image is the source domain image and a second probability that the target domain prediction sample image is the target domain image;

[0243] determine, based on the first probability and the second probability, an adversarial learning loss value between the source domain image segmentation network to be trained and the target domain image segmentation network to be trained and the discriminator;

[0244] determine, based on the source label and the source prediction label, a segmentation loss value of the source domain image segmentation network to be trained;

[0245] obtain the target loss value based on the adversarial learning loss value and the segmentation loss value.

[0246] Optionally, the parameter adjustment module 1005 is specifically configured to:

[0247] determine, based on the network parameters of the source domain classifier to be trained and the network parameters of the target domain classifier to be trained, a distance loss value between the source domain classifier to be trained and the target domain classifier to be trained;

[0248] obtain the target loss value based on the adversarial learning loss value, the segmentation loss value and the distance loss value.

[0249] In the embodiments of the present application, different classifiers are used for the source domain and the target domain, which tolerates the inconsistency of the edge distribution probability of the source domain and the target domain, does not force the edge distribution probability of the source domain and the target domain to match, thereby giving the classifier better fitting capability and improving the classification accuracy of the classifier. Secondly, different classifiers are used for different domains, which solves the application of complex problems in the field migration on large-scale neural networks, and has good application prospect on more complex tasks other than classification tasks.

[0250] Based on the same technical concept, the embodiments of the present application provide a computer device, such as Figure 11As shown, the computer device 1100 includes at least one processor 1101, and a memory 1102 connected with the at least one processor 1101. In the embodiments of the present application, the specific connection medium between the processor 1101 and the memory 1102 is not limited, Figure 11 For example, the processor 1101 and the memory 1102 are connected through a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0251] In the embodiments of the present application, the memory 1102 stores instructions executable by the at least one processor 1101. The at least one processor 1101 can execute the steps included in the image segmentation method or the method of training the target domain image segmentation network by executing the instructions stored in the memory 1102.

[0252] The processor 1101 is the control center of the computer device, and can connect various parts of the computer device through various interfaces and lines, and realize image segmentation or training of the target domain image segmentation network by running or executing instructions stored in the memory 1102 and calling data stored in the memory 1102. Optionally, the processor 1101 can include one or more processing units. The processor 1101 can integrate an application processor and a modem processor. The application processor mainly processes the operating system, user interface, and application program, etc. The modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 1101. In some embodiments, the processor 1101 and the memory 1102 can be implemented on the same chip, and in some embodiments, they can also be implemented on independent chips respectively.

[0253] The processor 1101 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0254] The memory 1102, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 1102 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. The memory 1102 is any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory 1102 in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used for storing program instructions and / or data.

[0255] Based on the same inventive concept, the embodiments of the present application provide a computer readable storage medium storing a computer program executable by a computer device, which, when the program is executed on the computer device, causes the computer device to perform the steps of the image segmentation method or the method of training the target domain image segmentation network.

[0256] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0257] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a machine that implements the functions described in the flowcharts and / or block diagrams.Figure 1 one or more processes and / or functions specified in the block or blocks. Figure 1

[0258] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 one or more processes and / or functions specified in the block or blocks. Figure 1 one or more processes and / or functions specified in the block or blocks.

[0259] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the Figure 1 one or more processes and / or functions specified in the block or blocks. Figure 1 one or more processes and / or functions specified in the block or blocks.

[0260] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to encompass within their scope all such variations and modifications as are included within the scope of the invention.

[0261] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.​

Claims

1. An image segmentation method characterized by, The method comprises: obtaining a target domain image to be processed including a target object; extracting a target image feature of the target domain image to be processed through a trained target domain image segmentation network, and obtaining a target domain prediction image based on the target image feature, the target domain prediction image including the target object with a target label; wherein the trained target domain image segmentation network is obtained by jointly and iteratively training a to-be-trained target domain image segmentation network and a to-be-trained source domain image segmentation network, and training samples used in the training include source domain sample images and target domain sample images, wherein the source domain sample images include source domain sample objects with source labels, and the to-be-trained target domain image segmentation network and the to-be-trained source domain image segmentation network share one to-be-trained encoder; in the training process, for the source domain and the target domain, network layers with shared weights and network layers without shared weights in the encoder are obtained through automatic search technology; the to-be-trained target domain image segmentation network further includes a to-be-trained target domain classifier, and the to-be-trained source domain image segmentation network further includes a to-be-trained source domain classifier; the target domain classifier and the source domain classifier are different classifiers; in the training process, network layers with shared weights and network layers without shared weights in the target domain classifier and the source domain classifier are obtained through automatic search technology; in the training process, a maximum mean difference between source domain prediction sample images and target domain prediction sample images is determined, and the distribution of the source domain and the target domain is matched through the maximum mean difference, so that the to-be-trained target domain classifier is close to the to-be-trained source domain classifier; a distance loss value is used to prevent overfitting of the to-be-trained target domain classifier, and the distance loss value is a norm between network parameters of the to-be-trained source domain classifier and network parameters of the to-be-trained target domain classifier.

2. The method of claim 1, wherein, the trained target domain image segmentation network shares one trained encoder with a trained source domain image segmentation network, and the trained target domain image segmentation network further includes a trained target domain classifier; the trained target domain image segmentation network extracts the target image feature of the target domain image to be processed through the trained encoder, and obtains the target domain prediction image based on the target image feature; the trained target domain image segmentation network is obtained by jointly and iteratively training a to-be-trained target domain image segmentation network and a to-be-trained source domain image segmentation network, and comprises: in each iteration process, the following operations are performed:

3. The method of claim 1, wherein, extracting source domain sample features of the source domain sample images and target domain sample features of the target domain sample images through the to-be-trained encoder respectively; obtaining source domain prediction sample images based on the source domain sample features through the to-be-trained source domain classifier, wherein the source domain prediction sample images include source domain sample objects with source prediction labels; ​ ​ obtaining a target domain prediction sample image based on the target domain image feature through the target domain classifier to be trained, wherein the target domain prediction sample image comprises a target domain sample object with a target prediction label; obtaining a target loss value based on the source domain prediction sample image and the target domain prediction sample image; adjusting parameters of the source domain image segmentation network to be trained and the target domain image segmentation network to be trained respectively by using the target loss value.

4. The method of claim 3, wherein, The method further comprises: determining, through the discriminator, a first probability that the source domain prediction sample image is a source domain image and a second probability that the target domain prediction sample image is a target domain image; determining, based on the first probability and the second probability, an adversarial learning loss value between the source domain image segmentation network to be trained and the target domain image segmentation network to be trained and the discriminator; determining, based on the source label and the source prediction label, a segmentation loss value of the source domain image segmentation network to be trained; obtaining the target loss value based on the adversarial learning loss value and the segmentation loss value.

5. The method of claim 4, wherein, The method further comprises: determining, based on network parameters of the source domain classifier to be trained and network parameters of the target domain classifier to be trained, a distance loss value between the source domain classifier to be trained and the target domain classifier to be trained; obtaining the target loss value based on the adversarial learning loss value, the segmentation loss value, and the distance loss value.

6. The method of claim 3, wherein, The method further comprises: determining a maximum mean discrepancy between the source domain prediction sample image and the target domain prediction sample image; determining, based on the source label and the source prediction label, a segmentation loss value of the source domain image segmentation network to be trained; obtaining the target loss value based on the maximum mean discrepancy and the segmentation loss value.

7. The method of claim 4, wherein, The method further comprises: determining, based on network parameters of the source domain classifier to be trained and network parameters of the target domain classifier to be trained, a distance loss value between the source domain classifier to be trained and the target domain classifier to be trained; obtaining the target loss value based on the maximum mean discrepancy, the segmentation loss value, and the distance loss value.

8. A method for training a target domain image segmentation network, characterized in that, The method further comprises: The target domain image segmentation network to be trained and the source domain image segmentation network to be trained are jointly iteratively trained to obtain a trained target domain image segmentation network, wherein the source domain image segmentation network to be trained and the target domain image segmentation network to be trained share an encoder to be trained; in the training process, for the source domain and the target domain, network layers with shared weights and network layers without shared weights in the encoder are obtained through an automatic search technology; the target domain image segmentation network to be trained further comprises a target domain classifier to be trained, and the source domain image segmentation network to be trained further comprises a source domain classifier to be trained; the target domain classifier and the source domain classifier are different classifiers; in the training process, network layers with shared weights and network layers without shared weights in the target domain classifier and the source domain classifier are obtained through an automatic search technology; In the training process, a maximum mean difference between the source domain prediction sample image and the target domain prediction sample image is determined, and the distribution of the source domain and the target domain is matched through the maximum mean difference, so that the target domain classifier to be trained is close to the source domain classifier to be trained; a distance loss value is used to prevent overfitting of the target domain classifier to be trained, and the distance loss value is a norm between network parameters of the source domain classifier to be trained and network parameters of the target domain classifier to be trained; In each iteration process, the following operations are performed: source domain sample features of source domain sample images and target domain sample features of target domain sample images are extracted through the encoder to be trained, wherein the source domain sample images include source domain sample objects with source labels; a source domain prediction sample image is obtained based on the source domain sample features through the source domain classifier to be trained, wherein the source domain prediction sample image includes source domain sample objects with source prediction labels; a target domain prediction sample image is obtained based on the target domain image features through the target domain classifier to be trained, wherein the target domain prediction sample image includes target domain sample objects with target prediction labels; a target loss value is obtained based on the source domain prediction sample image and the target domain prediction sample image; the target loss value is used to adjust parameters of the source domain image segmentation network to be trained and the target domain image segmentation network to be trained respectively.

9. The method of claim 8, wherein, The target loss value is obtained based on the source domain prediction sample image and the target domain prediction sample image, comprising: a first probability that the source domain prediction sample image is a source domain image and a second probability that the target domain prediction sample image is a target domain image are determined through a discriminator; an adversarial learning loss value between the source domain image segmentation network to be trained and the target domain image segmentation network to be trained and the discriminator is determined based on the first probability and the second probability; a segmentation loss value of the source domain image segmentation network to be trained is determined based on the source labels and the source prediction labels; the target loss value is obtained based on the adversarial learning loss value and the segmentation loss value.

10. The method of claim 9, wherein, The target loss value is obtained based on the adversarial learning loss value and the segmentation loss value, and the target loss value comprises: a distance loss value between the source domain classifier to be trained and the target domain classifier to be trained is determined based on network parameters of the source domain classifier to be trained and network parameters of the target domain classifier to be trained; the target loss value is obtained based on the adversarial learning loss value, the segmentation loss value and the distance loss value.

11. An image segmentation apparatus characterized by comprising: The method comprises: an acquisition module configured to acquire a target domain image to be processed including a target object; an image segmentation module configured to extract a target image feature of the target domain image to be processed by using a trained target domain image segmentation network, and obtain a target domain predicted image including the target object with a target label based on the target image feature; wherein the trained target domain image segmentation network is obtained by jointly and iteratively training a target domain image segmentation network to be trained and a source domain image segmentation network to be trained, and training samples used in the training include source domain sample images and target domain sample images, wherein the source domain sample images include source domain sample objects with source labels, and the target domain image segmentation network to be trained and the source domain image segmentation network to be trained share one encoder to be trained; in the training process, the network layers with shared weights and the network layers without shared weights in the encoder are obtained by using an automatic search technology for the source domain and the target domain; the target domain image segmentation network to be trained further comprises a target domain classifier to be trained, and the source domain image segmentation network to be trained further comprises a source domain classifier to be trained; the target domain classifier and the source domain classifier are different classifiers; in the training process, the network layers with shared weights and the network layers without shared weights in the target domain classifier and the source domain classifier are obtained by using the automatic search technology; in the training process, a maximum mean difference between a source domain predicted sample image and a target domain predicted sample image is determined, and the distribution of the source domain and the target domain is matched by using the maximum mean difference, so that the target domain classifier to be trained is close to the source domain classifier to be trained; the distance loss value between the network parameters of the source domain classifier to be trained and the network parameters of the target domain classifier to be trained is used to prevent overfitting of the target domain classifier to be trained.

12. An apparatus for training a target domain image segmentation network, the apparatus comprising: The method comprises: an acquisition module configured to acquire a target domain image to be processed including a target object; an image segmentation module configured to extract a target image feature of the target domain image to be processed by using a trained target domain image segmentation network, and obtain a target domain predicted image including the target object with a target label based on the target image feature; wherein the trained target domain image segmentation network is obtained by jointly and iteratively training a target domain image segmentation network to be trained and a source domain image segmentation network to be trained, and training samples used in the training include source domain sample images and target domain sample images, wherein the source domain sample images include source domain sample objects with source labels, and the target domain image segmentation network to be trained and the source domain image segmentation network to be trained share one encoder to be trained; in the training process, the network layers with shared weights and the network layers without shared weights in the encoder are obtained by using an automatic search technology for the source domain and the target domain; the target domain image segmentation network to be trained further comprises a target domain classifier to be trained, and the source domain image segmentation network to be trained further comprises a source domain classifier to be trained; the target domain classifier and the source domain classifier are different classifiers; in the training process, the network layers with shared weights and the network layers without shared weights in the target domain classifier and the source domain classifier are obtained by using the automatic search technology; in the training process, a maximum mean difference between a source domain predicted sample image and a target domain predicted sample image is determined, and the distribution of the source domain and the target domain is matched by using the maximum mean difference, so that the target domain classifier to be trained is close to the source domain classifier to be trained; the distance loss value between the network parameters of the source domain classifier to be trained and the network parameters of the target domain classifier to be trained is used to prevent overfitting of the target domain classifier to be trained. The method comprises: The model training module is configured to jointly and iteratively train a target domain image segmentation network to be trained and a source domain image segmentation network to be trained, and obtain a trained target domain image segmentation network, wherein the target domain image segmentation network to be trained and the source domain image segmentation network to be trained share an encoder to be trained; in the training process, network layers with shared weights and network layers without shared weights in the encoder are obtained through an automatic search technique for the source domain and the target domain; the target domain image segmentation network to be trained further comprises a target domain classifier to be trained, and the source domain image segmentation network to be trained further comprises a source domain classifier to be trained; the target domain classifier and the source domain classifier are different classifiers; in the training process, network layers with shared weights and network layers without shared weights in the target domain classifier and the source domain classifier are obtained through the automatic search technique. In the training process, a maximum mean difference between a source domain prediction sample image and a target domain prediction sample image is determined, and distributions of the source domain and the target domain are matched through the maximum mean difference, so that the target domain classifier to be trained is close to the source domain classifier to be trained; distance loss values are used to prevent overfitting of the target domain classifier to be trained, and the distance loss values are norms between network parameters of the source domain classifier to be trained and network parameters of the target domain classifier to be trained. The model training module comprises a feature extraction module, a first prediction module, a second prediction module and a parameter adjustment module. The feature extraction module is configured to extract source domain sample features of a source domain sample image and target domain sample features of a target domain sample image through the encoder to be trained, wherein the source domain sample image comprises source domain sample objects with source labels. The first prediction module is configured to obtain a source domain prediction sample image through the source domain classifier to be trained based on the source domain sample features, wherein the source domain prediction sample image comprises source domain sample objects with source prediction labels. The second prediction module is configured to obtain a target domain prediction sample image through the target domain classifier to be trained based on the target domain image features, wherein the target domain prediction sample image comprises target domain sample objects with target prediction labels. The parameter adjustment module is configured to obtain a target loss value based on the source domain prediction sample image and the target domain prediction sample image, and adjust parameters of the source domain image segmentation network to be trained and the target domain image segmentation network to be trained respectively based on the target loss value.

13. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the method of any one of claims 1-7, or the steps of the method of any one of claims 8-10.

14. A computer-readable storage medium, characterized in that, The computer program stored in the computer device can be executed by the computer device, and when the program runs on the computer device, the computer device executes the steps of the method of any one of claims 1-7, or the steps of the method of any one of claims 8-10.

Images