Segmentation network training method and apparatus, device, computer program and medium

Abstract

The application provides a segmentation network training method and device, electronic equipment and a server. The method comprises the following steps: obtaining a trained basic segmentation network through a source domain data set; extracting a first prototype vector corresponding to the source domain data set through the basic segmentation network; extracting a target feature vector corresponding to a target domain data set through the basic segmentation network; performing comparison migration processing according to a similarity function of the first prototype vector and the target feature vector to determine a migration loss function of the basic segmentation network; adjusting the basic segmentation network through the migration loss function to obtain a first segmentation network; updating the first prototype vector by using the target feature vector to obtain a second prototype vector; and adjusting the first segmentation network through the second prototype vector to obtain a second segmentation network. Thus, the generalization ability of the segmentation network is stronger while the training accuracy is taken into account.

Description

Technical Field

[0001] The present invention relates to image information processing technology, and in particular to segmentation network training methods, devices, electronic devices, computer program products, and storage media, so that the fields in which this solution can be applied include but are not limited to autonomous driving, vehicle networking, smart transportation, and other fields. Background Art

[0002] Semantic segmentation assigns a semantic label to each pixel in an image. Current deep learning-based semantic segmentation methods require large-scale, detailed manual annotation, which is extremely time- and labor-intensive. Therefore, models are trained using existing labeled source domain data to perform effective inference on an unlabeled target domain, thereby achieving unsupervised semantic segmentation in the target domain. This approach is called unsupervised domain adaptation semantic segmentation. The core of unsupervised domain adaptation semantic segmentation is to narrow the inter-domain gap between the source and target domains, ensuring that the model trained on the source domain generalizes as strongly as possible to the target domain. Currently, mainstream domain adaptation strategies can be categorized as distance optimization-based methods and generative adversarial network (GAN)-based methods. The former achieves domain adaptation by constructing a distance evaluation strategy to minimize the distance between the source and target domain feature distributions; the latter uses an adversarial generative strategy to enhance the model's generalization capabilities to inter-domain differences. However, in practice, test scenario data often mismatches the model training data, necessitating manual re-annotation, increasing model training costs. Summary of the Invention

[0003] In view of this, the embodiments of the present invention provide a segmentation network training method, apparatus, electronic device, computer program product, and storage medium. The technical solution of the embodiments of the present invention is implemented as follows:

[0004] An embodiment of the present invention provides a segmentation network training method, comprising:

[0005] Acquire a source domain data set and a target domain data set, wherein the source domain data set includes source domain images and carries labels, and the target domain data set includes target domain images and does not carry labels;

[0006] Training a basic segmentation network using the source domain data set to obtain a trained basic segmentation network;

[0007] Extracting a first prototype vector corresponding to the source domain data set through the basic segmentation network;

[0008] Extracting a target feature vector corresponding to the target domain data set through the basic segmentation network;

[0009] Performing comparative migration processing based on a similarity function between the first prototype vector and the target feature vector to determine a migration loss function of the basic segmentation network;

[0010] Adjusting the basic segmentation network by using the migration loss function to obtain a first segmentation network;

[0011] Updating the first prototype vector using the target feature vector to obtain a second prototype vector;

[0012] The first segmentation network is adjusted using the second prototype vector to obtain a second segmentation network, wherein the second segmentation network is used to segment the image to be processed to obtain an image segmentation result.

[0013] An embodiment of the present invention further provides a segmentation network training device, comprising:

[0014] An information transmission module, configured to obtain a source domain data set and a target domain data set, wherein the source domain data set carries a label and the target domain data set does not carry a label;

[0015] A training module, configured to train a basic segmentation network using the source domain data set to obtain a trained basic segmentation network;

[0016] The training module is configured to extract a first prototype vector corresponding to the source domain data set through the basic segmentation network;

[0017] The training module is used to extract the target feature vector corresponding to the target domain data set through the basic segmentation network;

[0018] The training module is configured to perform comparative migration processing based on a similarity function between the first prototype vector and the target feature vector to determine a migration loss function of the basic segmentation network;

[0019] The training module is configured to adjust the basic segmentation network using the transfer loss function to obtain a first segmentation network;

[0020] The training module is configured to update the first prototype vector using the target feature vector to obtain a second prototype vector;

[0021] The training module is used to adjust the first segmentation network using the second prototype vector to obtain a second segmentation network.

[0022] In the above scheme,

[0023] The training module is used to determine the number of images, the number of image categories, and image size data in the source domain data set;

[0024] The training module is used to perform one-hot encoding processing on the image category quantity and image size data to form a source domain image feature vector;

[0025] The training module is configured to determine a cross entropy loss function corresponding to the basic segmentation network based on the number of images, the number of image categories, the image size data, and the source domain image feature vector;

[0026] The training module is used to train the basic segmentation network based on the cross entropy loss function using the source domain data set to obtain a trained basic segmentation network.

[0027] In the above scheme,

[0028] The training module is used to extract the source domain feature vector corresponding to the source domain data set through the basic segmentation network.

[0029] The training module is used to calculate a first prototype vector corresponding to the source domain data set based on the feature label of the source domain data set, the number of images, the number of image categories, the image size data and the source domain feature vector.

[0030] In the above scheme,

[0031] The training module is used to extract the source domain feature vector corresponding to the source domain data set through the basic segmentation network.

[0032] The training module is used to calculate a first prototype vector corresponding to the source domain data set based on the feature label of the source domain data set, the number of images, the number of image categories, the image size data and the source domain feature vector.

[0033] In the above scheme,

[0034] The training module is used to calculate a similarity function between the first prototype vector and the target feature vector;

[0035] The training module is used to obtain pseudo labels of the target domain data set;

[0036] The training module is configured to determine a transfer learning loss function based on the similarity function, the number of image categories, the image size data, and the pseudo labels;

[0037] The training module is configured to determine an inter-class constraint loss function based on the similarity function, the number of image categories, the image size data, and the source domain image feature vector;

[0038] The training module is used to determine the transfer loss function of the basic segmentation network based on the transfer learning loss function and the inter-class constraint loss function.

[0039] In the above scheme,

[0040] The training module is used to determine the loss function of the label output space of the source domain data set;

[0041] The training module is used to determine the transfer loss function of the basic segmentation network based on the loss function of the label output space, the transfer learning loss function and the inter-class constraint loss function.

[0042] In the above scheme,

[0043] The training module is used when the target domain data set is a medical image set.

[0044] The training module is used to substitute the medical image set into the migration loss function of the basic segmentation network;

[0045] The training module is used to determine the update parameters corresponding to the encoder and decoder of the basic segmentation network when the migration loss function of the basic segmentation network meets the corresponding convergence condition;

[0046] The training module is used to adjust the basic segmentation network through the updated parameters to obtain a first segmentation network.

[0047] In the above scheme,

[0048] The training module is used to determine the hyperparameters that control the update rate;

[0049] The training module is used to obtain the number of pixels of any type in the target domain data set at different update stages;

[0050] The training module is used to update the first prototype vector based on the number of pixels and the hyperparameter and based on the target feature vector to obtain the second prototype vector.

[0051] The above scheme also includes:

[0052] An information processing module is configured to obtain the image to be processed, wherein the image to be processed includes at least one of the following:

[0053] Medical images, road information monitoring images, and media information display images;

[0054] The information processing module is used to segment the image to be processed through the second segmentation network to obtain the image segmentation result.

[0055] An embodiment of the present invention further provides an electronic device, comprising:

[0056] a memory for storing executable instructions;

[0057] The processor is configured to implement the preceding segmentation network training method when running the executable instructions stored in the memory.

[0058] An embodiment of the present invention further provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the aforementioned segmentation network training method.

[0059] The embodiments of the present invention have the following beneficial effects:

[0060] The present invention obtains a source domain data set and a target domain data set, wherein the source domain data set carries labels and the target domain data set does not carry labels; a basic segmentation network is trained through the source domain data set to obtain a trained basic segmentation network; a first prototype vector corresponding to the source domain data set is extracted through the basic segmentation network; a target feature vector corresponding to the target domain data set is extracted through the basic segmentation network; a comparative migration process is performed based on a similarity function between the first prototype vector and the target feature vector to determine a migration loss function of the basic segmentation network; the basic segmentation network is adjusted through the migration loss function to obtain a first segmentation network; the first prototype vector is updated using the target feature vector to obtain a second prototype vector; the first segmentation network is adjusted through the second prototype vector to obtain a second segmentation network.

[0061] Therefore, while taking into account the training accuracy, the generalization ability and data processing ability of the segmentation network are made stronger, adapting to different data processing environments, reducing the cost of data labeling, enhancing the robustness of the segmentation network classification, and reducing the problem of easy overfitting of the segmentation network. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Schematic diagram of the application environment of the segmentation network training method in an embodiment of the present invention;

[0063] Figure 2 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0064] Figure 3Schematic diagram of the effect of the transfer learning algorithm in an embodiment of the present invention;

[0065] Figure 4 An optional flowchart of the segmentation network training method provided in this application;

[0066] Figure 5 An optional flowchart of the segmentation network training method provided in this application;

[0067] Figure 6 An optional flowchart of the segmentation network training method provided in this application;

[0068] Figure 7 An optional flowchart of the segmentation network training method provided in this application;

[0069] Figure 8 An optional flowchart of the segmentation network training method provided in an embodiment of the present invention;

[0070] Figure 9 Schematic diagram of the effect of network segmentation in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0072] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0073] Before further explaining the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.

[0074] 1) Based on: This is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations can be executed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are executed.

[0075] 2) Client: The carrier that implements specific functions in the terminal. For example, the mobile client (APP) is the carrier of specific functions in the mobile terminal, such as executing the function of online live broadcast (video streaming) or the function of playing online videos.

[0076] 3) Convolutional Neural Networks (CNNs) are a type of feed-forward neural network with a deep structure that incorporates convolutional computations. They are a representative algorithm for deep learning. Convolutional neural networks have representation learning capabilities and can perform shift-invariant classification of input information based on their hierarchical structure.

[0077] 4) Model training: Perform multi-classification learning on image datasets. This model can be built using deep learning frameworks such as TensorFlow and Torch, using multiple layers of neural network layers such as CNN to form a multi-segmentation network. The model input is a three-channel or original channel matrix generated by reading the image using tools such as OpenCV. The model output is multi-class probabilities, and the image segmentation results are finally output using algorithms such as softmax. During training, the model approaches the correct trend using objective functions such as cross entropy.

[0078] 5) Neural Network (NN): Artificial Neural Network (ANN), also known as neural network or neural network-like network, is a mathematical model or computational model that imitates the structure and function of biological neural networks (the central nervous system of animals, especially the brain) in the fields of machine learning and cognitive science. It is used to estimate or approximate functions.

[0079] 6) Contrastive loss: This function learns a mapping relationship that brings distant points of the same class in high-dimensional space closer together in low-dimensional space, while close points of different classes become further apart in low-dimensional space. As a result, points of the same class are clustered in low-dimensional space, while the means of different classes are separated. This is similar to Fisher dimensionality reduction, but it lacks out-of-sample extension and cannot be applied to new samples.

[0080] The following describes the segmentation network training method provided by the embodiment of the present invention, wherein: Figure 1 This is a schematic diagram of the application environment of the segmentation network training method in an embodiment of the present invention, see Figure 1The terminals (including terminal 10-1 and terminal 10-2) are equipped with client applications that feature facial liveness detection. A trained segmentation network is deployed on the server to verify the face of the terminal. For example, in the financial payment field, users can use their smartphones to perform operations requiring identity verification, such as transferring money, making payments, or modifying account information. This can be accomplished by detecting the user's facial liveness. During this process, the terminal device uploads a facial image or video to the server, or the server directly retrieves the facial image or video from a database. The server then uses the trained segmentation network to verify the received facial image or video to obtain a detection result. The server can either feedback the detection result to the terminal device or store it locally for other business applications or processing. The terminal connects to server 200 via network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both, using a wireless link for data transmission.

[0081] As an example, the server 200 is used to deploy a segmentation network training device to implement the segmentation network training method provided by the present invention. Of course, before the segmentation network is used to process different face images to generate corresponding classification results, the segmentation network needs to be trained, specifically including: obtaining a source domain data set and a target domain data set, wherein the source domain data set carries labels and the target domain data set does not carry labels; training the basic segmentation network through the source domain data set to obtain a trained basic segmentation network; extracting the first prototype vector corresponding to the source domain data set through the basic segmentation network; extracting the target feature vector corresponding to the target domain data set through the basic segmentation network; performing comparative migration processing based on the similarity function of the first prototype vector and the target feature vector to determine the migration loss function of the basic segmentation network; adjusting the basic segmentation network through the migration loss function to obtain a first segmentation network; updating the first prototype vector using the target feature vector to obtain a second prototype vector; adjusting the first segmentation network through the second prototype vector to obtain a second segmentation network.

[0082] Of course, the segmentation network training device provided by the present invention can be applied to virtual or physical resources for financial activities, or in physical financial resource payment environments (including but not limited to face detection environments in various types of physical financial resource payments), or in social software environments for information exchange. In various types of physical financial resource financial activities or virtual resource payments, financial information from different data sources is typically processed, and ultimately a detection result corresponding to the target user is presented on the user interface (UI), determining whether the detected image is a live face image of the user or attack information. The face classification result obtained by the user in the current display interface (for example, determining whether the current detection is attack information) can also be used by other applications.

[0083] In some embodiments, the terminal can be a smart device such as a mobile phone or an in-vehicle smart system. The segmentation network training method provided by the present invention can be provided as a cloud service to various clients (packaged in an in-vehicle terminal or in various mobile electronic devices). The specific use scenario is not specifically limited in this application. Specifically, the cloud service can be provided to enterprise clients to help them train a segmentation network and detect information such as the type, location, and speed of vehicles appearing on the road, as well as the location and speed of pedestrians.

[0084] Among them, the segmentation network training method provided in the embodiment of the present application is based on artificial intelligence. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0085] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0086] In the embodiments of the present application, the artificial intelligence software technologies mainly involved include the above-mentioned speech processing technology and machine learning. For example, it may involve the speech recognition technology (Automatic Speech Recognition, ASR) in speech technology, including speech signal preprocessing, speech signal frequency analysis, speech signal feature extraction, speech signal feature matching / recognition, speech training, etc.

[0087] For example, this may involve machine learning (ML), a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of artificial intelligence. Machine learning typically includes techniques such as deep learning, which includes artificial neural networks (ANNs), such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and deep neural networks (DNNs).

[0088] The structure of the split network training device according to the embodiment of the present invention is described in detail below. The split network training device can be implemented in various forms, such as a dedicated terminal with a split network training device processing function, or a server with a split network training device processing function, such as the preceding embodiment. Figure 1 Server 200 in. Figure 2 The schematic diagram of the structure of the electronic device provided in the embodiment of the present invention can be understood as follows: Figure 2 Only the exemplary structure of the segmentation network training device is shown, not the entire structure, and can be implemented as needed. Figure 2 Partial or complete structure shown.

[0089] The segmentation network training device provided in the embodiment of the present invention includes: at least one processor 201, a memory 202, a user interface 203 and at least one network interface 204. The various components in the segmentation network training device are coupled together via a bus system 205. It is understood that the bus system 205 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 205 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 205 is not described in detail. Figure 2 Various buses are labeled as bus system 205 .

[0090] The user interface 203 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.

[0091] It will be appreciated that the memory 202 may be volatile memory or non-volatile memory, or may include both. The memory 202 in this embodiment of the present invention can store data to support the operation of the terminal (e.g., 10-1). Examples of such data include any computer program used to operate on the terminal (e.g., 10-1), such as an operating system and application programs. The operating system includes various system programs, such as a framework layer, a core library layer, and a driver layer, which implement various basic services and handle hardware-based tasks. Application programs may include various application programs.

[0092] In some embodiments, the segmentation network training apparatus provided in embodiments of the present invention can be implemented using a combination of software and hardware. As an example, the segmentation network training apparatus provided in embodiments of the present invention can be a processor in the form of a hardware decoding processor that is programmed to execute the segmentation network training method provided in embodiments of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0093] As an example of a segmentation network training device provided in an embodiment of the present invention being implemented using a combination of software and hardware, the segmentation network training device provided in an embodiment of the present invention can be directly embodied as a combination of software modules executed by the processor 201. The software module can be located in a storage medium, and the storage medium is located in the memory 202. The processor 201 reads the executable instructions included in the software module in the memory 202 and completes the segmentation network training method provided in the embodiment of the present invention in combination with the necessary hardware (for example, including the processor 201 and other components connected to the bus 205).

[0094] As an example, the processor 201 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0095] As an example of a hardware implementation of the segmentation network training device provided in an embodiment of the present invention, the device provided in an embodiment of the present invention can be directly executed by a processor 201 in the form of a hardware decoding processor. For example, the segmentation network training method provided in an embodiment of the present invention can be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0096] The memory 202 in the embodiment of the present invention is used to store various types of data to support the operation of the segmentation network training device. Examples of such data include any executable instructions for operating on the segmentation network training device, such as executable instructions. A program implementing the segmentation network training method of the embodiment of the present invention may be included in the executable instructions.

[0097] In other embodiments, the segmentation network training device provided by the embodiment of the present invention can be implemented in software. Figure 2The segmentation network training device stored in the memory 202 is shown. The device may be software in the form of a program or plug-in, and may include a series of modules. As an example of a program stored in the memory 202, a segmentation network training device may be included. The segmentation network training device includes the following software modules: an information transmission module 2081 and an information processing module 2082. When the software modules in the segmentation network training device are read into the RAM by the processor 201 and executed, the segmentation network training method provided in the embodiment of the present invention is implemented. The functions of the various software modules in the segmentation network training device include:

[0098] An information transmission module 2081 is configured to obtain a source domain data set and a target domain data set, wherein the source domain data set carries a label and the target domain data set does not carry a label;

[0099] A training module 2082 is configured to train a basic segmentation network using the source domain data set to obtain a trained basic segmentation network;

[0100] The training module 2082 is configured to extract a first prototype vector corresponding to the source domain data set through the basic segmentation network;

[0101] The training module 2082 is configured to extract a target feature vector corresponding to the target domain data set through the basic segmentation network;

[0102] The training module 2082 is configured to perform comparative migration processing based on a similarity function between the first prototype vector and the target feature vector to determine a migration loss function of the basic segmentation network;

[0103] The training module 2082 is configured to adjust the basic segmentation network using the transfer loss function to obtain a first segmentation network;

[0104] The training module 2082 is configured to update the first prototype vector using the target feature vector to obtain a second prototype vector;

[0105] The training module 2082 is configured to adjust the first segmentation network using the second prototype vector to obtain a second segmentation network.

[0106] according to Figure 2The segmentation network training device encapsulated in the electronic device shown, in one aspect of the present application, further provides a computer program product or computer program, the computer program product or computer program including computer instructions, the computer instructions stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the different embodiments and combinations of embodiments provided in various optional implementations of the segmentation network training method described above.

[0107] Before introducing the segmentation network training method provided by this application, we first introduce the transfer learning algorithm in the related art. Figure 3 , Figure 3 It is a schematic diagram of the effect of the transfer learning algorithm in the embodiment of the present invention, wherein the relevant technical solutions are mainly divided into two types: adversarial training and direct distribution alignment. Adversarial training usually adds an additional discriminator, and the discriminator globally trains the source domain and the target domain, so that the generator obtains a domain-invariant feature expression. The direct distribution alignment method usually directly calculates the distribution distance under the two domains, and by adding a regularization constraint of distribution consistency, the two domains are pulled together as much as possible, and finally a domain-independent feature expression is obtained. Most of the existing transfer learning algorithms do not consider category information, but only constrain from a global perspective, trying to obtain a domain-independent feature. However, semantic segmentation is a multi-classification problem, and distribution alignment that does not consider category information will often cause misaligned classification between categories. As Figure 3 As shown, Figure 3 (a) is a schematic diagram showing the effect of not considering the constraints of category time, which causes the target domain to be pulled to the wrong classification interface. Figure 3 (b) The target domain is correctly pulled to the classification interface of the classifier.

[0108] Transfer learning is a subfield of machine learning that aims to transfer existing labeled data (also known as source domain data) to unlabeled data (also known as target domain data), thereby leveraging the knowledge embodied in the source domain data to improve the learning performance of target domain predictive models. Typically, source domain data is plentiful, while target domain data is limited. A common transfer learning approach involves pre-training a deep neural network on a source domain dataset with a large amount of labeled data. The network's weights are then used as initialization values ​​or as a feature extractor for a related task. Gradient descent is then used to fine-tune the deep neural network, either in its entirety or in part, on a dataset related to the target task. This approach makes learning more efficient on a smaller dataset of the target task. However, transfer learning typically requires a sufficient amount of source domain data. Furthermore, because the pre-trained model based on source domain data still needs to be retrained on target domain data to adapt it to the target task, overfitting can occur when the target domain data is too small.

[0109] To solve the above defects, refer to Figure 4 , Figure 4 This is an optional flow chart of the segmentation network training method provided in this application. It can be understood that: Figure 4 The steps shown can be performed by various electronic devices running the segmentation network training device, such as a dedicated terminal with image detection function, a server or server cluster with segmentation network training method function, to achieve training and deployment of segmentation networks adapted to different usage scenarios. Figure 4 The steps shown are explained.

[0110] Step 401: A segmentation network training apparatus obtains a source domain data set and a target domain data set, wherein the source domain data set carries a label, and the target domain data set does not carry a label.

[0111] In some embodiments of the present invention, the source domain data set may include: source domain images, and the target domain data set may include: target domain images. Source domain images refer to images that can provide rich annotation information, and target domain images refer to images in the field where the test data set is located and lack annotation information. The target domain images include a test data set and a training data set. The test data set contains the images to be segmented that we need. During the training of the segmentation network, the target domain images used refer to the images in the training data set. The source domain images and the target domain images describe the same scene and solve the same type of tasks. The data distribution of the source domain images and the target domain images are related but different. Therefore, the source domain images and the target domain images may be very different in appearance style, but they will have many similarities in the segmentation results, such as spatial layout and local context. The present application can apply a segmentation network that performs well in source domain images to target domain images by performing domain adaptation training on the segmentation network. The source domain images can be collected by an image acquisition device, and the images can be annotated by professionals and then sent to the segmentation network training device. For example, the source domain image can be specifically expressed as a medical image. Then the source domain image can be used by various medical image acquisition devices, such as a computed tomography (CT) scanner or a magnetic resonance imaging device, to acquire images of living tissues, and then be annotated by an radiologist and then provided to the segmentation network training device, that is, it can receive medical image samples sent by the medical image acquisition device.

[0112] In some embodiments of the present invention, the source domain image can be a street view image. The source domain image can be collected by street surveillance equipment, annotated by professionals, and then provided to the segmentation network training device. Alternatively, a street view image synthesized by a game engine can be collected from game videos as the source domain image. Medical images refer to images of a living organism or a portion of a living organism obtained non-invasively in medical treatment or medical research, such as images of the human brain, intestines, liver, heart, throat, and vagina. These images can be CT images, MRI images, or positron emission tomography scans. A living organism refers to an independent individual with a living form, such as a human or an animal. The source domain image can refer to an image that has been acquired by medical image acquisition equipment through various means, such as from a database or the Internet. The source domain image can be an image sample that has been annotated with specific meaning by professionals, or an image sample that has not been processed in any way.

[0113] In some embodiments of the present invention, using a segmentation network for facial feature liveness detection as an example, the source domain data set is a set of facial images. After obtaining a user image captured by a terminal, facial detection techniques can be used to first frame the area where the user's face is located. This area is then expanded 1.8 times to obtain more background content, and the facial image including the background content is cropped. For example, the following method can be used: a face detection algorithm is used to frame the target face; a facial feature location algorithm is used to mark facial feature points such as the eyes, mouth, and nose; and the facial image including the background content is cropped based on the detected face position. A depth estimation network is then used to calculate the depth map corresponding to the cropped real face.

[0114] In some embodiments of the present invention, when identifying image frames of media information, the segmentation network of the present application can be used to perform pixel-based image classification to obtain the video object images contained in each video frame to be played, and identify the position and category of the video object images.

[0115] Step 402: The segmentation network training device trains the basic segmentation network using the source domain data set to obtain a trained basic segmentation network.

[0116] In some embodiments of the present invention, training a basic segmentation network using the source domain data set to obtain a trained basic segmentation network can be achieved by:

[0117] Determine the number of images, the number of image categories, and the image size data in the source domain data set; perform one-hot encoding on the number of image categories and the image size data to form a source domain image feature vector; determine the cross entropy loss function corresponding to the basic segmentation network based on the number of images, the number of image categories, the image size data, and the source domain image feature vector; based on the cross entropy loss function, train the basic segmentation network through the source domain data set to obtain a trained basic segmentation network. The image size data has different values ​​for different usage scenarios. For example, for medical images, it can be the height and diameter of the tumor image. For characters in media information, it can be length and width, and for liveness detection, it can be length, width, and height. Different usage scenarios have different combinations, and users can adjust them flexibly.

[0118] In some embodiments of the present invention, the segmentation model may adopt a deep experiment (DeepLab) structure, including but not limited to DeepLabV1, DeepLabV2, DeepLabV3 and DeepLabV3+. Among them, the DeepLabV2 structure is a CNN model structure for image segmentation, which inputs a picture and outputs a mask image of the same size as the original image. The value of each pixel in the image represents the category label value to which the pixel belongs. The DeepLabV3+ structure is an improved CNN model structure for image segmentation based on DeeplabV2, which can usually achieve better results in image segmentation competitions. CNN is a development of the neural network model, which replaces the fully connected layer structure in the artificial neural network with a convolutional layer, and has achieved excellent performance in various computer vision fields.

[0119] In some embodiments of the present invention, when using the DeepLabV2 structure, for the source domain dataset and unlabeled target domain datasets , we can use the cross entropy loss function to train a basic segmentation model using formula 1:

[0120] Formula 1

[0121] Among them, N S is the number of images in the source domain, H, W are the height and width of the image respectively, and C is the number of categories. It is a one-hot encoded vector, where different categories of data in the source domain are encoded by one-hot encoding, specifically including: using an N-bit state register to encode N states, each state has its own independent register bit, and at any time, only one of them is valid.

[0122] Step 403: The segmentation network training device extracts the first prototype vector corresponding to the source domain data set through the basic segmentation network.

[0123] In some embodiments of the present invention, extracting the first prototype vector corresponding to the source domain data set may be achieved by:

[0124] Extracting a source domain feature vector corresponding to the source domain data set through the basic segmentation network; and calculating a first prototype vector corresponding to the source domain data set based on the feature labels of the source domain data set, the number of images, the number of image categories, the image size data, and the source domain feature vector. When the source domain image size data used includes width and height, the prototype vector of each category is calculated using Formula 2:

[0125] Formula 2

[0126] in, is the extracted source domain feature vector, and Y is the label of the source domain data set.

[0127] Step 404: The segmentation network training device extracts the target feature vector corresponding to the target domain data set through the basic segmentation network.

[0128] Step 405: The segmentation network training device performs comparative migration processing based on the similarity function between the first prototype vector and the target feature vector to determine the migration loss function of the basic segmentation network.

[0129] refer to Figure 5 , Figure 5 This is an optional flow chart of the segmentation network training method provided in this application. It can be understood that: Figure 5 The steps shown can be performed by various electronic devices running the segmentation network training device, such as a dedicated terminal with image detection function, a server or server cluster with segmentation network training method function, to achieve training and deployment of segmentation networks adapted to different usage scenarios. Figure 5 The steps shown are explained.

[0130] Step 501: The segmentation network training device calculates a similarity function between the first prototype vector and the target feature vector.

[0131] Step 502: The segmentation network training device obtains pseudo labels of the target domain data set.

[0132] Step 503: The segmentation network training device determines a transfer learning loss function based on the similarity function, the number of image categories, the image size data, and the pseudo labels.

[0133] Step 504: The segmentation network training device determines an inter-class constraint loss function based on the similarity function, the number of image categories, the image size data, and the source domain image feature vector.

[0134] Step 505: The segmentation network training device determines the transfer loss function of the basic segmentation network based on the transfer learning loss function and the inter-class constraint loss function.

[0135] Among them, taking the processing of medical images by segmentation network as an example, for a target domain image, is the target domain feature extracted by the training device. The similarity function between each feature and the prototype feature can be calculated by formula 3:

[0136] Formula 3

[0137] For the data in the target domain, obtain the corresponding pseudo label , then the transfer learning loss function calculation from the target domain data set to the source domain data set refers to Formula 4:

[0138] Formula 4

[0139] In order to achieve inter-class constraints in the source domain and improve the accuracy of the segmentation network, the inter-class loss function of the source domain data set is calculated using Formula 5:

[0140] Formula 5

[0141] The migration loss function of the basic segmentation network is the sum of the above two loss functions. The calculation refers to Formula 6:

[0142] Formula 6

[0143] refer to Figure 6 , Figure 6 This is an optional flow chart of the segmentation network training method provided in this application, wherein Figure 5 On the basis of the processing process shown above, in order to increase the reliability of the migration loss function of the basic segmentation network, the output space of the label can also be used for comparative migration, which includes the following steps:

[0144] Step 5051: The segmentation network training device determines a loss function of the label output space of the source domain data set;

[0145] Step 5052: The segmentation network training device determines the transfer loss function of the basic segmentation network based on the loss function of the label output space, the transfer learning loss function and the inter-class constraint loss function.

[0146] Among them, the migration loss function of the basic segmentation network can be calculated by formula 7:

[0147] Formula 7

[0148] Step 406: The segmentation network training device adjusts the basic segmentation network using the transfer loss function to obtain a first segmentation network.

[0149] In some embodiments of the present invention, adjusting the basic segmentation network by using the transfer loss function to obtain the first segmentation network can be achieved by:

[0150] When the target domain data set is a medical image set, the medical image set is substituted into the migration loss function of the basic segmentation network; when it is determined that the migration loss function of the basic segmentation network meets the corresponding convergence condition, the encoder and decoder of the basic segmentation network correspond to the update parameters; the basic segmentation network is adjusted by the update parameters to obtain a first segmentation network.

[0151] Step 407: The segmentation network training device updates the first prototype vector using the target feature vector to obtain a second prototype vector.

[0152] In some embodiments of the present invention, updating the first prototype vector using the target feature vector to obtain the second prototype vector can be achieved in the following manner:

[0153] Determine a hyperparameter that controls the update rate; obtain the number of pixels of any type in the target domain data set at different update stages; based on the number of pixels and the hyperparameter, update the first prototype vector based on the target feature vector to obtain the second prototype vector. Updating the source domain data set using the vector of the target domain data set can be achieved using Formula 8:

[0154] Formula 8

[0155] in, Refers to the number of pixels belonging to category C as of the last update, where category C is any category of pixels in the target domain data set, and this application does not impose any specific restrictions on this. Refers to the number of pixels belonging to category c at the time of the current update. However, when the prototype vector is updated by the update method shown in Formula 8, due to The number is far greater than , so the updated prototype vector does not fully absorb the characteristics of the target domain data set. To this end, the momentum prototype vector update can be achieved through Formula 9:

[0156] Formula 9

[0157] Among them, m is a hyperparameter used to control the update rate. Users can flexibly adjust the hyperparameter according to different application environments of the segmentation network.

[0158] Step 408: The segmentation network training device adjusts the first segmentation network using the second prototype vector to obtain a second segmentation network.

[0159] Once the segmentation network training is completed, it can be deployed on the server and process the image to be processed. Figure 7 , Figure 7 This is an optional flowchart of the segmentation network training method provided in this application, which specifically includes the following steps:

[0160] Step 701: Acquire an image to be processed, wherein the image to be processed includes at least one of the following:

[0161] Medical images, road information monitoring images, and media information display images.

[0162] Step 702: Segment the image to be processed using the second segmentation network to obtain an image segmentation result.

[0163] In order to better illustrate the processing process of the segmentation network training method provided by this application, the following uses the segmentation network to process road images as an example to illustrate the usage environment of the segmentation network training method provided by this application. Figure 1 The schematic diagram of the usage scenario shows that the segmentation network training method provided by the present invention can be used as a cloud service to serve various types of customers (packaged in a vehicle-mounted terminal or packaged in different mobile electronic devices). The specific usage scenario is not specifically limited in this application. Among them, it is provided to corporate customers as a cloud service to help corporate customers test different autonomous driving algorithms.

[0164] In some embodiments of the present invention, reference Figure 8 , Figure 8 An optional flow chart of the segmentation network training method provided in an embodiment of the present invention specifically includes the following steps:

[0165] Step 801: Acquire a source domain data set and a target domain data set, wherein the source domain data set carries labels, and the target domain data set does not carry labels.

[0166] Step 802: extracting a first prototype vector corresponding to the source domain data set and a target feature vector corresponding to the target domain data set through a basic segmentation network.

[0167] Step 803: performing comparative migration processing based on the similarity function between the first prototype vector and the target feature vector to determine the migration loss function of the basic segmentation network.

[0168] Step 804: Adjust the basic segmentation network using the transfer loss function to obtain a first segmentation network.

[0169] Step 805: Use the target feature vector to update the first prototype vector to obtain a second prototype vector.

[0170] Step 806: Adjust the first segmentation network using the second prototype vector to obtain a second segmentation network.

[0171] Step 807: Deploy a second segmentation network and obtain a road information monitoring image.

[0172] Step 808: Segment the image to be processed by the second segmentation network to obtain an image segmentation result, and mark it.

[0173] refer to Figure 9 , Figure 9 This is a schematic diagram of the effect of segmenting the network in an embodiment of the present invention. The segmentation results can be marked to indicate pedestrian collision risk, vehicle collision risk, traffic congestion, road hazards (water accumulation, ice, potholes), and traffic light status. The system can also detect the position and speed of vehicles and pedestrians within the field of view in real time, and calculate real-time traffic flow on the road. It can also obtain information such as the type, position, and speed of vehicles, and the position and speed of pedestrians. It can also detect information in road information monitoring images, such as vehicle congestion queue status, vehicle position and type, driving direction, visibility (fog), and people and animals.

[0174] Beneficial technical effects:

[0175] The present invention obtains a source domain data set and a target domain data set, wherein the source domain data set carries labels and the target domain data set does not carry labels; a basic segmentation network is trained through the source domain data set to obtain a trained basic segmentation network; a first prototype vector corresponding to the source domain data set is extracted through the basic segmentation network; a target feature vector corresponding to the target domain data set is extracted through the basic segmentation network; a comparative migration process is performed based on a similarity function between the first prototype vector and the target feature vector to determine a migration loss function of the basic segmentation network; the basic segmentation network is adjusted through the migration loss function to obtain a first segmentation network; the first prototype vector is updated using the target feature vector to obtain a second prototype vector; the first segmentation network is adjusted through the second prototype vector to obtain a second segmentation network, thereby taking into account the training accuracy while making the generalization ability and data processing ability of the segmentation network stronger, adapting to different data processing environments, reducing the cost of data labeling, enhancing the classification robustness of the segmentation network, and reducing the problem of easy overfitting of the segmentation network.

[0176] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A segmentation network training method, characterized in that: The method comprises: Acquire a source domain data set and a target domain data set, wherein the source domain data set includes source domain images and carries labels, and the target domain data set includes target domain images and does not carry labels; Training a basic segmentation network using the source domain data set to obtain a trained basic segmentation network; Extracting a first prototype vector corresponding to the source domain data set and a target feature vector corresponding to the target domain data set through the basic segmentation network; Performing comparative migration processing based on a similarity function between the first prototype vector and the target feature vector to determine a migration loss function of the basic segmentation network; Adjusting the basic segmentation network by using the migration loss function to obtain a first segmentation network; Updating the first prototype vector using the target feature vector to obtain a second prototype vector; The first segmentation network is adjusted using the second prototype vector to obtain a second segmentation network, wherein the second segmentation network is used to segment the image to be processed to obtain an image segmentation result.

2. The method according to claim 1, characterized in that The step of training a basic segmentation network using the source domain data set to obtain a trained basic segmentation network includes: Determining the number of images, the number of image categories, and image size data in the source domain data set; Performing one-hot encoding on the image category quantity and image size data to form a source domain image feature vector; Determining a cross entropy loss function corresponding to the basic segmentation network based on the number of images, the number of image categories, the image size data, and the source domain image feature vector; Based on the cross entropy loss function, the basic segmentation network is trained using the source domain data set to obtain a trained basic segmentation network.

3. The method according to claim 2, characterized in that The extracting, through the basic segmentation network, a first prototype vector corresponding to the source domain data set includes: Extracting a source domain feature vector corresponding to the source domain data set through the basic segmentation network; A first prototype vector corresponding to the source domain data set is calculated based on the feature labels of the source domain data set, the number of images, the number of image categories, the image size data, and the source domain feature vector.

4. The method according to claim 2, characterized in that The performing comparative migration processing according to the similarity function between the first prototype vector and the target feature vector to determine the migration loss function of the basic segmentation network includes: Calculating a similarity function between the first prototype vector and the target feature vector; Obtaining pseudo labels for the target domain data set; Determining a transfer learning loss function based on the similarity function, the number of image categories, the image size data, and the pseudo labels; determining an inter-class constraint loss function according to the similarity function, the number of image categories, the image size data, and the source domain image feature vector; Based on the transfer learning loss function and the inter-class constraint loss function, a transfer loss function of the basic segmentation network is determined.

5. The method according to claim 4, characterized in that The method further comprises: Determining a loss function in the label output space of the source domain data set; A transfer loss function of the basic segmentation network is determined based on the loss function of the label output space, the transfer learning loss function, and the inter-class constraint loss function.

6. The method according to claim 4, characterized in that The step of adjusting the basic segmentation network by using the migration loss function to obtain a first segmentation network includes: When the target domain data set is a medical image set, Substituting the medical image set into the migration loss function of the basic segmentation network; Determining update parameters corresponding to the encoder and decoder of the basic segmentation network when the migration loss function of the basic segmentation network meets corresponding convergence conditions; The basic segmentation network is adjusted by using the update parameters to obtain a first segmentation network.

7. The method according to claim 1, characterized in that The updating of the first prototype vector by using the target feature vector to obtain a second prototype vector includes: Determine the hyperparameters that control the update rate; Obtaining the number of pixels of any type in the target domain data set at different update stages; Based on the number of pixels and the hyperparameter, the first prototype vector is updated based on the target feature vector to obtain the second prototype vector.

8. The method according to claim 1, characterized in that The method further comprises: Acquire the image to be processed, wherein the image to be processed includes at least one of the following: Medical images, road information monitoring images, and media information display images; Segmenting the image to be processed by the second segmentation network to obtain the image segmentation result; The second segmentation network is trained based on the method according to any one of claims 1 to 7.

9. A segmentation network training device, characterized in that: The device comprises: An information transmission module, configured to obtain a source domain data set and a target domain data set, wherein the source domain data set includes source domain images and carries labels, and the target domain data set includes target domain images and does not carry labels; A training module, configured to train a basic segmentation network using the source domain data set to obtain a trained basic segmentation network; The training module is configured to extract a first prototype vector corresponding to the source domain data set through the basic segmentation network; The training module is used to extract the target feature vector corresponding to the target domain data set through the basic segmentation network; The training module is configured to perform comparative migration processing based on a similarity function between the first prototype vector and the target feature vector to determine a migration loss function of the basic segmentation network; The training module is configured to adjust the basic segmentation network using the transfer loss function to obtain a first segmentation network; The training module is configured to update the first prototype vector using the target feature vector to obtain a second prototype vector; The training module is used to adjust the first segmentation network through the second prototype vector to obtain a second segmentation network, wherein the second segmentation network is used to segment the image to be processed to obtain an image segmentation result.

10. The device according to claim 9, characterized in that The training module is also used to: Determining the number of images, the number of image categories, and image size data in the source domain data set; Performing one-hot encoding on the image category quantity and image size data to form a source domain image feature vector; Determining a cross entropy loss function corresponding to the basic segmentation network based on the number of images, the number of image categories, the image size data, and the source domain image feature vector; Based on the cross entropy loss function, the basic segmentation network is trained using the source domain data set to obtain a trained basic segmentation network.

11. The device according to claim 10, characterized in that The training module is also used to: Extracting a source domain feature vector corresponding to the source domain data set through the basic segmentation network; A first prototype vector corresponding to the source domain data set is calculated based on the feature labels of the source domain data set, the number of images, the number of image categories, the image size data, and the source domain feature vector.

12. The device according to claim 10, characterized in that The training module is also used to: Calculating a similarity function between the first prototype vector and the target feature vector; Obtaining pseudo labels for the target domain data set; Determining a transfer learning loss function based on the similarity function, the number of image categories, the image size data, and the pseudo labels; determining an inter-class constraint loss function according to the similarity function, the number of image categories, the image size data, and the source domain image feature vector; Based on the transfer learning loss function and the inter-class constraint loss function, a transfer loss function of the basic segmentation network is determined.

13. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the segmentation network training method according to any one of claims 1 to 8 is implemented.

14. An electronic device, characterized in that: The electronic device comprises: a memory for storing executable instructions; A processor, configured to implement the segmentation network training method according to any one of claims 1 to 8 when running the executable instructions stored in the memory.

15. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by a processor, the segmentation network training method according to any one of claims 1 to 8 is implemented.

Images