A neural network training method, image processing method and related equipment

Abstract

The embodiment of the present application discloses a neural network training method, an image processing method and related equipment, which can be used in the field of image processing in the field of artificial intelligence. The method includes: obtaining a first training data set and a second training data set, wherein the first training image in the first training data set is obtained by mixing a labeled training image of a source domain with a labeled training image of a target domain at the pixel level, and the second training data set includes unlabeled training images from the target domain; training a first neural network according to the first training data set to obtain a first teacher network; and training a first student network according to the first teacher network and the second training data set by using knowledge distillation. The difficulty of the teacher network in the training process is increased, thereby improving the processing capability of the trained teacher network for the image of the target domain, which is conducive to improving the performance of the trained student network in processing the image of the target domain.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a neural network training method, an image processing method, and related equipment. Background Art

[0002] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Using artificial intelligence for image processing is a common application of artificial intelligence.

[0003] At present, the widespread data domain gap problem leads to very low generalization ability of image feature extraction. The trained neural network can only be deployed on application data of the same scenario as the training data from the source domain. If the trained neural network is used to process images from the target domain, the prediction results obtained will have relatively poor performance.

[0004] Therefore, a solution to improve the performance of neural networks in processing images in the target domain is urgently needed. Summary of the invention

[0005] The embodiments of the present application provide a neural network training method, an image processing method, and related equipment, which are used to improve the processing capability of a trained teacher network for images of a target domain, that is, to improve the generalization capability of the trained teacher network, so that the teacher network can generate prediction results with better performance for unlabeled training images from the target domain, that is, the quality of the training data of the student network is higher, which is conducive to improving the performance of the trained student network in processing images of the target domain.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a neural network training method, which can be used in the field of image processing in the field of artificial intelligence. A training device obtains a first training data set and a second training data set, wherein the first training data set includes at least one first labeled training image, the first training image is obtained by mixing a labeled training image from a source domain with a labeled training image from a target domain at a pixel level, the label of the first training image is obtained according to the first label and the second label, the first label corresponds to the labeled training image from the source domain, the second label corresponds to the labeled training image from the target domain, and the second training data set includes unlabeled training images from the target domain. The training device uses a first loss function to train a first neural network according to a first training image in the first training data set until a preset condition is met to obtain a first teacher network, the first loss function indicates the similarity between a prediction result corresponding to the first training image and a label of the first training image (also referred to as an expected result of the first training image), the prediction result corresponding to the first training image is output by the first neural network, and the first teacher network is a first neural network that has performed a training operation. The training device trains the first student network in a knowledge distillation manner according to the first teacher network and the second training data set until a preset condition is met to obtain a second student network, where the second student network is the first student network that has been trained. Knowledge distillation refers to the training goal in the process of training the student network to narrow the similarity between the prediction results output by the student network and the prediction results output by the teacher network, so that the student network can learn the image processing ability of the teacher network.

[0008] In this implementation, a teacher network is trained using the first training data set, and then the student network is trained using the teacher network and the second training data set using a knowledge distillation method. Since the training images in the first training data set are obtained by mixing labeled training images from the source domain with labeled training images from the target domain at the pixel level, the difficulty of the teacher network in the training process is increased, thereby improving the processing capability of the trained teacher network for images of the target domain, that is, improving the generalization capability of the trained teacher network, so that the teacher network can generate better prediction results for unlabeled training images from the target domain, that is, the quality of the training data of the student network is higher, which is conducive to improving the performance of the trained student network in processing images of the target domain.

[0009] In a possible implementation of the first aspect, the training device may use a cutting-mixing (CutMix) technology to mix the source domain image to be mixed and the target domain image to be mixed to generate a first training image, in which the first training image has both a part from the target domain and a part from the source domain. The training device also mixes a first label (also referred to as an expected result) corresponding to the source domain image to be mixed and a second label corresponding to the target domain image to be mixed to obtain a label of the first training image.

[0010] In a possible implementation of the first aspect, the training device may adopt a mixing (Mixup) technology to mix a labeled source domain image to be mixed and a labeled target domain image to be mixed to generate a first training image, the training device assigns a first weight to a first label corresponding to the source domain image to be mixed, assigns a second weight to a second label corresponding to the target domain image to be mixed, and performs a weighted sum of the first label and the second label to obtain a label of the first training image (which may also be referred to as an expected result of the first training image).

[0011] In a possible implementation of the first aspect, the method further includes: the training device uses a second loss function to train the first neural network according to the third training image in the third training data set until a preset condition is met to obtain a second teacher network. The third training data set includes labeled training images from the source domain and labeled training images from the target domain; the second loss function indicates the similarity between the prediction result of the third training image and the label of the third training image (also referred to as the expected result of the third training image), and the prediction result of the third training image is output by the first neural network; the second teacher network is the first neural network that has performed the training operation. The training device uses a knowledge distillation method to train the first student network according to the first teacher network and the second training data set until the preset condition is met to obtain the second student network, including: the training device uses a knowledge distillation method to train the first student network according to the first teacher network, the second teacher network and the second training data set until the preset condition is met to obtain the second student network.

[0012] In this implementation, the labeled training images from the source domain and the labeled training images from the target domain are directly mixed at the image level to obtain a third training data set, and the second teacher network is trained based on the third training data set. Since the first training image breaks the data gap between the target domain images and the source domain images at the pixel level, the difficulty of the first teacher network in the training process is increased, resulting in the first teacher network classifying objects without regular shapes in the target domain (such as the sky, wall, sea, etc.) more accurately; the second teacher network has a clearer overall information of the target domain image, and classifies objects that need to be predicted with the help of shape in the target domain (such as street lights, traffic lights, etc.) more accurately. The prediction results output by the first teacher network and the second teacher network are complementary. Combining the prediction results of the first teacher network and the second teacher network as the learning goals of the student network is conducive to further improving the accuracy of the second student network in processing images in the target domain.

[0013] In a possible implementation of the first aspect, the training device trains the first student network by using the knowledge distillation method according to the first teacher network, the second teacher network and the second training data set, including: the training device obtains a second training image, the second training image is any training image in the second training data set; the training device processes the second training image using the first teacher network to generate a first prediction result corresponding to the second training image; the training device processes the second training image using the second teacher network to generate a second prediction result corresponding to the second training image. The training device performs a weighted summation of the first prediction result and the second prediction result to obtain a target prediction result corresponding to the second training image; optionally, the weights of the first prediction result and the second prediction result can be 0.5 and 0.5 respectively, that is, the training device can average the first prediction result and the second prediction result to obtain the target prediction result. The training device processes the second training image using the first student network to generate a third prediction result corresponding to the second training image; the training device trains the first student network according to the target loss function, and the target loss function indicates the similarity between the third prediction result and the target prediction result.

[0014] In this implementation, when the first student network is trained by using knowledge distillation based on the first teacher network, the second teacher network and the second training data set, the first prediction result and the second prediction result are weightedly summed to obtain the target prediction result corresponding to the second training image, that is, the learning target of the first student network. The operation is simple and easy to implement. Moreover, since the outputs of the first teacher network and the second teacher network are complementary, the weighted summation method is conducive to giving play to the advantages brought by complementarity and is conducive to further improving the accuracy of the learning target of the first student network.

[0015] In a possible implementation of the first aspect, after the training device obtains the second student network, the method further includes: the training device processes the second training image in the second training data set through the second student network to generate a prediction result corresponding to the second training image, and determines the label of the second training image according to the prediction result corresponding to the second training image. The training device performs a pixel-level mixing operation based on the labeled training image from the source domain, the labeled training image from the target domain, and the labeled second training image to obtain an updated first training data set; the training device trains the first neural network based on the updated first training data set until a preset condition is met to obtain an updated first teacher network; the training device trains the first student network based on the updated first teacher network and the second training data set by using knowledge distillation until a preset condition is met to obtain an updated second student network, and the updated second student network is the first student network that has performed the training operation.

[0016] In this implementation, after the second student network is obtained through initial training, the second student network is also used to generate pseudo labels for the second training images, that is, pseudo labels are generated for a large number of unlabeled training images from the target domain, and based on a large number of labeled second training images from the target domain, the labeled training images from the source domain and the labeled training images from the target domain are again subjected to image-level mixing operations to obtain an updated first training data set. Since more training data from the target domain are introduced into the updated first training data set, the updated first teacher network will perform better when processing images from the target domain, and thus the updated second student network will also perform better when processing images from the target domain, which is conducive to obtaining a second student network with higher accuracy.

[0017] In a possible implementation of the first aspect, the method further includes: the training device performs an image-level mixing operation based on the labeled training images from the source domain, the labeled training images from the target domain, and the labeled second training images to obtain an updated third training data set. The training device trains the first neural network based on the updated third training data set until a preset condition is met to obtain an updated second teacher network. The training device trains the first student network based on the updated first teacher network and the second training data set using a knowledge distillation method, including: the training device trains the first student network based on the updated first teacher network, the updated second teacher network, and the second training data set using a knowledge distillation method.

[0018] In a possible implementation of the first aspect, the training device trains the first student network by using knowledge distillation according to the first teacher network, the second teacher network and the second training data set, including: the training device trains the first student network by using knowledge distillation according to the first teacher network, the second teacher network and the second training data set. The training device also trains the first student network by using a fourth loss function according to a fourth training data set; wherein the fourth training data set includes a fourth training image with a label from the target domain, that is, the fourth training data set includes a fourth training image from the target domain and a label corresponding to each fourth training image, and the fourth loss function indicates the similarity between the label corresponding to the fourth training image and the fourth prediction result, and the fourth prediction result is output after the first student network processes the fourth training image.

[0019] In this implementation, the first student network is also trained using labeled training images from the target domain to further improve the processing capability of the second student network for the images of the target domain, so as to improve the performance of the second student network in processing the images of the target domain.

[0020] In a possible implementation of the first aspect, the training images of the source domain and the training images of the target domain are derived from different image generation devices, or the training images of the source domain and the training images of the target domain are derived from different image acquisition locations, or the image acquisition time of the training images of the source domain and the image acquisition time of the training images of the target domain are different, or the photographed objects in the training images of the source domain and the photographed objects in the training images of the target domain are different object types. In this implementation, various examples are given for source domain images and target domain images, which expands the application scenarios of this solution and improves the implementation flexibility of this solution.

[0021] In a possible implementation manner of the first aspect, the first neural network and the first student network are both used to perform any one of the following tasks: image semantic segmentation, image classification, and object detection on images.

[0022] In a second aspect, an embodiment of the present application provides an image processing method, which can be used in the field of image processing in the field of artificial intelligence, wherein an execution device obtains an image to be processed from a target domain; the execution device inputs the image to be processed from the target domain into a second student network to process the image to be processed through the second student network to obtain a prediction result output by the second student network; wherein the second student network is trained by using a knowledge distillation method based on a first teacher network and a second training data set, the first teacher network is trained based on a first training data set, the first training data set includes at least one labeled first training image, the first training image is obtained by mixing a labeled training image from a source domain with a labeled training image from a target domain at a pixel level, the label of the first training image is obtained based on the first label and the second label, the first label corresponds to the labeled training image from the source domain, the second label corresponds to the labeled training image from the target domain, and the second training data set includes unlabeled training images from the target domain.

[0023] In a possible implementation of the second aspect, the second student network is trained by using knowledge distillation based on the first teacher network, the second teacher network and the second training data set, and the second teacher network is trained based on a third training data set. The third training data set includes labeled training images from a source domain and labeled training images from a target domain, and the third training data set is obtained by mixing the labeled training images from the source domain and the labeled training images from the target domain at the image level.

[0024] For the specific implementation steps of the second aspect of the embodiment of the present application and the various possible implementation methods of the second aspect, as well as the beneficial effects brought about by each possible implementation method, please refer to the description of the various possible implementation methods in the second aspect, and will not be repeated here one by one.

[0025] In a third aspect, an embodiment of the present application provides a neural network training device, which can be used in the field of image processing in the field of artificial intelligence, the device comprising: an acquisition module, used to acquire a first training data set and a second training data set, wherein the first training data set comprises at least one first labeled training image, the first training image is obtained by mixing a labeled training image from a source domain with a labeled training image from a target domain at a pixel level, the label of the first training image is obtained according to the first label and the second label, the first label corresponds to the labeled training image from the source domain, the second label corresponds to the labeled training image from the target domain, and the second training data set comprises unlabeled training images from the target domain; a training module, used to train the first neural network according to the first training data set until a preset condition is met to obtain a first teacher network, the first teacher network being the first neural network that has performed a training operation; the training module, further used to train the first student network according to the first teacher network and the second training data set by adopting a knowledge distillation method until a preset condition is met to obtain a second student network, the second student network being the first student network that has performed a training operation.

[0026] The neural network training device provided in the third aspect of the embodiment of the present application can also execute the steps executed by the execution device in each possible implementation method of the first aspect. For the specific implementation steps of the third aspect of the embodiment of the present application and the various possible implementation methods of the third aspect, as well as the beneficial effects brought about by each possible implementation method, reference can be made to the description of the various possible implementation methods in the first aspect, and they will not be repeated here one by one.

[0027] In a fourth aspect, an embodiment of the present application provides an image processing device, which can be used in the field of image processing in the field of artificial intelligence, the device comprising: an acquisition module, for acquiring an image to be processed from a target domain; a processing module, for inputting the image to be processed from the target domain into a second student network, so as to process the image to be processed through the second student network, and obtain a prediction result output by the second student network; wherein the second student network is trained by using a knowledge distillation method according to a first teacher network and a second training data set, the first teacher network is trained according to a first training data set, the first training data set comprises at least one first labeled training image, the first training image is obtained by mixing a labeled training image from a source domain with a labeled training image from a target domain at a pixel level, the label of the first training image is obtained according to the first label and the second label, the first label corresponds to the labeled training image from the source domain, the second label corresponds to the labeled training image from the target domain, and the second training data set comprises unlabeled training images from the target domain.

[0028] The image processing device provided in the fourth aspect of the embodiment of the present application can also execute the steps performed by the training device in each possible implementation method of the second aspect. For the specific implementation steps of the fourth aspect of the embodiment of the present application and the various possible implementation methods of the fourth aspect, as well as the beneficial effects brought about by each possible implementation method, you can refer to the description of the various possible implementation methods in the second aspect, and will not be repeated here one by one.

[0029] In a fifth aspect, an embodiment of the present application provides an execution device, which may include a processor, the processor and a memory are coupled, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the image processing method described in the second aspect above is implemented.

[0030] In a sixth aspect, an embodiment of the present application provides a training device, which may include a processor, the processor and a memory are coupled, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the training method of the neural network described in the first aspect above is implemented.

[0031] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is run on a computer, the computer executes the image processing method described in the second aspect, or the neural network training method described in the first aspect.

[0032] In an eighth aspect, an embodiment of the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the image processing method, or the processing circuit is configured to execute the neural network training method described in the first aspect above.

[0033] In a ninth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program runs on a computer, it enables the computer to execute the image processing method described in the second aspect above, or enables the computer to execute the neural network training method described in the first aspect above.

[0034] In a tenth aspect, an embodiment of the present application provides a chip system, which includes a processor for implementing the functions involved in the above-mentioned various aspects, for example, sending or processing the data and / or information involved in the above-mentioned method. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for a server or communication device. The chip system can be composed of a chip, or it can include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1A schematic diagram of the structure of the artificial intelligence main framework provided in the embodiment of the present application;

[0036] Figure 2a A system architecture diagram of an image processing system provided in an embodiment of the present application;

[0037] Figure 2b A schematic diagram of a flow chart of a neural network training method provided in an embodiment of the present application;

[0038] Figure 3 Another schematic diagram of a flow chart of a neural network training method provided in an embodiment of the present application;

[0039] Figure 4 A schematic diagram of a first training image obtained in the neural network training method provided in an embodiment of the present application;

[0040] Figure 5 A schematic diagram of a flow chart of training a first neural network in the neural network training method provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of a process for training a first neural network and a second neural network in a neural network training method provided in an embodiment of the present application;

[0042] Figure 7 A schematic diagram of a flow chart of training a first student network in a neural network training method provided in an embodiment of the present application;

[0043] Figure 8 A schematic diagram of another flow chart of a neural network training method provided in an embodiment of the present application;

[0044] Fig. 9 A schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application;

[0045] Fig.10 A schematic diagram of the structure of a neural network training device provided in an embodiment of the present application;

[0046] Fig.11 Another structural schematic diagram of a neural network training device provided in an embodiment of the present application;

[0047] Fig.12 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0048] Fig.13 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0049] Fig.14 A schematic diagram of the structure of a training device provided in an embodiment of the present application;

[0050] Fig.15 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0052] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0053] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0054] (1) Infrastructure

[0055] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and is supported by the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips, which can specifically use hardware acceleration chips such as central processing units (CPU), embedded neural network processing units (NPU), graphics processing units (GPU), application specific integrated circuits (ASIC) or field programmable gate arrays (FPGA); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0056] (2) Data

[0057] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0058] (3) Data processing

[0059] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0060] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0061] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0062] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0063] (4) General ability

[0064] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0065] (5) Smart products and industry applications

[0066] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart manufacturing, smart transportation, smart homes, smart medical care, smart security, autonomous driving, safe cities, etc.

[0067] The embodiments of the present application can be applied to application scenarios for image processing in the above-mentioned fields. Specifically, a training device can store a large number of labeled training images from a source domain, a large number of unlabeled training images from a target domain, and a small number of labeled training images from a target domain for training a neural network. The trained neural network is used to process images from the target domain.

[0068] Among them, the label corresponding to the training image can also be called the expected result corresponding to the training image. The neural network can be used for image semantic segmentation, or the neural network can be used for image classification, or the neural network can be used for object detection in the image, etc., which are not exhaustive here; as an example, in the fields of smart home, smart medical care, and autonomous driving, there are application scenarios in which image semantic segmentation operations are performed by neural networks; as another example, in the fields of smart terminals and smart security, there are application scenarios in which image segmentation is performed by neural networks, etc., which are not exhaustive here.

[0069] In some scenarios, the image of the source domain and the image of the target domain come from different image generating devices. As an example, the image of the source domain is generated by simulation software, and the image of the target domain is obtained by taking a camera. As another example, the image of the source domain and the image of the target domain are obtained by taking different cameras, etc.

[0070] In other scenarios, the source domain images and the target domain images come from different image acquisition locations. For example, the source domain images are acquired in Beijing, while the target domain images are acquired in Shenzhen. In other scenarios, the image acquisition time of the source domain images is different from the image acquisition time of the target domain images. For example, the image acquisition time of the source domain images is during the day, while the image acquisition time of the target domain images is at night.

[0071] In other scenarios, the photographed object in the image of the source domain and the photographed object in the image of the target domain are different object types. For example, the photographed object in the image of the source domain is a Schnauzer, and the photographed object in the image of the target domain is a Shiba Inu, etc. It should be understood that the examples of the source domain image and the target domain image here are only for the convenience of understanding the difference between the source domain image and the target domain image, and are not used to limit this solution. In the embodiments of the present application, various examples are given for the source domain image and the target domain image, which expands the application scenarios of this solution and improves the implementation flexibility of this solution.

[0072] In the above application scenarios, since there are only a small number of labeled training images from the target domain in the training equipment, and the trained neural network needs to process the images of the target domain, but it is difficult to manually annotate the unlabeled training images from the target domain, it is necessary to make full use of the existing source domain data and target domain data so that the trained neural network can perform well when processing the images of the target domain.

[0073] In order to achieve the above-mentioned purpose, the present application embodiment provides a neural network training method. Before introducing the neural network training method provided by the present application embodiment in detail, Figure 2a The image processing system provided in the embodiment of the present application is introduced. Figure 2a , Figure 2a A system architecture diagram of an image processing system provided in an embodiment of the present application, in Figure 2a In the figure, the image processing system 200 includes an execution device 210, a training device 220, a database 230 and a data storage system 240, and the execution device 210 includes a computing module 211.

[0074] Among them, the database 230 stores a first training data set and a second training data set, and the training device 220 generates a first target model / rule 201 and a second target model / rule 202. The first target model / rule 201 and the second target model / rule 202 have the same functions. The first target model / rule 201 and the second target model / rule 202 can be implemented using a neural network or a non-neural network model. In the subsequent embodiments of the present application, the first target model / rule 201 and the second target model / rule 202 are implemented using a neural network as an example, that is, the first target model / rule 201 adopts a first neural network, and the second target model / rule 202 adopts a first student network as an example.

[0075] Specifically, Figure 2b As shown, Figure 2bA flow chart of a neural network training method provided in an embodiment of the present application. S1. The training device 220 obtains a first training data set and a second training data set from the database 230, wherein the first training data set includes at least one first labeled training image, the first training image is obtained by mixing a labeled training image from a source domain with a labeled training image from a target domain at a pixel level, the label of the first training image is obtained according to the first label and the second label, the first label corresponds to the labeled training image from the source domain, the second label corresponds to the labeled training image from the target domain, and the second training data set includes unlabeled training images from the target domain. S2. The training device 220 trains the first neural network (i.e., the first target model / rule 201) according to the first training data set in the database 230 until a preset condition is met to obtain a first teacher network, the first teacher network being the first neural network (i.e., the first target model / rule 201) that has performed a training operation. S3. The training device 220 trains the first student network (that is, the second target model / rule 202) by knowledge distillation according to the first teacher network (that is, the first target model / rule 201 that has been trained) and the second training data set until the preset conditions are met to obtain a second student network, which is the first student network that has been trained.

[0076] The second target model / rule 202 that has been trained and obtained by the training device 220 can be applied to different systems or devices, such as autonomous driving vehicles, desktop computers, mobile phones, tablets, laptops, monitoring systems, radar data processing systems, etc. Among them, the execution device 210 can call data, codes, etc. in the data storage system 240, and can also store data, instructions, etc. in the data storage system 240. The data storage system 240 can be placed in the execution device 210, or the data storage system 240 can be an external memory relative to the execution device 210.

[0077] The computing module 211 can process images from the target domain by executing the second target model / rule 202 that has been trained. Since the training images in the first training data set are obtained by mixing labeled training images from the source domain with labeled training images from the target domain at the pixel level, the difficulty of the training process of the first target model / rule 201 is increased, thereby improving the processing capability of the trained first target model / rule 201 for images of the target domain, so that the teacher network can generate prediction results with better performance for unlabeled training images from the target domain, that is, the quality of the training data of the second target model / rule 202 is higher, which is conducive to improving the performance of the trained second target model / rule 202 in processing images of the target domain.

[0078] In some embodiments of this application, please refer to Figure 2a The "user" can directly interact with the execution device 210, that is, the execution device 210 can directly display the predicted image output by the second target model / rule 202 to the "user". It is worth noting that Figure 2a This is only a schematic diagram of the architecture of the image processing system provided by an embodiment of the present invention, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in other embodiments of the present application, the execution device 210 and the client device can also be separate devices, and the execution device 210 is configured with an input / output (in / out, I / O) interface, and the execution device 210 exchanges data with the client device through the I / O interface.

[0079] In combination with the above description, it can be seen that the specific implementation process of the training phase and the reasoning phase of the neural network provided in the embodiment of the present application will be described below.

[0080] 1. Training Phase

[0081] In the embodiment of the present application, the training phase describes the process of how the training device 220 generates a mature second target model / rule 202 using the image data set in the database 230. For details, please refer to Figure 3 , Figure 3 A flowchart of a neural network training method provided in an embodiment of the present application is provided. The neural network training method provided in an embodiment of the present application may include:

[0082] 301. A training device obtains a first training data set and a second training data set, where the first training data set includes at least one first labeled training image, where the first training image is obtained by mixing a labeled training image from a source domain with a labeled training image from a target domain at a pixel level, where the label of the first training image is obtained based on the first label and the second label, where the first label corresponds to the labeled training image from the source domain, and the second label corresponds to the labeled training image from the target domain, and where the second training data set includes unlabeled training images from the target domain.

[0083] In an embodiment of the present application, before performing a training operation, the training device acquires a first training data set and a second training data set, where the first training data set includes at least one first labeled training image, where the first training image is obtained by mixing a labeled training image from a source domain with a labeled training image from a target domain at a pixel level, and the second training data set includes unlabeled training images from the target domain.

[0084] In one case, the training device stores a large number of labeled training images from a source domain and a small number of labeled training images from a target domain, that is, the training device stores a large number of training images from the source domain and expected results corresponding to the training images from the source domain, and also stores a small number of training images from the target domain and expected results corresponding to the training images from the target domain.

[0085] The training device mixes any labeled training image from the source domain (hereinafter referred to as "source domain image to be mixed") with any labeled training image from the target domain (hereinafter referred to as "target domain image to be mixed") at the pixel level to generate a first training image in the first training data set. In each mixing process, the training device can adopt any selection rule to obtain a source domain image to be mixed from multiple labeled training images from the source domain, and obtain a mixed target domain image from multiple labeled training images from the target domain.

[0086] A mixing process for a source domain image to be mixed and a target domain image to be mixed. Specifically, in one implementation, the training device may use a cutting-mixing (CutMix) technique to mix the source domain image to be mixed and the target domain image to be mixed to generate a new training image (i.e., the first training image), in which the first training image has both a part from the target domain and a part from the source domain. The training device will also mix the first label of the source domain image to be mixed (also referred to as the expected result of the source domain image to be mixed) and the second label of the target domain image to be mixed to obtain the label of the aforementioned first training image.

[0087] Specifically, the training device can cut out a first rectangular area from the source domain image to be mixed, mix the target domain image to be mixed with the first rectangular area to generate a first training image; and perform a mixing operation on the first label and the second label according to the area of ​​the first rectangular area to obtain a label corresponding to the first training image.

[0088] Alternatively, the training device captures a second rectangular area from the target domain image to be mixed, mixes the source domain image to be mixed with the second rectangular area to generate a new first training image; and performs a mixing operation on the first label and the second label according to the area of ​​the second rectangular area to obtain a label corresponding to the first training image.

[0089] To further understand the present solution, an example of a formula for mixing a source domain image to be mixed and a target domain image to be mixed using the CutMix technology is disclosed as follows:

[0090]

[0091]

[0092] Among them, formula (1) and formula (2) take the application scenario of neural network applied to image semantic segmentation as an example. represents a first training image obtained by mixing the source domain image to be mixed after the clipping operation is performed with the second rectangular area, M represents a binary mask used to clip the second rectangular area from the target domain image to be mixed, represents the target domain image to be mixed, represents the source domain image to be mixed, represents the label of the first training image, represents the label corresponding to the target domain image to be mixed, Represents the label corresponding to the source domain image to be mixed. It should be understood that the examples in equations (1) and (2) are only for the convenience of understanding this solution and are not used to limit this solution.

[0093] For a more intuitive understanding of this solution, please refer to Figure 4 , Figure 4 A schematic diagram of a first training image obtained in the neural network training method provided in an embodiment of the present application, Figure 4 In the example, the target domain image to be mixed is mixed with the first rectangular area, such as Figure 4 As shown, a rectangular area (i.e., a first rectangular area) is intercepted from the source domain image to be mixed using a mask, and the target domain image to be mixed is mixed with the first rectangular area to obtain a new training image (i.e., the first training image). It should be noted that the first rectangular area intercepted from the source domain image to be mixed can be placed at any position in the target domain image to be mixed, and the placement position of the first rectangular area does not need to correspond to the interception position of the first rectangular area. Figure 4 This is only an example to facilitate understanding of the present invention and is not intended to limit the present invention.

[0094] In another implementation, the training device may use a mixing (Mixup) technology to mix the labeled source domain image to be mixed and the labeled target domain image to be mixed. More specifically, the training device assigns a first weight to the source domain image to be mixed, assigns a second weight to the target domain image to be mixed, and performs a weighted summation on the source domain image to be mixed and the target domain image to be mixed, that is, performs a weighted summation on each pixel point in the source domain image to be mixed and each pixel point in the target domain image to be mixed.

[0095] Correspondingly, the training device assigns a first weight to the first label corresponding to the source domain image to be mixed, assigns a second weight to the second label corresponding to the target domain image to be mixed, and performs weighted summation of the first label and the second label to obtain the label of the first training image (which may also be referred to as the label corresponding to the first training image, or the expected result of the first training image). It should be noted that the training device may also use other methods to implement the mixing operation of the source domain image to be mixed and the target domain image to be mixed at the pixel level, which are not exhaustive here.

[0096] In another case, the training device may not perform the above mixing operation, but directly receive the first training data set from other electronic devices.

[0097] 302. The training device trains the first neural network according to the first training data set until a preset condition is met to obtain a first teacher network.

[0098] In an embodiment of the present application, the training device uses a first loss function to train the first neural network according to the training images in the first training data set. Specifically, the training device obtains the first training image from the first training data set, inputs the first training image into the first neural network, obtains the prediction result corresponding to the first training image output by the first neural network, and the training device generates the function value of the first loss function according to the prediction result corresponding to the first training image and the label of the first training image (also referred to as the expected result corresponding to the first training image), performs gradient differentiation on the function value of the first loss function, and performs gradient update on the weight parameters of the first neural network by back propagation to complete one training of the first neural network. The training device can repeat the above training operation according to other training images in the first training data set to implement iterative training of the first neural network until the first preset condition is met and the first teacher network is obtained.

[0099] Among them, the first neural network can be a neural network for image semantic segmentation, or a neural network for image classification, or a neural network for object detection in images, or other types of neural networks, etc., which are not limited here.

[0100] The first loss function indicates the similarity between the prediction result corresponding to the first training image and the label of the first training image. Further, the first loss function may adopt a cross entropy loss function, an L1 loss function, an L2 loss function, a 0-1 loss function or other types of loss functions, etc. The specific type of loss function may also be determined in combination with the function of the first neural network.

[0101] The type of information indicated by the above prediction results and the above labels is related to the function of the first neural network. As an example, if the first neural network is a neural network for image semantic segmentation, then the above prediction results and the above labels are used to indicate the category of each pixel in the image; as another example, if the first neural network is used for image classification, then the above prediction results and the above labels are used to indicate the category of the entire image, etc., which are not exhaustive here.

[0102] The first preset condition may be that the function value of the first loss function satisfies the preset value condition, or the first preset condition may be that the number of iterations of the first neural network is greater than or equal to a preset number threshold. The first teacher network is a first neural network that has performed a training operation, or the first teacher network is a mature first neural network.

[0103] To further understand the present solution, an example of the first loss function is disclosed as follows:

[0104]

[0105] in, represents the first loss function, represents a first training image in the first training dataset, represents the prediction result corresponding to the first training image generated by the first neural network, represents the label of the first training image, Representative calculation and It should be understood that the example in formula (3) is only for the convenience of understanding this solution and is not used to limit this solution.

[0106] For a more intuitive understanding of this solution, please refer to Figure 5 , Figure 5 A schematic diagram of a process for training a first neural network in a neural network training method provided in an embodiment of the present application, such as Figure 5 As shown, the training device inputs the first training image into the first neural network to output a prediction result corresponding to the first training image through the first neural network, generates a function value of a first loss function according to the prediction result corresponding to the first training image and a label of the first training image (also referred to as a label corresponding to the first training image, or referred to as an expected result of the first training image), and reversely updates the weight parameter of the first neural network according to the function value of the first loss function to complete one training of the first neural network. It should be understood that Figure 5 This is only an example to facilitate understanding of the present invention and is not intended to limit the present invention.

[0107] 303. The training device trains the first neural network according to a third training data set until a preset condition is met to obtain a second teacher network, wherein the third training data set includes labeled training images from the source domain and labeled training images from the target domain.

[0108] In some embodiments of the present application, the training device can also obtain a third training image from a third training data set, use a second loss function, and train the first neural network according to the third training image until a preset condition is met to obtain a second teacher network, where the second teacher network is the first neural network that has performed the training operation. The specific implementation of step 303 can be found in the description of step 302 above, which will not be repeated here.

[0109] The third training data set includes labeled training images from the source domain and labeled training images from the target domain, that is, a third training image can be a training image from the source domain, or it can also be a training image from the target domain. The second loss function indicates the similarity between the prediction result of the third training image and the label of the third training image (also called the expected result). The meanings of the terms such as the training image from the source domain, the training image from the target domain, and the prediction result, the type of information specifically indicated by the label, and the type of the second loss function can all be referred to the above description, which will not be repeated here.

[0110] To further understand the present solution, an example of the second loss function is disclosed as follows:

[0111]

[0112] in, represents the second loss function, represents the prediction result obtained by inputting a third training image from the source domain into the first neural network, represents the label corresponding to a third training image from the source domain, Representative calculation and The cross entropy loss between represents the prediction result obtained by inputting a third training image from the target domain into the first neural network, represents the label corresponding to a third training image from the target domain, Representative calculation and It should be understood that the example in formula (4) is only for the convenience of understanding this solution and is not used to limit this solution.

[0113] Regarding the method for obtaining the third training data set, in one case, a large number of labeled training images from the source domain and a small number of labeled training images from the target domain may be pre-stored in the training device, and the training device directly mixes the multiple labeled training images from the source domain with the multiple labeled training images from the target domain at the image level to obtain the third training data set. In another case, the training device may receive the third training data set sent by other devices.

[0114] For a more intuitive understanding of this solution, please refer to Figure 6 , Figure 6 A schematic diagram of a process for training a first neural network and a second neural network in a neural network training method provided in an embodiment of the present application, Figure 6 Can be combined Figure 4 as well as Figure 5 The training device mixes the target domain image to be mixed with the source domain image to be mixed at the pixel level to obtain a first training image. The specific implementation of the above steps can refer to the above Figure 4 The training device trains the first neural network using the first training image. The specific implementation of the above steps can refer to the above Figure 5 The training device mixes the target domain image to be mixed with the source domain image to be mixed at the image level to obtain a third training data set. The training device obtains a training image from the target domain and a training image from the source domain from the third training data set, and inputs the two training images into the first neural network to process the two training images respectively through the first training image to obtain Figure 6 A2 in which A2 represents the prediction result corresponding to the training image from the target domain and the prediction result corresponding to the training image from the source domain. The training device generates the function value of the second loss function according to A1 and A2, and uses the function value of the second loss function to reversely update the weight parameter of the first neural network. A1 represents the label corresponding to the training image from the target domain and the label corresponding to the training image from the source domain. It should be understood that Figure 6 This is only an example to facilitate understanding of the present invention and is not intended to limit the present invention.

[0115] 304. The training device trains the first student network by using knowledge distillation according to the first teacher network and the second training data set.

[0116] In an embodiment of the present application, the training device trains the first student network according to the second training data set by using knowledge distillation. The second training data set includes a plurality of unlabeled second training images from the target domain, that is, the second training data set only contains a plurality of second training images from the target domain, but does not contain the expected results corresponding to each second training image. Knowledge distillation refers to the training goal in the process of training the student network to narrow the similarity between the prediction results output by the student network and the prediction results output by the teacher network, so that the student network can learn the image processing capabilities of the teacher network.

[0117] The first student network has the same function as the first neural network. As an example, if the first neural network is a neural network for image semantic segmentation, then the first student network is also a neural network for image semantic segmentation. As another example, if the first neural network is a neural network for image classification, then the first student network is also a neural network for image classification, etc., which are not exhaustive here. Furthermore, in one implementation, the first student network and the first neural network are the same neural network. In another implementation, the first student network and the first neural network may also be different neural networks. For example, the first neural network may be a neural network with a more complex structure, or the first neural network may be a larger neural network, etc.

[0118] Step 303 is an optional step. If step 303 is performed, step 304 may include: the training device iteratively trains the first student network according to the first teacher network, the second teacher network and the second training data set by using knowledge distillation. In the embodiment of the present application, the labeled training images from the source domain and the labeled training images from the target domain are directly mixed at the image level to obtain a third training data set, and the second teacher network is trained according to the third training data set. Since the first training image breaks the data gap between the target domain image and the source domain image at the pixel level, the difficulty of the first teacher network in the training process is increased, resulting in the first teacher network classifying objects without regular shapes in the target domain (such as the sky, wall, sea, etc.) more accurately; the second teacher network has a clearer overall information of the target domain image, and the objects that need to be predicted by shape in the target domain (such as street lights, traffic lights, etc.) are more accurately classified. The prediction results output by the first teacher network and the second teacher network complement each other, and the prediction results of the first teacher network and the second teacher network are combined as the learning target of the student network, which is conducive to further improving the accuracy of the second student network in processing images in the target domain.

[0119] The second training data set includes multiple second training images, and the training device can use the multiple second training images to perform multiple trainings on the first student network for any of the aforementioned multiple trainings. Specifically, the training device obtains the second training image from the second training data set, and the second training image is any training image in the second training data set. The training device uses the first teacher network to process the second training image to generate a first prediction result corresponding to the second training image; and uses the second teacher network to process the second training image to generate a second prediction result corresponding to the second training image. The training device uses the first student network to process the second training image to generate a third prediction result corresponding to the second training image. The training device trains the first student network based on the first prediction result, the second prediction result and the third prediction result by using knowledge distillation.

[0120] More specifically, in one implementation, after obtaining the first prediction result and the second prediction result, the training device may perform a weighted sum of the first prediction result and the second prediction result to obtain a target prediction result corresponding to the second training image. Optionally, the weights of the first prediction result and the second prediction result may be 0.5 and 0.5, respectively, that is, the training device may average the first prediction result and the second prediction result to obtain the target prediction result.

[0121] The training device calculates the function value of the target loss function according to the third prediction result and the target prediction result, and performs gradient update on the weight parameters of the first student network by back propagation to complete a training of the first student network. Among them, the target loss function indicates the similarity between the third prediction result and the target prediction result. The goal of the training device to iteratively train the first student network is to narrow the similarity between the third prediction result and the target prediction result. For the description of the function type used by the target loss function, please refer to the above description of the function type of the first loss function, which will not be repeated here. The type of information indicated by the third prediction result and the target prediction result is related to the function of the first student network. For specific examples, please refer to the description in the above step 302, which will not be repeated here.

[0122] In an embodiment of the present application, when the first student network is trained by using knowledge distillation based on the first teacher network, the second teacher network and the second training data set, the first prediction result and the second prediction result are weightedly summed to obtain a target prediction result corresponding to the second training image, that is, the learning target of the first student network. The operation is simple and easy to achieve. Moreover, since the outputs of the first teacher network and the second teacher network are complementary, the weighted summation method is conducive to bringing out the advantages brought by complementarity and is conducive to further improving the accuracy of the learning target of the first student network.

[0123] In another implementation, after the training device obtains the first prediction result and the second prediction result, if the first neural network is used for image semantic segmentation, the first prediction result and the second prediction result are both used to indicate the prediction category of each pixel in a second training image, and the training device can calculate the entropy value of the first prediction result and the entropy value of the second prediction result, select a prediction result with a smaller entropy value from the first prediction result and the second prediction result as the target prediction result, and then calculate the function value of the target loss function according to the third prediction result and the target prediction result, and perform gradient update on the weight parameters of the first student network by back propagation to complete a training of the first student network. The training device can also use other methods to generate the target prediction result, and the method for generating the target prediction result is not limited here.

[0124] If step 303 is not performed, step 304 may include: the training device iteratively trains the first student network according to the first teacher network and the second training data set by using knowledge distillation until a preset condition is met to obtain a second student network.

[0125] Specifically, the training device obtains a second training image from a second training data set. The training device processes the second training image using the first teacher network to generate a first prediction result corresponding to the second training image; processes the second training image using the first student network to generate a third prediction result corresponding to the second training image, and the training device generates a function value of a third loss function according to the first prediction result and the third prediction result, and reversely updates the weight parameter of the first student network according to the function value of the third loss function to complete a training of the first student network, and the third loss function indicates the similarity between the third prediction result and the first prediction result. The training device repeatedly performs the aforementioned operations to implement iterative training of the first student network.

[0126] 305. The training device trains the first student network according to a fourth training data set, where the fourth training data set includes labeled training images from the target domain.

[0127] In some embodiments of the present application, the training device also iteratively trains the first student network using a fourth loss function based on a fourth training data set, where the fourth training data set includes fourth training images from the target domain and labels corresponding to each fourth training image (also called expected results).

[0128] Specifically, in one iteration, the training device inputs the fourth training image into the first student network, obtains the fourth prediction result corresponding to the fourth training image output by the first student network, generates the function value of the fourth loss function according to the prediction result corresponding to the fourth training image and the label corresponding to the fourth training image, and reversely updates the weight parameters of the first student network according to the function value of the fourth loss function to complete one training of the first student network. The fourth loss function indicates the similarity between the label corresponding to the fourth training image and the fourth prediction result.

[0129] It should be noted that step 305 is an optional step. If step 305 is executed, the embodiment of the present application does not limit the execution order between step 305 and step 304. Step 304 and step 305 can be executed alternately or simultaneously. The condition for the training device to stop iterative training may be that the sum of the target loss function (or the third loss function) and the fourth loss function meets the preset value condition, or the total number of iterative training times is greater than the preset threshold value, so as to obtain the second student network (that is, the mature first student network).

[0130] To further understand the present solution, an example of the target loss function and the fourth loss function is disclosed as follows:

[0131]

[0132] in, Represents the target loss function. Here, the target loss function uses the KL divergence loss function as an example. Represents the fourth loss function. Here, the fourth loss function adopts the cross entropy loss function as an example. kl represents the weight of the objective loss function, λ ce represents the weight of the fourth loss function, λ kl and λ ce are two hyperparameters, represents unlabeled training images from the target domain, and Represent the first prediction result and the second prediction result respectively, represents the target prediction result after averaging the first prediction result and the second prediction result, Represents the third prediction result. The meanings of other letters in formula (5) can be found in the above description. It should be understood that the examples in formula (5) are only for the convenience of understanding this solution and are not used to limit this solution.

[0133] For a more intuitive understanding of this solution, please refer to Figure 7 , Figure 7 A schematic diagram of a flow chart of training a first student network in a neural network training method provided in an embodiment of the present application, represents a labeled training image from the target domain (i.e., the fourth training image), represents the unlabeled training image from the target domain (i.e., the second training image), represents the label corresponding to the fourth training image, represents the function value of the target loss function, Represents the function value of the fourth loss function. Figure 7 As shown, the training device obtains a second training image and a fourth training image, and the training device inputs the second training image into the first teacher network, the second teacher network and the first student network respectively, processes the second training image through the first teacher network to output a first prediction result, processes the second training image through the second teacher network to output a second prediction result, and outputs a third prediction result (that is, an example of B1) corresponding to the second training image through the first student network. The training device averages the first prediction result and the second prediction result to obtain a target prediction result. The training device generates according to the third prediction result and the target prediction result. The value of .

[0134] The training device also inputs the fourth training image into the first student network to obtain a fourth prediction result output by the first student network, and the training device generates a label corresponding to the fourth training image according to the fourth prediction result and the label corresponding to the fourth training image. The value of The value and The value of generates the function value of the total loss function, and the weight parameters of the first student network are reversely updated according to the function value of the total loss function. It should be understood that Figure 7 This is only an example to facilitate understanding of the present invention and is not intended to limit the present invention.

[0135] In an embodiment of the present application, the first student network is also trained using labeled training images from the target domain to further improve the processing capability of the second student network for the images of the target domain, so as to improve the performance of the second student network in processing the images of the target domain.

[0136] 306. The training device processes the second training image in the second training data set through the second student network to generate a prediction result corresponding to the second training image, and determines a label of the second training image according to the prediction result corresponding to the second training image.

[0137] In some embodiments of the present application, after obtaining the second student network, the training device can also process each second training image in the second training data set to generate a prediction result corresponding to each second training image, and determine the label of the second training image according to the prediction result corresponding to the second training image (also referred to as the label corresponding to the second training image, or the expected result of the second training image), that is, a corresponding pseudo label is generated for each second training image through the second student network. The prediction result corresponding to the second training image and the label of the second training image have different specific information representation forms. As an example, for example, the second student network is applied to an application scenario of image semantic segmentation, then the prediction results and labels corresponding to the second training image both indicate the predicted category of each pixel in the image, and the prediction result corresponding to a pixel includes N probability values, and the aforementioned N probability values ​​indicate the probability that the pixel value belongs to each of the N categories; the label corresponding to a pixel includes N values ​​corresponding to N categories, and the aforementioned N values ​​include a 1 and N-1 0s, and the category of the pixel is the category corresponding to the value of 1; it should be understood that when the second student network is applied to other application scenarios, the prediction results and labels may be expressed in other forms, which are not enumerated here.

[0138] 307. The training device performs a pixel-level mixing operation based on the labeled training image from the source domain, the labeled training image from the target domain, and the labeled second training image to obtain an updated first training data set.

[0139] In some embodiments of the present application, the training device is configured with labeled training images from the source domain and labeled training images from the target domain. After obtaining the label corresponding to each second training image (i.e., obtaining a large number of labeled training images from the target domain), the training device can perform a pixel-level mixing operation on the labeled training images from the source domain and the labeled training images from the target domain to obtain an updated first training data set, and the updated first training data set includes multiple first training images. Among them, the first training image is obtained by mixing the labeled training image from the source domain with the labeled training image from the target domain at the pixel level, and the label of the first training image is obtained based on the first label and the second label, the first label corresponds to the labeled training image from the source domain, and the second label corresponds to the labeled training image from the target domain.

[0140] For the specific implementation method of mixing the labeled training images from the source domain and the labeled training images from the target domain at the pixel level, please refer to the description in the above step 301, which will not be repeated here.

[0141] 308. The training device trains the first neural network according to the updated first training data set until a preset condition is met to obtain an updated first teacher network.

[0142] In the embodiment of the present application, the specific implementation method of step 308 can refer to the description in step 302 and will not be repeated here. The only difference is that the updated first training data set used in step 308 contains more training data.

[0143] 309. The training device performs an image-level mixing operation based on the labeled training images from the source domain, the labeled training images from the target domain, and the labeled second training images to obtain an updated third training data set.

[0144] In some embodiments of the present application, the training device is configured with labeled training images from a source domain and labeled training images from a target domain. After obtaining the label corresponding to each second training image (that is, a large number of labeled training images from the target domain are obtained), the training device can perform an image-level mixing operation on the labeled training images from the source domain and the labeled training images from the target domain to obtain an updated third training data set. For the specific implementation method of performing image-level mixing on the labeled training images from the source domain and the labeled training images from the target domain, please refer to the description in the above step 303, which will not be repeated here.

[0145] 310. The training device trains the first neural network according to the updated third training data set until a preset condition is met to obtain an updated second teacher network.

[0146] In the embodiment of the present application, the specific implementation method of step 310 can refer to the description in step 303, which will not be repeated here. The only difference is that the updated third training data set used in step 310 contains more training data.

[0147] 311. The training device trains the first student network according to the updated first teacher network and the second training data set by using knowledge distillation until a preset condition is met to obtain an updated second student network.

[0148] In the embodiment of the present application, the specific implementation of step 311 can refer to the description in step 304, which is not repeated here. The difference is that the updated first teacher network is used in step 311, or the updated first teacher network and the updated second teacher network are used.

[0149] 312. The training device determines whether the loop termination condition is met. If the loop termination condition is not met, the training device re-enters step 306. If the loop termination condition is met, the final second student network is obtained.

[0150] In the embodiment of the present application, after executing step 311, the training device can determine whether the loop termination condition is met. The loop termination condition can be whether the number of executions of steps 306 to 312 reaches a preset number of times. The value of the preset number of times can be 1 time, 2 times, 3 times, 4 times, 5 times or other values, etc. If the loop termination condition is not met, step 306 is re-entered, that is, after using the updated second student network to generate labels for each second training image again, steps 307 to 311 are executed again, and step 312 is executed again. If the loop termination condition is met, the updated second student network obtained by step 311 is used as the final second student network.

[0151] For a more intuitive understanding of this solution, please refer to Figure 8 , Figure 8 A flowchart of a neural network training method provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, a large amount of labeled data from the source domain, a small amount of labeled data from the target domain, and a large amount of unlabeled data from the target domain are stored on the training device. The labeled data from the source domain includes labeled training images from the source domain, the labeled data from the target domain includes labeled training images from the target domain, and the unlabeled data from the target domain includes unlabeled training images from the target domain. The dual domain mixing module 101 is used to perform a mixing operation on the labeled training images from the source domain and the labeled training images from the target domain at the pixel level to obtain a first training data set; the dual domain mixing module 101 is also used to perform a mixing operation on the labeled training images from the source domain and the labeled training images from the target domain at the image level to obtain a third training data set. The dual domain mixing module 101 is also used to train the first neural network using the first training data set to obtain a first teacher network, and the dual domain mixing module 101 is also used to train the first neural network using the third training data set to obtain a second teacher network.

[0152] The multi-teacher distillation module 102 is used to train the first student network by using knowledge distillation based on the first teacher network, the second teacher network and a large number of unlabeled training images from the target domain (i.e., the second training data set); the multi-teacher distillation module 102 is also used to train the first student network using a small number of labeled training images from the target domain to obtain a second student network.

[0153] The self-training module 103 uses the second student network to train unlabeled training images (corresponding to Figure 8 The target domain has a large amount of unlabeled data in the target domain) to generate pseudo labels, and add the training images with pseudo labels from the target domain to the labeled training images from the target domain (corresponding to Figure 8 The dual domain mixing module 101 uses the updated labeled data of the target domain and the labeled data from the target domain to repeatedly perform the above operations, that is, perform mixing operations at the pixel level and at the image level, respectively, and obtain an updated first teacher network and an updated second teacher network. Then, the multi-teacher distillation module 102 repeatedly performs the above operations based on the updated first teacher network and the updated second teacher network to obtain an updated second teacher network. It should be understood that Figure 8 This is only an example to facilitate understanding of the present invention and is not intended to limit the present invention.

[0154] In the embodiment of the present application, after the training device obtains the second student network through initial training, it will also use the second student network to generate pseudo labels for the second training images, that is, generate pseudo labels for a large number of unlabeled training images from the target domain, and based on a large number of labeled second training images from the target domain, perform image-level mixing operations on the labeled training images from the source domain and the labeled training images from the target domain again to obtain an updated first training data set. Since more training data from the target domain is introduced into the updated first training data set, the updated first teacher network will perform better when processing images from the target domain, and then the updated second student network will also perform better when processing images from the target domain, which is conducive to obtaining a more accurate second student network.

[0155] It should be noted that steps 306 to 312 are optional steps. If steps 306 to 312 are not performed, the final second student network can be obtained after step 305 is performed.

[0156] In an embodiment of the present application, a teacher network is trained using a first training data set, and then a student network is trained using the teacher network and the second training data set by adopting a knowledge distillation method. Since the training images in the first training data set are obtained by mixing labeled training images from the source domain with labeled training images from the target domain at the pixel level, the difficulty of the teacher network in the training process is increased, thereby improving the processing capability of the trained teacher network for images of the target domain, that is, improving the generalization capability of the trained teacher network, so that the teacher network can generate prediction results with better performance for unlabeled training images from the target domain, that is, the quality of the training data of the student network is higher, which is conducive to improving the performance of the trained student network in processing images of the target domain.

[0157] 2. Reasoning Stage

[0158] In the embodiment of the present application, the reasoning stage describes how the execution device 210 uses the mature second target model / rule 202 to perform image processing. For details, please refer to Fig. 9 , Fig. 9 A schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application. The image processing method provided in an embodiment of the present application may include:

[0159] 901. An execution device obtains an image to be processed from a target domain.

[0160] 902. The execution device inputs the image to be processed from the target domain into the second student network to process the image to be processed through the second student network to obtain a prediction result output by the second student network.

[0161] In the embodiments of the present application, the concepts of the target domain to be processed image, the second student network, the prediction result output by the second student network, etc. can all be referred to in Figure 3 In the corresponding embodiment, the second student network is Figure 3 The corresponding embodiment is trained.

[0162] Among them, the second student network is trained according to the first teacher network and the second training data set by using knowledge distillation, the first teacher network is trained according to the first training data set, the first training data set includes at least one labeled first training image, the first training image is obtained by mixing the labeled training image from the source domain with the labeled training image from the target domain at the pixel level, the label of the first training image is obtained according to the first label and the second label, the first label corresponds to the labeled training image from the source domain, the second label corresponds to the labeled training image from the target domain, and the second training data set includes unlabeled training images from the target domain.

[0163] Optionally, the second student network is trained by knowledge distillation based on the first teacher network, the second teacher network and the second training data set, and the second teacher network is trained based on a third training data set. The third training data set includes labeled training images from the source domain and labeled training images from the target domain. The third training data set is obtained by mixing the labeled training images from the source domain and the labeled training images from the target domain at the image level.

[0164] In an embodiment of the present application, an implementation method of the second student network in the reasoning stage is provided, which expands the application scenario of the present solution; since the training images in the first training data set are obtained by mixing labeled training images from the source domain with labeled training images from the target domain at the pixel level, the difficulty of the teacher network in the training process is increased, thereby improving the processing capability of the trained teacher network for images of the target domain, that is, improving the generalization capability of the trained teacher network, so that the teacher network can generate prediction results with better performance for unlabeled training images from the target domain, that is, the quality of the training data of the student network is higher, which is conducive to improving the performance of the trained student network in processing images of the target domain.

[0165] In order to have a more intuitive understanding of the beneficial effects brought about by the embodiments of the present application, the following is explained in combination with actual experimental data. Here, the images in the source domain come from the data set GTA5, and the images in the target domain come from the data set Cityscapes. Please refer to the following Table 1. In Table 1, the measurement indicator mIoU is used as an example. mIoU represents the accuracy of the trained neural network in processing the images in the target domain. The larger the value of this indicator, the better the performance of the trained neural network.

[0166] Table 1

[0167] 100 200 500 1000 2975 Baseline 52.6％ 53.6％ 58.4％ 61.6％ 66.6％ ASS 54.2％ 56.0％ 60.2％ 64.5％ 69.1％ Embodiments of the present application 61.2％ 60.5％ 64.3％ 66.6％ 69.8％

[0168] Among them, Baseline and ASS are two existing methods for training the first student network. In Table 1, 100, 200, 500, 1000 and 2975 respectively represent the number of labeled training images from the target domain used in the training data. From the above Table 1, no matter how many labeled training images from the target domain are used, the performance of the trained neural network obtained by the training method provided in the embodiment of the present application is better.

[0169] exist Figures 1 to 9 On the basis of the corresponding embodiments, in order to better implement the above solutions of the embodiments of the present application, the following also provides related devices for implementing the above solutions. Fig.10 , Fig.10 A structural schematic diagram of a neural network training device provided in an embodiment of the present application, the neural network training device 1000 includes: an acquisition module 1001, used to acquire a first training data set and a second training data set, wherein the first training data set includes at least one first labeled training image, the first training image is obtained by mixing a labeled training image from a source domain with a labeled training image from a target domain at a pixel level, the label of the first training image is obtained according to the first label and the second label, the first label corresponds to the labeled training image from the source domain, the second label corresponds to the labeled training image from the target domain, and the second training data set includes unlabeled training images from the target domain; a training module 1002, used to train the first neural network according to the first training data set until a preset condition is met to obtain a first teacher network, the first teacher network is the first neural network that has performed a training operation; the training module 1002 is also used to train the first student network according to the first teacher network and the second training data set by using knowledge distillation until a preset condition is met to obtain a second student network, the second student network is the first student network that has performed a training operation.

[0170] In one possible design, the training module 1002 is further used to train the first neural network according to a third training data set until a preset condition is met to obtain a second teacher network, wherein the third training data set includes labeled training images from a source domain and labeled training images from a target domain, and the second teacher network is the first neural network that has performed a training operation; the training module 1002 is specifically used to train the first student network according to the first teacher network, the second teacher network and the second training data set by using knowledge distillation until a preset condition is met to obtain a second student network.

[0171] In one possible design, see Fig.11 , Fig.11A structural schematic diagram of a neural network training device provided in an embodiment of the present application, the training module 1002 includes: an acquisition submodule 10021, used to acquire a second training image, the second training image is any one of the training images in the second training data set; a processing submodule 10022, used to process the second training image using the first teacher network to generate a first prediction result corresponding to the second training image; the processing submodule 10022 is also used to process the second training image using the second teacher network to generate a second prediction result corresponding to the second training image; a summation submodule 10023 is used to perform weighted summation of the first prediction result and the second prediction result to obtain a target prediction result corresponding to the second training image; the processing submodule 10022 is also used to process the second training image using the first student network to generate a third prediction result corresponding to the second training image; a training submodule 10024 is used to train the first student network according to a target loss function, the target loss function indicating the similarity between the third prediction result and the target prediction result.

[0172] In one possible design, see Fig.11 The neural network training device 1000 also includes: a processing module 1003, which is used to process the second training image in the second training data set through the second student network to generate a prediction result corresponding to the second training image, and determine the label of the second training image according to the prediction result corresponding to the second training image; a mixing module 1004, which is used to perform a pixel-level mixing operation according to the labeled training image from the source domain, the labeled training image from the target domain and the labeled second training image to obtain an updated first training data set; the training module 1002 is also used to train the first neural network according to the updated first training data set until a preset condition is met to obtain an updated first teacher network; the training module 1002 is also used to train the first student network according to the updated first teacher network and the second training data set by using a knowledge distillation method until a preset condition is met to obtain an updated second student network, and the updated second student network is the first student network that has performed the training operation.

[0173] In one possible design, the training module 1002 is specifically used to train the first student network based on the first teacher network, the second teacher network and the second training data set by using knowledge distillation, and to train the first student network based on the fourth training data set, wherein the fourth training data set includes labeled training images from the target domain.

[0174] In one possible design, the training images of the source domain and the training images of the target domain come from different image generating devices, or the training images of the source domain and the training images of the target domain come from different image acquisition locations, or the image acquisition time of the training images of the source domain and the image acquisition time of the training images of the target domain are different, or the photographed objects in the training images of the source domain and the photographed objects in the training images of the target domain are different object types.

[0175] In one possible design, both the first neural network and the first student network are used to perform the image semantic segmentation task.

[0176] It should be noted that the information interaction, execution process, etc. between the modules / units in the neural network training device 1000 are the same as those in the present application. Figures 3 to 8 The corresponding method embodiments are based on the same concept, and the specific contents can be found in the description of the method embodiments shown above in this application, which will not be repeated here.

[0177] The present application also provides an image processing device. Fig.12 , Fig.12 A structural schematic diagram of an image processing device provided in an embodiment of the present application, the image processing device 1200 includes: an acquisition module 1201, used to acquire an image to be processed from a target domain; a processing module 1202, used to input the image to be processed from the target domain into a second student network, so as to process the image to be processed through the second student network, and obtain a prediction result output by the second student network; wherein the second student network is trained by using a knowledge distillation method according to a first teacher network and a second training data set, the first teacher network is trained according to a first training data set, the first training data set includes at least one first labeled training image, the first training image is obtained by mixing a labeled training image from a source domain with a labeled training image from a target domain at a pixel level, the label of the first training image is obtained according to the first label and the second label, the first label corresponds to the labeled training image from the source domain, the second label corresponds to the labeled training image from the target domain, and the second training data set includes unlabeled training images from the target domain.

[0178] In a possible design, the second student network is trained by knowledge distillation based on the first teacher network, the second teacher network and the second training data set, and the second teacher network is trained based on the third training data set. The third training data set includes labeled training images from the source domain and labeled training images from the target domain. The third training data set is obtained by mixing the labeled training images from the source domain and the labeled training images from the target domain at the image level.

[0179] It should be noted that the information interaction, execution process, etc. between the modules / units in the image processing device 1200 are the same as those in the present application. Fig. 9 The corresponding method embodiments are based on the same concept, and the specific contents can be found in the description of the method embodiments shown above in this application, which will not be repeated here.

[0180] Next, an execution device provided by an embodiment of the present application is introduced. Fig.13 , Fig.13 A schematic diagram of the structure of the execution device provided in the embodiment of the present application, the execution device 1300 can be specifically manifested as a virtual reality VR device, a mobile phone, a tablet, a laptop, a smart wearable device, a monitoring data processing device or a radar data processing device, etc., which is not limited here. Among them, the execution device 1300 can be deployed with Fig.12 The image processing device 1200 described in the corresponding embodiment is used to implement Fig. 9 The functions of the execution device in the corresponding embodiment. Specifically, the execution device 1300 includes: a receiver 1301, a transmitter 1302, a processor 1303 and a memory 1304 (wherein the number of the processor 1303 in the execution device 1300 can be one or more, Fig.13 In the example of FIG. 1301 , the processor 1303 may include an application processor 13031 and a communication processor 13032. In some embodiments of the present application, the receiver 1301, the transmitter 1302, the processor 1303 and the memory 1304 may be connected via a bus or other means.

[0181] The memory 1304 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1303. A portion of the memory 1304 may also include a non-volatile random access memory (NVRAM). The memory 1304 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0182] The processor 1303 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0183] The method disclosed in the above embodiment of the present application can be applied to the processor 1303, or implemented by the processor 1303. The processor 1303 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1303. The above processor 1303 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1303 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be combined and executed. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1304, and the processor 1303 reads the information in the memory 1304 and completes the steps of the above method in combination with its hardware.

[0184] The receiver 1301 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1302 can be used to output digital or character information through the first interface; the transmitter 1302 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1302 can also include a display device such as a display screen.

[0185] In the embodiment of the present application, the processor 1303 is used to execute Fig. 9 The specific manner in which the application processor 13031 in the processor 1303 executes the above steps is the same as that in the present application. Fig. 9 The corresponding method embodiments are based on the same concept, and the technical effects they bring are the same as those in this application. Fig. 9 The corresponding method embodiments are the same. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0186] The present application also provides a training device. Fig.14, Fig.14 is a schematic diagram of a structure of a training device provided in an embodiment of the present application. The training device 1400 may be deployed with Fig.10 or Fig.11 The training device 1000 of the neural network described in the corresponding embodiment is used to implement Figures 3 to 8 The functions of the training device in the corresponding embodiment, specifically, the training device 1400 is implemented by one or more servers, and the training device 1400 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1422 (for example, one or more processors) and memory 1432, one or more storage media 1430 (for example, one or more mass storage devices) storing application programs 1442 or data 1444. Among them, the memory 1432 and the storage medium 1430 can be short-term storage or permanent storage. The program stored in the storage medium 1430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processing unit 1422 can be configured to communicate with the storage medium 1430 to execute a series of instruction operations in the storage medium 1430 on the training device 1400.

[0187] The training device 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458, and / or, one or more operating systems 1441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0188] In the embodiment of the present application, the central processor 1422 is used to execute Figures 3 to 8 The specific manner in which the central processor 1422 performs the above steps is the same as that in the present application. Figures 3 to 8 The corresponding method embodiments are based on the same concept, and the technical effects they bring are the same as those in this application. Figures 3 to 8 The corresponding method embodiments are the same. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0189] The present application also provides a computer program product, which includes a computer program. When the computer program is run on a computer, the computer executes the above Fig. 9 The method described in the embodiment shown in the figure executes the steps executed by the device, or causes the computer to execute the above-mentioned Figures 3 to 8The illustrated embodiment describes the steps performed by the training device in the method.

[0190] The present application also provides a computer-readable storage medium in which a computer program is stored. When the computer program is run on a computer, the computer executes the above-mentioned Fig. 9 The method described in the embodiment shown in the figure executes the steps executed by the device, or causes the computer to execute the above-mentioned Figures 3 to 8 The illustrated embodiment describes the steps performed by the training device in the method.

[0191] The image processing device, neural network training device, execution device and training device provided in the embodiments of the present application may be specifically a chip, and the chip includes: a processing unit and a communication unit, the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip executes the above Fig. 9 The image processing method described in the embodiment shown in the figure, or, to enable the chip to perform the above Figures 3 to 8 The training method of the neural network described in the illustrated embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0192] For details, please refer to Fig.15 , Fig.15 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 150, NPU 150 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 1503, which is controlled by the controller 1504 to extract matrix data in the memory and perform multiplication operations.

[0193] In some implementations, the operation circuit 1503 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 1503 is a two-dimensional systolic array. The operation circuit 1503 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1503 is a general-purpose matrix processor.

[0194] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1502 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1501 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 1508.

[0195] The unified memory 1506 is used to store input data and output data. The weight data is directly transferred to the weight memory 1502 through the direct memory access controller (DMAC) 1505. The input data is also transferred to the unified memory 1506 through the DMAC.

[0196] BIU stands for Bus Interface Unit 1510 , which is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1509 .

[0197] The bus interface unit 1510 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 1509 to obtain instructions from the external memory, and is also used for the storage unit access controller 1505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0198] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1506 or to transfer weight data to the weight memory 1502 or to transfer input data to the input memory 1501.

[0199] The vector calculation unit 1507 includes multiple operation processing units, and further processes the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0200] In some implementations, the vector calculation unit 1507 can store the processed output vector to the unified memory 1506. For example, the vector calculation unit 1507 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1503, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1507 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1503, for example, for use in a subsequent layer in a neural network.

[0201] An instruction fetch buffer 1509 connected to the controller 1504 is used to store instructions used by the controller 1504;

[0202] Unified memory 1506, input memory 1501, weight memory 1502 and instruction fetch memory 1509 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0203] Wherein, if the chip is configured in a training device, then Figures 3 to 8 The operations of each layer in each neural network shown can be performed by the operation circuit 1503 or the vector calculation unit 1507, or, if the chip is configured in an execution device, Fig. 9 The operations of each layer in the second student network shown can be performed by the operation circuit 1503 or the vector calculation unit 1507.

[0204] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above-mentioned first aspect method.

[0205] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0206] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. According to such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0207] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0208] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A neural network training method, characterized in that: The method comprises: Acquire a first training data set and a second training data set, wherein the first training data set includes at least one first labeled training image, the first training image is obtained by mixing a labeled training image from a source domain with a labeled training image from a target domain at a pixel level, the label of the first training image is obtained according to the first label and the second label, the first label corresponds to the labeled training image from the source domain, the second label corresponds to the labeled training image from the target domain, and the second training data set includes unlabeled training images from the target domain; Training the first neural network according to the first training data set until a preset condition is met to obtain a first teacher network, wherein the first teacher network is the first neural network that has performed a training operation; According to the first teacher network and the second training data set, the first student network is trained by knowledge distillation until a preset condition is met to obtain a second student network, where the second student network is the first student network that has been trained.

2. The method according to claim 1, characterized in that The method further comprises: Training the first neural network according to a third training data set until a preset condition is met to obtain a second teacher network, wherein the third training data set includes the labeled training images from the source domain and the labeled training images from the target domain, and the second teacher network is the first neural network that has performed a training operation; The method of training the first student network by knowledge distillation according to the first teacher network and the second training data set until a preset condition is met to obtain a second student network includes: According to the first teacher network, the second teacher network and the second training data set, the first student network is trained by knowledge distillation until a preset condition is met to obtain the second student network.

3. The method according to claim 2, characterized in that The step of training the first student network by using knowledge distillation according to the first teacher network, the second teacher network and the second training data set includes: Acquire a second training image, where the second training image is any training image in the second training data set; Processing the second training image using the first teacher network to generate a first prediction result corresponding to the second training image; Processing the second training image using the second teacher network to generate a second prediction result corresponding to the second training image, and performing a weighted summation of the first prediction result and the second prediction result to obtain a target prediction result corresponding to the second training image; Processing the second training image using the first student network to generate a third prediction result corresponding to the second training image; The first student network is trained according to a target loss function, wherein the target loss function indicates a similarity between the third prediction result and the target prediction result.

4. The method according to any one of claims 1 to 3, characterized in that: After obtaining the second student network, the method further includes: Processing a second training image in the second training data set by the second student network to generate a prediction result corresponding to the second training image, and determining a label of the second training image according to the prediction result corresponding to the second training image; Performing a pixel-level blending operation according to the labeled training image from the source domain, the labeled training image from the target domain, and the second labeled training image to obtain an updated first training data set; Training the first neural network according to the updated first training data set until a preset condition is met to obtain an updated first teacher network; According to the updated first teacher network and the second training data set, the first student network is trained by knowledge distillation until a preset condition is met to obtain an updated second student network, wherein the updated second student network is the first student network that has been trained.

5. The method according to claim 2 or 3, characterized in that: The step of training the first student network by using knowledge distillation according to the first teacher network, the second teacher network and the second training data set includes: The first student network is trained by knowledge distillation according to the first teacher network, the second teacher network and the second training data set, and the first student network is trained according to a fourth training data set, wherein the fourth training data set includes the labeled training images from the target domain.

6. The method according to any one of claims 1 to 3, characterized in that: The training images of the source domain and the training images of the target domain are derived from different image generating devices, or the training images of the source domain and the training images of the target domain are derived from different image acquisition locations, or the image acquisition time of the training images of the source domain and the image acquisition time of the training images of the target domain are different, or the photographed objects in the training images of the source domain and the photographed objects in the training images of the target domain are different object types.

7. The method according to any one of claims 1 to 3, characterized in that: The first neural network and the first student network are both used to perform any one of the following tasks: image semantic segmentation, image classification, or object detection on an image.

8. An image processing method, characterized in that: The method comprises: Obtain an image to be processed from the target domain; Inputting the image to be processed from the target domain into a second student network, so as to process the image to be processed by the second student network and obtain a prediction result output by the second student network; Among them, the second student network is trained by knowledge distillation based on the first teacher network and the second training data set, the first teacher network is trained based on the first training data set, the first training data set includes at least one labeled first training image, the first training image is obtained by mixing the labeled training image from the source domain with the labeled training image from the target domain at the pixel level, the label of the first training image is obtained based on the first label and the second label, the first label corresponds to the labeled training image from the source domain, the second label corresponds to the labeled training image from the target domain, and the second training data set includes unlabeled training images from the target domain.

9. The method according to claim 8, characterized in that The second student network is trained by knowledge distillation based on the first teacher network, the second teacher network and the second training data set, and the second teacher network is trained based on a third training data set, and the third training data set includes the labeled training images from the source domain and the labeled training images from the target domain.

10. A neural network training device, characterized in that: The device comprises: an acquisition module, configured to acquire a first training data set and a second training data set, wherein the first training data set includes at least one first labeled training image, the first training image is obtained by mixing a labeled training image from a source domain with a labeled training image from a target domain at a pixel level, the label of the first training image is obtained according to the first label and the second label, the first label corresponds to the labeled training image from the source domain, the second label corresponds to the labeled training image from the target domain, and the second training data set includes unlabeled training images from the target domain; A training module, configured to train the first neural network according to the first training data set until a preset condition is met to obtain a first teacher network, wherein the first teacher network is the first neural network that has been trained; The training module is also used to train the first student network by knowledge distillation according to the first teacher network and the second training data set until a preset condition is met to obtain a second student network, where the second student network is the first student network that has performed the training operation.

11. The device according to claim 10, characterized in that The training module is further used to train the first neural network according to a third training data set until a preset condition is met to obtain a second teacher network, wherein the third training data set includes the labeled training images from the source domain and the labeled training images from the target domain, and the second teacher network is the first neural network that has performed a training operation; The training module is specifically used to train the first student network by knowledge distillation according to the first teacher network, the second teacher network and the second training data set until a preset condition is met to obtain the second student network.

12. The device according to claim 11, characterized in that The training module comprises: An acquisition submodule, used to acquire a second training image, where the second training image is any training image in the second training data set; a processing submodule, configured to process the second training image using the first teacher network to generate a first prediction result corresponding to the second training image; The processing submodule is further used to process the second training image using the second teacher network to generate a second prediction result corresponding to the second training image; a summing submodule, configured to perform a weighted summation of the first prediction result and the second prediction result to obtain a target prediction result corresponding to the second training image; The processing submodule is further used to process the second training image using the first student network to generate a third prediction result corresponding to the second training image; A training submodule is used to train the first student network according to a target loss function, wherein the target loss function indicates a similarity between the third prediction result and the target prediction result.

13. The device according to any one of claims 10 to 12, characterized in that The device also includes: a processing module, configured to process the second training image in the second training data set through the second student network to generate a prediction result corresponding to the second training image, and determine a label of the second training image according to the prediction result corresponding to the second training image; a mixing module, configured to perform a pixel-level mixing operation according to the labeled training image from the source domain, the labeled training image from the target domain, and the second labeled training image to obtain an updated first training data set; The training module is further used to train the first neural network according to the updated first training data set until a preset condition is met to obtain an updated first teacher network; The training module is also used to train the first student network by knowledge distillation according to the updated first teacher network and the second training data set until a preset condition is met to obtain an updated second student network, wherein the updated second student network is the first student network that has performed the training operation.

14. The device according to claim 11 or 12, characterized in that The training module is specifically used to train the first student network by knowledge distillation according to the first teacher network, the second teacher network and the second training data set, and to train the first student network according to a fourth training data set, wherein the fourth training data set includes labeled training images from the target domain.

15. The device according to any one of claims 10 to 12, characterized in that The training images of the source domain and the training images of the target domain are derived from different image generating devices, or the training images of the source domain and the training images of the target domain are derived from different image acquisition locations, or the image acquisition time of the training images of the source domain and the image acquisition time of the training images of the target domain are different, or the photographed objects in the training images of the source domain and the photographed objects in the training images of the target domain are different object types.

16. The device according to any one of claims 10 to 12, characterized in that The first neural network and the first student network are both used to perform any one of the following tasks: image semantic segmentation, image classification, or object detection on an image.

17. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be processed from a target domain; A processing module, used for inputting the image to be processed from the target domain into a second student network, so as to process the image to be processed through the second student network and obtain a prediction result output by the second student network; Among them, the second student network is trained by knowledge distillation based on the first teacher network and the second training data set, the first teacher network is trained based on the first training data set, the first training data set includes at least one labeled first training image, the first training image is obtained by mixing the labeled training image from the source domain with the labeled training image from the target domain at the pixel level, the label of the first training image is obtained based on the first label and the second label, the first label corresponds to the labeled training image from the source domain, the second label corresponds to the labeled training image from the target domain, and the second training data set includes unlabeled training images from the target domain.

18. The device according to claim 17, characterized in that The second student network is trained by knowledge distillation based on the first teacher network, the second teacher network and the second training data set, and the second teacher network is trained based on a third training data set, and the third training data set includes the labeled training images from the source domain and the labeled training images from the target domain.

19. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 7, or the method according to claim 8 or 9.

20. A computer-readable storage medium, characterized in that: The invention comprises a program, which, when being executed on a computer, enables the computer to execute the method according to any one of claims 1 to 7, or enables the computer to execute the method according to claim 8 or 9.

21. An execution device, characterized in that: comprising a processor and a memory, the processor being coupled to the memory, The memory is used to store programs; The processor is used to execute the program in the memory, so that the execution device executes the method according to claim 8 or 9.

22. A training device, characterized in that comprising a processor and a memory, the processor being coupled to the memory, The memory is used to store programs; The processor is used to execute the program in the memory so that the training device performs the method as claimed in any one of claims 1 to 7.

Images