Method and device for training network model

Abstract

The embodiment of the invention provides a method and device for training a network model. The network model comprises a teacher network model and a student network model, and the method comprises the following steps: training based on a first image to obtain the teacher network model and a first student network model, the first image having an artificial mask; the second image is input into the teacher network model, a first prediction mask of the second image and an uncertainty value of the first prediction mask are obtained, and the second image does not have an artificial mask; processing the second image based on the uncertainty value of the first prediction mask; and training a first student network model based on the first image and the processed second image to obtain a second student network model. According to the method and the device provided by the embodiment of the invention, the robustness and the performance of the network model can be improved.

Description

[0001] This application claims the priority of the invention application filed with the China Patent Office on August 3, 2023, with application number 202310973705.0 and invention name “Method and device for training instance segmentation network model”, all contents of which are incorporated into this application by application. Technical Field

[0002] The embodiments of the present application belong to the field of computer vision, and in particular, to a method and device for training a network model. Background Art

[0003] Currently, training object detection and / or instance segmentation network models using fully supervised learning has the best performance, but requires a large amount of annotated image data, including object locations and pixel-level segmentation masks. Manually annotating this data is a time-consuming and laborious task, especially for large-scale datasets and complex scenes. Summary of the invention

[0004] In view of this, an embodiment of the present application provides a method and apparatus for training a network model, which can improve the robustness and performance of the network model.

[0005] In a first aspect, a method for training a network model is provided, wherein the network model includes a teacher network model and a student network model, the method comprising: obtaining a teacher network model and a first student network model based on a first image training, wherein the first image has an artificial mask; inputting a second image into the teacher network model to obtain a first prediction mask of the second image and an uncertainty value of the first prediction mask, wherein the second image does not have an artificial mask; processing the second image based on the uncertainty value of the first prediction mask; and training the first student network model based on the first image and the processed second image to obtain a second student network model.

[0006] In this embodiment, the teacher network model guides the student network model so that the second image without artificial mask can be used to train the network model, which can achieve the compression of the artificial annotation cost required for training, and can improve the performance and generalization of the network model with a small amount of artificial annotation data. In addition, by introducing the estimation of the uncertainty value of the predicted mask, the accuracy of the predicted mask can be improved, thereby improving the robustness and performance of the network model, expanding the use scenarios of the network model, and can be widely used in image processing, computer vision, artificial intelligence and other fields.

[0007] In one possible implementation, the method also includes: inputting the second image into the first student network model to obtain a second prediction mask of the second image and an uncertainty value of the second prediction mask; processing the second image based on the uncertainty value of the first prediction mask, including: when the uncertainty value of the first prediction mask is less than the uncertainty value of the second prediction mask, determining that the second image is used to train the first student network model; training the first student network model based on the first image and the processed second image to obtain a second student network model, including: training the first student network model based on the first image annotated with an artificial mask and the second image annotated with the first prediction mask to obtain a second student network model.

[0008] In this embodiment, the prediction mask output by the teacher network model is learned only when the uncertainty value output by the teacher network model is less than the uncertainty value output by the first student network model, which can reduce the probability that the noise of automated pseudo-labeling introduced in the semi-supervised learning process will be learned by the network model.

[0009] In one possible implementation, the method also includes: refining the first prediction mask to obtain a third prediction mask; processing the second image based on the uncertainty value of the first prediction mask, including: weighting the third prediction mask using the uncertainty value of the first prediction mask to obtain a fourth prediction mask; training the first student network model based on the first image and the processed second image to obtain a second student network model, including: training the first student network model based on the first image annotated with an artificial mask and the second image annotated with a fourth prediction mask to obtain a second student network model.

[0010] In this embodiment, by refining the prediction mask output by the teacher network model for the second image without manual mask annotation, and sending the refined prediction mask as the prediction mask of the second image to the student network model for training, the performance of the student network model can be improved.

[0011] In one possible implementation, the method includes: when using the first image to train the network model, obtaining the fifth prediction mask of the first image; training a teacher network model and a student network model based on the first image, including: based on the prediction mask of each position in the fifth prediction mask, the artificial mask of each position in the artificial mask image, the probability of each position predicted by the network model as the foreground, and the constraint relationship of the uncertainty value of each position in the fifth prediction mask, training an uncertainty estimation network model in the network model.

[0012] In this embodiment, based on the constraint relationship between the predicted mask of the first image with an artificial mask and the artificial mask, the probability of the network model predicting it as a foreground, and the uncertainty value, an uncertainty estimation network model in the network model is trained, which is beneficial to reducing the interference caused by the inaccurate mask predicted for the second image without an artificial mask during the introduction of semi-supervised learning.

[0013] In a possible implementation, the constraint relationship is: Wherein, dice is the set similarity measurement function of the fifth predicted mask and the artificial mask image; bce is the binary cross entropy of each position in the fifth predicted mask and the artificial mask image; G x is the artificial mask at each position in the artificial mask image, d x is the probability that the network model predicts each position as the foreground, σ x is the uncertainty value at each position.

[0014] In this embodiment, the The constraint relationship training uncertainty estimation network model can better reduce the impact of noise labels on network training.

[0015] In a possible implementation manner, the method further includes: determining self-information of each position in the first prediction mask; and determining an edge region in the first prediction mask based on the self-information of each position.

[0016] In this embodiment, the edge area in the prediction mask is determined by self-information. The larger the self-information, the greater the high-frequency information, which means it is closer to the edge of the target object. This promotes the network model's perception of the high-frequency signal of the target edge, thereby optimizing the edge segmentation effect.

[0017] In a possible implementation, determining the self-information of each position in the first prediction mask includes: determining the self-information of each position in the first prediction mask based on the probability of the prediction mask of the current position and the probability of the prediction mask of the adjacent position in the first prediction mask.

[0018] In this embodiment, based on the probability of the prediction mask of the current position in the prediction mask and the probability of the prediction mask of the adjacent position, the self-information of each position in the prediction mask is determined. After obtaining the self-information of each position, it can be used as the weight of the prediction mask loss of each position, thereby driving the network model to pay more attention to the area with larger self-information, that is, the edge area of ​​the target, thereby optimizing the edge segmentation effect.

[0019] In a possible implementation, determining the self-information of each position in the first prediction mask based on the probability of the prediction mask of the current position and the probability of the prediction mask of the adjacent position in the first prediction mask includes: determining the self-information of each position in the first prediction mask based on the following formula: Among them, S i is the self-information of the ith position, f i is the probability of the predicted mask at the ith position, f j is the probability of the predicted mask at the jth position in the neighborhood N(i) around the i-th position, ω and b are the weight coefficient and bandwidth of the Gaussian kernel.

[0020] In this embodiment, the Determining the self-information of each position in the predicted mask is beneficial to optimizing the segmentation effect of the edge.

[0021] In a possible implementation, the method further includes: updating parameters of the teacher network model using model parameters fed back by the second student network model.

[0022] In this embodiment, the model parameters fed back by the second student network model are used to update the parameters of the teacher network model, which can prevent the biased prediction problem caused by class imbalance.

[0023] In a second aspect, a device for training a network model is provided, which network model includes a teacher network model and a student network model, and the device includes: a training module, used to train the teacher network model and the first student network model based on a first image, and the first image has an artificial mask; an acquisition module, used to input the second image into the teacher network model, obtain the first prediction mask of the second image and the uncertainty value of the first prediction mask, and the second image does not have an artificial mask; a processing module, used to process the second image based on the uncertainty value of the first prediction mask; the training module is also used to: train the first student network model based on the first image and the processed second image to obtain the second student network model.

[0024] In a possible implementation, the acquisition module is also used to: input the second image into the first student network model to obtain the second prediction mask of the second image and the uncertainty value of the second prediction mask; the processing module is specifically used to: when the uncertainty value of the first prediction mask is less than the uncertainty value of the second prediction mask, determine that the second image is used to train the first student network model; the training module is specifically used to: train the first student network model based on the first image annotated with the artificial mask and the second image annotated with the first prediction mask to obtain the second student network model.

[0025] In one possible implementation, the processing module is also used to: refine the first prediction mask to obtain a third prediction mask; the processing module is specifically used to: weight the third prediction mask using the uncertainty value of the first prediction mask to obtain a fourth prediction mask; the training module is specifically used to: train the first student network model based on the first image annotated with the artificial mask and the second image annotated with the fourth prediction mask to obtain the second student network model.

[0026] In a possible implementation, the acquisition module is also used to: acquire the fifth prediction mask of the first image when using the first image to train the network model; the training module is specifically used to: train the uncertainty estimation network model in the network model based on the prediction mask of each position in the fifth prediction mask, the artificial mask of each position in the artificial mask image, the probability of each position being the foreground predicted by the network model, and the constraint relationship of the uncertainty value of each position in the fifth prediction mask.

[0027] In a possible implementation, the constraint relationship is: Wherein, dice is the set similarity measurement function of the fifth predicted mask and the artificial mask image; bce is the binary cross entropy of each position in the fifth predicted mask and the artificial mask image; G x is the annotation of each position in the artificial mask image, d x is the probability that the network model predicts each position as the foreground, σ x is the uncertainty value at each position.

[0028] In a possible implementation manner, the processing module is further configured to: determine self-information of each position in the first prediction mask; and determine an edge region in the first prediction mask based on the self-information of each position.

[0029] In a possible implementation manner, the processing module is specifically configured to determine the self-information of each position in the first prediction mask based on the probability of the prediction mask at the current position and the probability of the prediction mask at the adjacent position in the first prediction mask.

[0030] In a possible implementation manner, the processing module is specifically configured to determine the self-information of each position in the first prediction mask based on the following formula: Among them, S i is the self-information of the ith position, f i is the probability of the predicted mask at the ith position, f j is the probability of the predicted mask at the jth position in the neighborhood N(i) around the i-th position, ω and b are the weight coefficient and bandwidth of the Gaussian kernel.

[0031] In a possible implementation, the device further includes: an updating module, configured to update parameters of the teacher network model using model parameters fed back by the second student network model.

[0032] In a third aspect, a method for target detection and / or instance segmentation is provided, which is performed using a second student model trained by the method in the first aspect and any possible implementation of the first aspect thereof.

[0033] In a fourth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store instructions, and the processor is used to read the instructions and execute the method in the first aspect and any possible implementation of the first aspect according to the instructions.

[0034] In a fifth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the method in the first aspect and any possible implementation manner of the first aspect.

[0035] In a sixth aspect, a chip is provided, comprising a processor for calling and running a computer program from a memory, so that a device equipped with the chip executes a method as in the first aspect and any possible implementation of the first aspect.

[0036] In a seventh aspect, a computer program is provided, which enables a computer to execute the method in the first aspect and any possible implementation manner of the first aspect.

[0037] In an eighth aspect, a program product is provided, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code, wherein when the computer-readable code runs in an electronic device, a processor in the electronic device executes a method as in the first aspect and any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the drawings without paying creative work.

[0039] Figure 1 It is a structural diagram of the system architecture provided by this application.

[0040] Figure 2 A schematic block diagram of a method for training an instance segmentation network model according to an embodiment of the present application.

[0041] Figure 3Another schematic block diagram of a method for training an instance segmentation network model according to an embodiment of the present application.

[0042] Figure 4 This is another schematic block diagram of a method for training an instance segmentation network model according to an embodiment of the present application.

[0043] Figure 5 A schematic block diagram of an apparatus for training an instance segmentation network model according to an embodiment of the present application.

[0044] Figure 6 Another schematic block diagram of an apparatus for training an instance segmentation network model according to an embodiment of the present application.

[0045] Figure 7 A schematic block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] Unless otherwise defined, all technical and scientific terms used in this application have the same meanings as those commonly understood by technicians in the technical field of this application; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned drawings and any variations thereof are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order or a primary and secondary relationship.

[0048] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0049] The term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.

[0050] In current technologies, the best performance is achieved by training target detection and instance segmentation network models using fully supervised learning methods. Such technologies require the collection of a large number of image datasets containing target and instance segmentation annotations. These images contain target objects and corresponding pixel-level segmentation masks. After collecting the datasets, select appropriate neural network models for target detection and instance segmentation network models based on business needs. Commonly used models include deep learning-based convolutional neural networks, such as Mask R-CNN, YOLO, Mask2Former, etc. Despite the success of existing fully supervised learning and instance segmentation technologies, such technologies require a large number of annotated image datasets, which include target locations and pixel-level segmentation masks. Manually annotating this data is a time-consuming and labor-intensive task, especially for large-scale datasets and complex scenes.

[0051] Therefore, an embodiment of the present application provides a method for training a network model. By introducing a semi-supervised learning method, a network model is trained using a small amount of labeled data and a large amount of unlabeled data, thereby significantly reducing the amount of manually labeled data and saving manpower, material and financial resources.

[0052] It should be understood that the method for training a network model in the embodiment of the present application is applicable not only to training a target detection network model, but also to training an instance segmentation network model, and can also be applied to training network models for other purposes.

[0053] In order to better understand the solution of the embodiment of the present application, Figure 1 The possible application scenarios of the embodiments of the present application are briefly introduced.

[0054] like Figure 1 As shown, the embodiment of the present application provides a system architecture 100. Figure 1 In the embodiment, the data acquisition device 160 can be used to collect defect detection data of the battery. For example, the data acquisition device 160 can be a defect detection device.

[0055] After the defect detection data of the battery is collected, some of the defect detection data can be manually annotated, and the annotated content is the manual mask described below. The data acquisition device 160 can store the manually annotated defect detection data and the defect detection data that have not been manually annotated in the database 130, and the training device 120 trains the target model / rule 101 based on the defect detection data with manual masks and the defect detection data without manual masks in the database 130. For example, the target model / rule can be an instance segmentation network model for defect detection.

[0056] The above-mentioned target model / rule 101 can be used to implement defect detection of batteries in the embodiment of the present application. The target model / rule 101 in the embodiment of the present application can specifically be a neural network. It should be noted that in actual applications, the defect detection data maintained in the database 130 does not necessarily all come from the collection of the data acquisition device 160, but may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the target model / rule 101 entirely based on the defect detection data maintained by the database 130, and it is also possible to obtain defect detection data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0057] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as Figure 1 The execution device 110 shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a notebook computer, etc., or a server or a cloud. Figure 1 In the embodiment, the execution device 110 configures an input / output (I / O) interface 112 for data interaction with an external device. A user may input data to the I / O interface 112 through a client device 140. The input data may include defect detection data of the battery to be tested in the embodiment of the present application.

[0058] In some implementations, the client device 140 may be the same device as the execution device 110 . For example, the client device 140 and the execution device 110 may both be terminal devices.

[0059] In other embodiments, the client device 140 may be a different device from the execution device 110. For example, the client device 140 is a terminal device, and the execution device 110 is a cloud device, a server, or other device. The client device 140 may interact with the execution device 110 through a communication network with any communication mechanism / communication standard. The communication network may be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0060] The calculation module 111 of the execution device 110 is used to process the input data received by the I / O interface 112. When the calculation module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0061] Finally, the I / O interface 112 returns the processing result, such as the abnormal indicator data obtained above, to the client device 140 to provide it to the user.

[0062] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0063] exist Figure 1 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0064] It is worth noting that Figure 1 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 1 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0065] like Figure 1As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 can be a neural network in the embodiment of the present application. Specifically, the neural network in the embodiment of the present application can be a convolutional neural network (CNN), a regional convolutional neural network (RCNN) or other types of neural networks, etc., and the present application does not make specific limitations on this.

[0066] Figure 2 A schematic block diagram of a method for training an instance segmentation network model is shown. The method can be performed by an electronic device, for example, Figure 1 The training device 120 shown in the figure is executed. Optionally, the network model includes a teacher network model and a student network model, such as Figure 2 As shown, the method 200 may include part or all of the following contents.

[0067] S210, obtaining a teacher network model and a first student network model based on training of a first image, wherein the first image has an artificial mask.

[0068] S220, inputting the second image into the teacher network model to obtain a first prediction mask of the second image and an uncertainty value of the first prediction mask, wherein the second image does not have an artificial mask.

[0069] S230: Process the second image based on the uncertainty value of the first prediction mask.

[0070] S240, training a first student network model based on the first image and the processed second image to obtain a second student network model.

[0071] First, it should be explained that the first image in the embodiment of the present application refers to an image with an artificial mask, and the second image refers to an image without an artificial mask. In addition, the mask in the embodiment of the present application can also be understood as a label.

[0072] The method 200 of the embodiment of the present application is only for a sampling round of training the network model, not the entire training process. In other words, the second image in the method 200 is not all the images without artificial masks, but part of them. For example, there are a total of 2000 second images without artificial masks, and only 8 second images are used in executing the method 200 once.

[0073] Specifically, in S210, the first image with an artificial mask can be used to train a student network model for multiple rounds, for example, 5000 rounds. Then, the first student network model obtained after multiple rounds of training is copied as a teacher network model, and then the teacher network model is used to guide the first student network model to learn the second image without an artificial mask to obtain a second student network model. When training the network model, it is necessary not only to train the mask prediction network model, but also to train the uncertainty estimation network model. Among them, the uncertainty estimation network model refers to a network model for estimating the uncertainty of the predicted mask. For example, an additional branch can be added to the prediction mask branch to predict the uncertainty value of each predicted mask. This branch can be learned by maximum likelihood estimation, that is, maximizing the quality of the predicted mask, and this branch is only trained on the first image with an artificial mask.

[0074] In S220, when the teacher network model is obtained, the second image without the artificial mask may be input into the teacher network model to obtain a first prediction mask of the second image and an uncertainty value of the first prediction mask.

[0075] In S230, after obtaining the first prediction mask predicted by the teacher network model for the second image without artificial mask and the uncertainty value of the first prediction mask, the second image can be further processed based on the uncertainty value of the first prediction mask. For example, the second image of the first prediction mask with a larger uncertainty value can be filtered. For another example, the first prediction mask can be refined.

[0076] In S240, after the second image is processed based on the uncertainty value of the first prediction mask, the first student network model can be trained based on the first image and the processed second image to obtain a second student network model.

[0077] In this embodiment, the teacher network model guides the student network model so that the second image without artificial mask can be used to train the model, which can achieve the compression of the artificial annotation cost required for training, and can improve the performance and generalization of the network model with a small amount of artificial annotation data. In addition, by introducing the uncertainty estimation of the prediction mask, the accuracy of the prediction mask can be improved, thereby improving the robustness and performance of the network model, expanding the use scenarios of the network model, and can be widely used in image processing, computer vision, artificial intelligence and other fields.

[0078] In one embodiment, if Figure 3As shown, method 200 also includes: S225, inputting the second image into the first student network model to obtain a second prediction mask of the second image and an uncertainty value of the second prediction mask; S230, namely, processing the second image based on the uncertainty value of the first prediction mask, including: S231, when the uncertainty value of the first prediction mask is less than the uncertainty value of the second prediction mask, determining that the second image is used to train the first student network model; S240, namely, training the first student network model based on the first image and the processed second image to obtain a second student network model, including: S241, training the first student network model based on the first image annotated with an artificial mask and the second image annotated with the first prediction mask to obtain a second student network model.

[0079] Specifically, the same second image can be input into the teacher network model and the first student network model respectively, the teacher network model outputs a first prediction mask and an uncertainty value of the first prediction mask, and the first student network model outputs a second prediction mask and an uncertainty value of the second prediction mask. The uncertainty value output by the teacher network model is compared with the uncertainty value output by the first student network model. Only when the uncertainty value output by the teacher network model is less than the uncertainty value output by the first student network model, the second image output by the teacher network model and annotated with the first prediction mask is used to train the student network model. That is, the first student network model is trained based on the first image annotated with the artificial mask and the second image annotated with the first prediction mask. When the uncertainty value output by the teacher network model is greater than or equal to the uncertainty value output by the first student network model, the second image is filtered out and not used to train the first student network model.

[0080] In this embodiment, the prediction mask output by the teacher network model is learned only when the uncertainty value output by the teacher network model is less than the uncertainty value output by the first student network model, which can reduce the probability that the noise of automated pseudo-labeling introduced in the semi-supervised learning process will be learned by the model.

[0081] In another embodiment, if Figure 4 As shown, method 200 also includes: S226, refining the first prediction mask to obtain a third prediction mask; S230, i.e. processing the second image based on the uncertainty value of the first prediction mask, including: S232, weighting the third prediction mask using the uncertainty value of the first prediction mask to obtain a fourth prediction mask; S240, i.e. training the first student network model based on the first image and the processed second image, including: S242, training the first student network model based on the first image annotated with an artificial mask and the second image annotated with the fourth prediction mask to obtain a second student network model.

[0082] Specifically, in the stage where the teacher network model guides the student network model to learn without artificial masks, the teacher network model first performs an inference process on the second image without artificial masks to obtain a predicted mask, for example, a first predicted mask. Then, non-maximum suppression is used to filter the fixed score threshold to obtain a rough mask, and then the predicted bounding box and the randomly sampled points on the predicted mask are used as the prompt input of SAM to obtain a more refined predicted mask, and the minimum bounding box of the refined predicted mask is taken as the new bounding box result. Finally, the uncertainty value of the first predicted mask is used to weight the refined predicted mask, and then it is sent to the first student network model as the predicted mask of the second image without artificial masks for training to obtain the second student network model.

[0083] In this embodiment, by refining the prediction mask output by the teacher network model for the second image without artificial mask, and sending the refined prediction mask as the prediction mask of the second image to the student network model for training, the performance of the student network model can be improved.

[0084] Optionally, in an embodiment of the present application, method 200 includes: when using the first image to train the network model, obtaining the fifth prediction mask of the first image; S210, that is, obtaining a teacher network model and a first student network model based on the first image training, including: based on the prediction mask of each position in the fifth prediction mask, the manually annotated mask of each position in the manual mask image, the probability of each position being predicted by the network model as the foreground, and the constraint relationship of the uncertainty value of each position in the fifth prediction mask, an uncertainty estimation network model in the network model is trained.

[0085] It should be understood that the foreground here can be understood as a target. For example, in defect detection, the foreground can be understood as a defect.

[0086] In this embodiment, based on the constraint relationship between the predicted mask of the first image with an artificial mask and the artificial mask image, the probability of the network model predicting it as a foreground, and the uncertainty value, an uncertainty estimation network model in the network model is trained, which is beneficial to reducing the interference caused by the inaccurate mask predicted for the second image without an artificially annotated mask during the introduction of semi-supervised learning.

[0087] Optionally, in the embodiment of the present application, the constraint relationship is:

[0088] Wherein, dice is the set similarity measurement function of the fifth predicted mask and the artificial mask image; bce is the binary cross entropy of each position in the fifth predicted mask and the artificial mask image; G x is the annotation of each position in the artificial mask image, d xis the probability that the network model predicts each position as the foreground, σ x is the uncertainty value at each position.

[0089] In this embodiment, the The constraint relationship training uncertainty estimation network model can better reduce the impact of noise labels on network training.

[0090] Optionally, in an embodiment, the method 200 further includes: determining self-information of each position in the first prediction mask; and determining an edge region in the first prediction mask based on the self-information of each position.

[0091] Conventional training methods often lack optimization of target edges, resulting in these high-frequency edge areas being less smooth. In this embodiment, the edge area in the prediction mask is determined by self-information. The greater the self-information, the greater the high-frequency information of the location, indicating that it is closer to the edge of the target object, thereby promoting the network model's perception of the high-frequency signal of the target edge, thereby optimizing the edge segmentation effect.

[0092] Optionally, in an embodiment, determining the self-information of each position in the first prediction mask includes: determining the self-information of each position in the first prediction mask based on the probability of the prediction mask of the current position in the first prediction mask and the probability of the prediction mask of the adjacent position.

[0093] In this embodiment, based on the probability of the prediction mask of the current position in the prediction mask and the probability of the prediction mask of the adjacent position, the self-information of each position in the prediction mask is determined. After obtaining the self-information of each position, it can be used as the weight of the prediction mask loss of each position, thereby driving the network model to pay more attention to the area with larger self-information, that is, the edge area of ​​the target, thereby optimizing the edge segmentation effect.

[0094] Optionally, in an embodiment, determining the self-information of each position in the first prediction mask based on the probability of the prediction mask of the current position and the probability of the prediction mask of the adjacent position in the first prediction mask includes: determining the self-information of each position in the first prediction mask based on the following formula: Among them, S i is the self-information of the ith position, f i is the probability of the predicted mask at the ith position, f j is the probability of the predicted mask at the jth position of the neighborhood N(i) around the i-th position, ω and b are the weight coefficient and bandwidth of the Gaussian kernel.

[0095] In this embodiment, the Determining the self-information of each position in the predicted mask is beneficial to optimizing the segmentation effect of the edge.

[0096] Optionally, in one embodiment, method 200 further includes: updating parameters of the teacher network model using model parameters fed back by the second student network model.

[0097] For example, during the entire training process, only the student network model accepts the direct stochastic gradient descent optimization algorithm to update the model parameters, while the teacher network model updates the parameters through the exponential moving average (EMA) to prevent the biased prediction problem caused by class imbalance.

[0098] The method for training a network model according to an embodiment of the present application is described in detail above. Figures 5 and 6 , describing an apparatus for training a network model according to an embodiment of the present application, the technical features described in the method embodiment are applicable to the following apparatus embodiments.

[0099] Figure 5 FIG. 3 is a schematic block diagram of an apparatus 300 for training a network model according to an embodiment of the present application. The network model includes a teacher network model and a student network model. Figure 5 As shown, the device 300 may include some or all of the following contents.

[0100] The training module 310 is used to obtain a teacher network model and a first student network model based on the first image training, and the first image has an artificial mask.

[0101] The acquisition module 320 is used to input the second image into the teacher network model to obtain the first prediction mask of the second image and the uncertainty value of the first prediction mask, and the second image does not have an artificial mask.

[0102] The processing module 330 is configured to process the second image based on the uncertainty value of the first prediction mask.

[0103] The training module 310 is also used to train the first student network model based on the first image and the processed second image to obtain a second student network model.

[0104] Optionally, in one embodiment, the acquisition module 320 is also used to: input the second image into the first student network model to obtain the uncertainty value of the second prediction mask of the second image; the processing module 330 is specifically used to: when the uncertainty value of the first prediction mask is less than the uncertainty value of the second prediction mask, determine to use the second image for training the first student network model; the training module 310 is specifically used to: train the first student network model based on the first image annotated with the artificial mask and the second image annotated with the first prediction mask to obtain the second student network model.

[0105] Optionally, in one embodiment, the processing module 330 is also used to: refine the first prediction mask to obtain a third prediction mask; the processing module 330 is specifically used to: weight the third prediction mask using the uncertainty value of the first prediction mask to obtain a fourth prediction mask; the training module 310 is specifically used to: train the first student network model based on the first image annotated with the artificial mask and the second image annotated with the fourth prediction mask to obtain the second student network model.

[0106] Optionally, in one embodiment, the acquisition module 320 is also used to: acquire the fifth prediction mask of the first image when using the first image to train the network model; the training module 310 is specifically used to: train the uncertainty estimation network model in the network model based on the prediction mask of each position in the fifth prediction mask, the artificial mask of each position in the artificial mask image, the probability of each position being predicted by the network model as the foreground, and the constraint relationship of the uncertainty value of each position in the fifth prediction mask.

[0107] Optionally, in one embodiment, the constraint relationship is: Wherein, dice is the set similarity measurement function of the fifth predicted mask and the artificial mask image; bce is the binary cross entropy of each position in the fifth predicted mask and the artificial mask image; G x is the annotation of each position in the artificial mask image, d x is the probability that the network model predicts each position as the foreground, σ x is the uncertainty value at each position.

[0108] Optionally, in an embodiment, the processing module 330 is further configured to: determine self-information of each position in the first prediction mask; and determine an edge region in the first prediction mask based on the self-information of each position.

[0109] Optionally, in an embodiment, the processing module 330 is specifically configured to determine the self-information of each position in the first prediction mask based on the probability of the prediction mask at the current position and the probability of the prediction mask at the adjacent position in the first prediction mask.

[0110] Optionally, in one embodiment, the processing module 330 is specifically configured to determine the self-information of each position in the first prediction mask based on the following formula: Among them, S i is the self-information of the ith position, f i is the probability of the predicted mask at the ith position, f j is the probability of the predicted mask at the jth position in the neighborhood N(i) around the i-th position, ω and b are the weight coefficient and bandwidth of the Gaussian kernel.

[0111] Alternatively, if Figure 6 As shown, the apparatus 300 further includes: an updating module 340, which is used to update the parameters of the teacher network model using the model parameters fed back by the second student network model.

[0112] It should be understood that the device 300 according to the embodiment of the present application may correspond to the execution subject in the embodiment of the method 200 of the present application, and the above and other operations and / or functions of each unit in the device 300 are respectively to implement Figures 2 to 4 For the sake of brevity, the corresponding processes in each method will not be repeated here.

[0113] An embodiment of the present application also provides a method for performing target detection and / or instance segmentation using a second student network model trained by the methods provided in the various embodiments described above.

[0114] like Figure 7 As shown, the embodiment of the present application further provides an electronic device 400, which can be Figures 5 and 6 The device 300 can be used to perform Figures 2 to 4 The contents of each method. Figure 7 The electronic device 400 shown includes a processor 410, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0115] Alternatively, if Figure 7 As shown, the electronic device 400 may further include a memory 420. The processor 410 may call and run a computer program from the memory 420 to implement the method in the embodiment of the present application.

[0116] The memory 420 may be a separate device independent of the processor 410 , or may be integrated into the processor 410 .

[0117] Alternatively, if Figure 7 As shown, the electronic device 400 may further include a transceiver 430, and the processor 410 may control the transceiver 430 to communicate with other devices, specifically, may send information or data to other devices, or receive information or data sent by other devices.

[0118] The transceiver 430 may include a transmitter and a receiver. The transceiver 430 may further include an antenna, and the number of the antennas may be one or more.

[0119] In a specific implementation, the training module, the acquisition module, the processing module and the update module in the apparatus 300 may be composed of Figure 7 The processor 410 in is implemented.

[0120] Optionally, an embodiment of the present application further provides a chip, including a processor, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0121] Optionally, the embodiments of the present application further provide a computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the method in the embodiments of the present application.

[0122] Optionally, an embodiment of the present application further provides a computer program product, including computer program instructions, which enable a computer to execute the method in the embodiment of the present application.

[0123] Optionally, the embodiment of the present application further provides a computer program, which enables a computer to execute the method in the embodiment of the present application.

[0124] It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and performed. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0125] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0126] Although the present application has been described with reference to preferred embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, the various technical features mentioned in the various embodiments may be combined in any manner as long as there are no structural conflicts. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for training a network model, characterized in that: The network model includes a teacher network model and a student network model, including: The teacher network model and the first student network model are obtained by training based on a first image, wherein the first image has an artificial mask; Inputting a second image into the teacher network model to obtain a first prediction mask of the second image and an uncertainty value of the first prediction mask, wherein the second image does not have an artificial mask; processing the second image based on the uncertainty value of the first prediction mask; The first student network model is trained based on the first image and the processed second image to obtain a second student network model.

2. The method according to claim 1, characterized in that The method further comprises: Inputting the second image into the first student network model to obtain a second prediction mask of the second image and an uncertainty value of the second prediction mask; The processing of the second image based on the uncertainty value of the first prediction mask comprises: In a case where the uncertainty value of the first prediction mask is less than the uncertainty value of the second prediction mask, determining to use the second image for training the first student network model; The step of training the first student network model based on the first image and the processed second image to obtain a second student network model includes: The first student network model is trained based on the first image annotated with a manual mask and the second image annotated with the first predicted mask to obtain the second student network model.

3. The method according to claim 1, characterized in that The method further comprises: Refining the first prediction mask to obtain a third prediction mask; The processing of the second image based on the uncertainty value of the first prediction mask comprises: Using the uncertainty value of the first prediction mask to perform weighted processing on the third prediction mask to obtain a fourth prediction mask; The step of training a first student network model based on the first image and the processed second image to obtain a second student network model includes: The first student network model is trained based on the first image annotated with the manual mask and the second image annotated with the fourth predicted mask to obtain the second student network model.

4. The method according to any one of claims 1 to 3, characterized in that The method comprises: When training the network model using the first image, obtaining a fifth prediction mask of the first image; The method of obtaining the teacher network model and the first student network model based on the first image training includes: Based on the prediction mask of each position in the fifth prediction mask, the artificial mask of each position in the artificial mask image, the probability of the network model predicting that each position is a foreground, and the constraint relationship of the uncertainty value of each position in the fifth prediction mask, an uncertainty estimation network model in the network model is trained.

5. The method according to claim 4, characterized in that The constraint relationship is: Wherein, dice is the set similarity measurement function of the fifth predicted mask and the artificial mask image, bce is the binary cross entropy of each position in the fifth predicted mask and the artificial mask image, G x is the artificial mask at each position in the artificial mask image, d x is the probability that the network model predicts that each position is a foreground, σ x is the uncertainty value of each position.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: determining self-information for each position in the first prediction mask; Based on the self-information of each position, an edge region in the first prediction mask is determined.

7. The method according to claim 6, characterized in that The determining of the self-information of each position in the first prediction mask comprises: Based on the probability of the prediction mask at the current position and the probability of the prediction mask at the adjacent position in the first prediction mask, the self-information of each position in the first prediction mask is determined.

8. The method according to claim 7, characterized in that The determining, based on the probability of the prediction mask at the current position and the probability of the prediction mask at the adjacent position in the first prediction mask, the self-information of each position in the first prediction mask comprises: The self-information of each position in the first prediction mask is determined based on the following formula: Among them, S i is the self-information of the ith position, f i is the probability of the predicted mask at the i-th position, f j is the probability of the predicted mask of the jth position in the neighborhood N(i) around the i-th position, ω and b are the weight coefficient and bandwidth of the Gaussian kernel.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: The model parameters fed back by the second student network model are used to update the parameters of the teacher network model.

10. A device for training a network model, characterized in that: The network model includes a teacher network model and a student network model, and the device includes: A training module, configured to obtain the teacher network model and the first student network model through training based on a first image, wherein the first image has an artificial mask; an acquisition module, configured to input a second image into the teacher network model, obtain a first prediction mask of the second image and an uncertainty value of the first prediction mask, wherein the second image does not have an artificial mask; a processing module, configured to process the second image based on the uncertainty value of the first prediction mask; The training module is also used to: The first student network model is trained based on the first image and the processed second image to obtain a second student network model.

11. The device according to claim 10, characterized in that The acquisition module is also used for: Inputting the second image into the first student network model to obtain a second prediction mask of the second image and an uncertainty value of the second prediction mask; The processing module is specifically used for: In a case where the uncertainty value of the first prediction mask is less than the uncertainty value of the second prediction mask, determining to use the second image for training the first student network model; The training module is specifically used for: The first student network model is trained based on the first image annotated with a manual mask and the second image annotated with the first predicted mask to obtain the second student network model.

12. The device according to claim 10, characterized in that The processing module is also used for: Refining the first prediction mask to obtain a third prediction mask; The processing module is specifically used for: Using the uncertainty value of the first prediction mask to perform weighted processing on the third prediction mask to obtain a fourth prediction mask; The training module is specifically used for: The first student network model is trained based on the first image annotated with the manual mask and the second image annotated with the fourth predicted mask to obtain the second student network model.

13. The device according to any one of claims 10 to 12, characterized in that The acquisition module is also used for: When training the network model using the first image, obtaining a fifth prediction mask of the first image; The training module is specifically used for: Based on the prediction mask of each position in the fifth prediction mask, the artificial mask of each position in the artificial mask image, the probability of the network model predicting that each position is a foreground, and the constraint relationship of the uncertainty value of each position in the fifth prediction mask, an uncertainty estimation network model in the network model is trained.

14. The device according to claim 13, characterized in that The constraint relationship is: Wherein, dice is the set similarity measurement function of the fifth predicted mask and the artificial mask image, bce is the binary cross entropy of each position in the fifth predicted mask and the artificial mask image, G x is the artificial mask at each position in the artificial mask, d x is the probability that the network model predicts that each position is a foreground, σ x is the uncertainty value of each position.

15. The device according to any one of claims 10 to 14, characterized in that The processing module is also used for: determining self-information for each position in the first prediction mask; Based on the self-information of each position, an edge region in the first prediction mask is determined.

16. The device according to claim 15, characterized in that The processing module is specifically used for: Based on the probability of the prediction mask at the current position and the probability of the prediction mask at the adjacent position in the first prediction mask, the self-information of each position in the first prediction mask is determined.

17. The device according to claim 16, characterized in that The processing module is specifically used for: The self-information of each position in the first prediction mask is determined based on the following formula: Among them, S i is the self-information of the ith position, f i is the probability of the predicted mask at the i-th position, f j is the probability of the predicted mask of the jth position in the neighborhood N(i) around the i-th position, ω and b are the weight coefficient and bandwidth of the Gaussian kernel.

18. The device according to any one of claims 10 to 17, characterized in that The device also includes: An updating module is used to update the parameters of the teacher network model using the model parameters fed back by the second student network model.

19. A method for performing target detection and / or instance segmentation using the second student network model trained by the method according to any one of claims 1 to 9.

20. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store instructions, and the processor is used to read the instructions and execute the method according to any one of claims 1 to 9 according to the instructions.

21. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 9.

Images