Training method, terminal and computer readable storage medium of instance segmentation network

Abstract

The application provides a kind of example segmentation network training method, terminal and computer readable storage medium, example segmentation network training method includes: the feature extraction of obtained sample image is carried out, and image feature map is obtained;Mask processing is carried out on the feature value of each pixel point in image feature map respectively, and the generation label corresponding to each pixel point is determined;Image feature map is carried out target segmentation processing based on example segmentation network, and the prediction information corresponding to each pixel point is obtained;The error value between the generation label and prediction information corresponding to each pixel point is iteratively trained example segmentation network.This application does not need to spend a lot of manpower and material resources to mark the boundary information of sample image, according to the sensitivity of each pixel point in feature map, the generation label corresponding to pixel point is judged, the prediction information corresponding to each pixel point is obtained through example segmentation network, and the error value between the generation label and prediction information of each pixel point can realize the training of example segmentation network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a training method of an instance segmentation network, a terminal and a computer readable storage medium. BACKGROUND

[0002] Image instance segmentation is an accurate target positioning technology, which is widely used in the fields of automatic driving, medical imaging, remote sensing image, etc. Compared with the target detection task which can only locate the target position and target rectangular frame and cannot accurately locate the pixel-level target boundary position, the instance segmentation can accurately locate the target boundary, and can better meet the actual application requirements for inclined targets, complex shape targets, etc. In recent years, the instance segmentation algorithm based on neural network has developed greatly, but it often depends on a large number of training samples, and for such pixel-level labeling tasks, labeling is extremely difficult and consumes a lot of manpower and material resources. SUMMARY

[0003] The technical problem solved by the present application is to provide a training method of an instance segmentation network, a terminal and a computer readable storage medium, which solves the problem of consuming a large amount of manpower and material resources in training the instance segmentation network in the prior art.

[0004] To solve the above technical problems, the first technical solution adopted by the present application is to provide a training method of an instance segmentation network, which comprises the following steps:

[0005] performing feature extraction on the obtained sample image to obtain an image feature map; the sample image contains a target object;

[0006] respectively performing mask processing on the feature values of each pixel point in the image feature map to determine the generated label corresponding to each pixel point;

[0007] performing target segmentation prediction on the image feature map based on the instance segmentation network to obtain the prediction information corresponding to each pixel point;

[0008] iteratively training the instance segmentation network based on the error value between the generated label and the prediction information corresponding to each pixel point.

[0009] Among them, respectively performing mask processing on the feature values of each pixel point in the image feature map to determine the generated label corresponding to each pixel point, comprising:

[0010] traversing all pixel points in the image feature map, and assigning the feature value of the selected pixel point to a preset value to obtain an updated feature map;

[0011] respectively processing the updated feature map and the image feature map through the target detection network to obtain the first detection information corresponding to the image feature map and the second detection information corresponding to the updated feature map;

[0012] determine the generated label corresponding to the pixel point based on an error value between the first detection information and the second detection information.

[0013] The first detection information includes a detection category, a detection confidence, and detection position information of the target object; and the second detection information includes a predicted category, a predicted confidence, and predicted position information of the updated target.

[0014] The method further includes:

[0015] In response to the detection category of the target object being the same as the predicted category of the updated target, the generated label corresponding to the pixel point is determined based on whether an error value between the detection confidence and the predicted confidence and an error value between the detection position information and the predicted position information meet corresponding preset requirements; and the generated label includes a foreground pixel point and a background pixel point.

[0016] The method further includes:

[0017] In response to the error value between the detection confidence and the predicted confidence being greater than a maximum confidence threshold value, or the error value between the detection position information and the predicted position information being greater than a maximum position variable threshold value, the generated label of the pixel point is determined to be the foreground pixel point.

[0018] In response to the error value between the detection confidence and the predicted confidence being less than a minimum confidence threshold value, and the error value between the detection position information and the predicted position information being less than a minimum position variable threshold value, the generated label of the pixel point is determined to be the background pixel point.

[0019] The method further includes:

[0020] The instance segmentation network is iteratively trained based on error values between the generated labels of the pixel points corresponding to the foreground pixel points and the background pixel points and the predicted information.

[0021] The first detection information corresponding to the image feature map and the second detection information corresponding to the updated feature map are obtained by processing the updated feature map and the image feature map by the target detection network, and the method further includes:

[0022] The target detection network performs target region detection on the feature map to obtain a candidate target region.

[0023] The target feature map is obtained based on feature extraction of the candidate target region in the feature map.

[0024] Determine the detection information corresponding to the feature map based on the target feature map, and the detection information includes target category, confidence and position information.

[0025] The training method of the target detection network comprises:

[0026] Obtain a plurality of training images, and the training images contain targets; the training images are associated with labeled categories and labeled position information;

[0027] Detect the targets in the training images based on the target detection network to obtain predicted categories and predicted position information of the targets;

[0028] Train the target detection network based on error values between the labeled categories and the predicted categories corresponding to the training images and error values between the labeled position information and the predicted position information.

[0029] The step of performing mask processing on the feature values of each pixel point in the image feature map to determine the generated label corresponding to each pixel point further comprises:

[0030] The size of the image composed of the generated labels of the pixel points is adjusted to a preset size by using the nearest neighbor interpolation method.

[0031] To solve the above technical problems, a second technical solution adopted by the present application is to provide a terminal, which comprises a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor is used to execute program data to realize the steps in the surface defect detection method.

[0032] To solve the above technical problems, a third technical solution adopted by the present application is to provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps in the surface defect detection method.

[0033] The beneficial effect of the present application is that: unlike the prior art, the example instance segmentation network training method, terminal and computer readable storage medium provided by the present application, the instance segmentation network training method comprises: performing feature extraction on the obtained sample image to obtain an image feature map; the sample image contains a target object; the feature values of each pixel point in the image feature map are respectively subjected to mask processing to determine the generated label corresponding to each pixel point; the image feature map is subjected to target segmentation processing based on the instance segmentation network to obtain the prediction information corresponding to each pixel point; and the instance segmentation network is iteratively trained based on the error value between the generated label and the prediction information corresponding to each pixel point. The present application does not need to spend a lot of manpower and material resources to label the boundary information of the sample image, judges the generated label corresponding to each pixel point according to the sensitivity of each pixel point in the feature map to the detection result, performs target segmentation on the image feature map through the instance segmentation network to obtain the prediction information corresponding to each pixel point, and can realize the training of the instance segmentation network based on the error value between the generated label and the prediction information of each pixel point. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0035] Figure 1 is a flowchart of the instance segmentation network training method provided by the present application;

[0036] Figure 2 is Figure 1 is a flowchart of a specific embodiment of step S2 in the instance segmentation network training method provided by the present application;

[0037] Figure 3 is a schematic diagram of the instance segmentation network training provided by the present application;

[0038] Figure 4 is a framework schematic diagram of an embodiment of the terminal provided by the present application;

[0039] Figure 5 is a framework schematic diagram of an embodiment of the computer readable storage medium provided by the present application. DETAILED DESCRIPTION

[0040] The technical solutions of the embodiments of the present application will be described in detail below with reference to the drawings in the specification.

[0041] In the following description, specific details such as specific system structures, interfaces, techniques, etc. are presented in order to thoroughly understand the present application, but are not intended to limit the present application.

[0042] The term "and / or", used herein, is only used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the associated objects. In addition, "multiple" herein means two or more than two.

[0043] In order to enable those skilled in the art to better understand the technical solutions of the present application, the training method of the example segmentation network provided by the present application will be further described in detail below in combination with the drawings and specific embodiments.

[0044] Please refer to Figure 1 , Figure Figure 1 is a flowchart of the training method of the example segmentation network provided by the present application. The training method of the example segmentation network provided in the embodiment includes the following steps.

[0045] S1: performing feature extraction on the obtained sample image to obtain an image feature map, the sample image containing a target object.

[0046] S2: performing mask processing on the feature values of each pixel point in the image feature map to determine the generated label corresponding to each pixel point.

[0047] S3: performing target segmentation on the image feature map based on the example segmentation network to obtain the prediction information corresponding to each pixel point.

[0048] S4: iteratively training the example segmentation network based on the error value between the generated label and the prediction information corresponding to each pixel point.

[0049] Before training the example segmentation network, the target detection network needs to be trained to detect the target object in the training image to obtain the class and target detection frame of the target object. The image in the target detection frame is extracted as a sample image in the training image. Based on the sensitivity of each pixel point, the generated label of each pixel point in the feature map corresponding to the sample image is generated, and the generated label is used as the supervision information. Then, the prediction information of each pixel point is obtained by the example segmentation network, and the example segmentation network is trained based on the error value between the generated label and the prediction information corresponding to each pixel point.

[0050] In an embodiment, the training method of the target detection network includes the following steps.

[0051] A plurality of training images are obtained, the training images containing a target; the training images are associated with a labeled category and labeled position information; a target detection network is used to detect the target in the training images, to obtain predicted category and predicted position information of the target; the target detection network is iteratively trained based on error values between the labeled category and the predicted category of the training images, and error values between the labeled position information and the predicted position information.

[0052] The obtained training images are detected using the trained target detection network, to obtain a target detection frame and a target category of each target object contained in the training images. An image containing the target object in the target detection frame is extracted as a sample image according to position information of the target detection frame in the training image.

[0053] A feature extraction model known in the art is used to extract features of the obtained sample image, to obtain an image feature map corresponding to the sample image. For example, a CNN convolutional neural network can be used to extract features of the sample image, to obtain the image feature map. The size of the image feature map is consistent with the size of the sample image.

[0054] In an embodiment, obtaining a generated label of each pixel point in the image feature map includes the following steps.

[0055] The sample image is composed of a foreground image and a background image. The foreground image is the target object. In order to determine whether each pixel point in the sample image is a background pixel point or a foreground pixel point, the influence degree of each pixel point on the category and position of the target object is distinguished, that is, the background pixel point and the foreground pixel point are distinguished based on the sensitivity degree of the pixel point.

[0056] Based on the principle that the influence degree of the foreground pixel point on the prediction result of the category and position of the target object is large, and the influence degree of the background pixel point on the prediction result of the category and position of the target object is small, the generated label of each pixel point is determined.

[0057] Referring to Figure 2 and Figure 3 , Figure 2 is Figure 1 a flowchart of a specific embodiment of step S2 in the training method of the instance segmentation network provided by the application; Figure 3 is a schematic diagram of the training of the instance segmentation network provided by the application.

[0058] S21: all pixel points in the image feature map are traversed, and the feature value of the selected pixel point is assigned as a preset value, to obtain an updated feature map.

[0059] Specifically, interference is added to the image feature map, so that the pixel point having a large influence on the prediction result is determined as a foreground pixel point, and the pixel point having a small influence on the prediction result is determined as a background pixel point.

[0060] In the embodiment, the feature value of the pixel point at the position coordinate (i, j) in the image feature map is assigned a preset value to obtain an updated feature map corresponding to the pixel point. The preset value can be 0 or other values that facilitate distinguishing the pixel point from surrounding pixel points.

[0061] All pixel points in the image feature map are traversed to obtain the updated feature map corresponding to each pixel point in the image feature map.

[0062] S22: The updated feature map and the image feature map are processed by the target detection network to obtain first detection information corresponding to the image feature map and second detection information corresponding to the updated feature map.

[0063] The first detection information includes the detection category, detection confidence and detection position information of the target object, and the second detection information includes the prediction category, prediction confidence and prediction position information of the updated target.

[0064] Specifically, the target detection network performs target region detection on the feature map to obtain a candidate target region, performs feature extraction on the feature map based on the candidate target region to obtain a target feature map, and determines the detection information corresponding to the feature map based on the target feature map. The detection information includes the target category, confidence and position information.

[0065] The detection category of the target object in the image feature map is C, the detection confidence is P cls , and the detection position information is P box , which are detected by the target detection network. The prediction category of the target object in the updated feature map is C(i, j), the prediction confidence is P cls (i, j), and the detection position information is P box (i, j), which are detected by the target detection network.

[0066] S23: Based on the error value between the first detection information and the second detection information, the generation label corresponding to the pixel point is determined.

[0067] Specifically, in response to the detection category of the target object being the same as the prediction category of the updated target, whether the error value between the detection confidence and the prediction confidence and the error value between the detection position information and the prediction position information meet the corresponding preset requirements is determined to determine the generation label corresponding to the pixel point. The generation label includes foreground pixel points and background pixel points.

[0068] In a specific embodiment, in response to the error value between the detection confidence and the prediction confidence being greater than the maximum confidence threshold, or the error value between the detection position information and the prediction position information being greater than the maximum position variable threshold, the generated label of the pixel point is determined as a foreground pixel point; in response to the error value between the detection confidence and the prediction confidence being less than the minimum confidence threshold, and the error value between the detection position information and the prediction position information being less than the minimum position variable threshold, the generated label of the pixel point is determined as a background pixel point.

[0069] Specifically, the error value between the detection confidence and the prediction confidence corresponding to the target is represented as δ cls (i, j) = ||P c (i, j) - P c (i, j) ||, and the error value between the detection position information and the prediction position information is represented as δ box (i, j) = IoU(P box (i, j) - P box (i, j).

[0070] The pixel point is determined to be a background pixel point and a foreground pixel point based on the following formula 1.

[0071]

[0072] In the formula, M(i, j) represents the generated label of the pixel point, 1 represents a foreground pixel point, 0 represents a background pixel point, and -1 represents that the pixel point is not cared. respectively represent the maximum confidence threshold, the minimum confidence threshold, the maximum position variable threshold, and the minimum position variable threshold.

[0073] In an embodiment, the target segmentation of the image feature map based on the instance segmentation network in step 3 is described in detail.

[0074] In order to facilitate the training of the instance segmentation network, the nearest neighbor interpolation method is used to adjust the size of the image corresponding to the generated label to a preset size. The preset size can be 28*28 or other sizes, which can be set according to actual conditions. The image is a mask image composed of the generated labels of each pixel point.

[0075] After the image feature map is processed by the ROI Align layer, the image feature map is segmented by the instance segmentation network to obtain a mask image corresponding to the image feature map. Based on the mask image, the prediction information corresponding to each pixel point can be obtained. The prediction information includes background pixel points and foreground pixel points.

[0076] In an embodiment, the step 4 is described in detail as follows: the instance segmentation network is iteratively trained based on the error value between the generated label and the prediction information corresponding to each pixel point.

[0077] The instance segmentation network is trained based on the loss function in the following formula 2.

[0078]

[0079] In the formula, L represents the error value between the generated label and the prediction information; Y(i, j) represents the generated label; N represents the total number of foreground pixel points and background pixel points; and represents the prediction information.

[0080] In an embodiment, the instance segmentation network is iteratively trained based on the error value between the generated label and the prediction information corresponding to the background pixel point and the foreground pixel point respectively.

[0081] In this embodiment, the gradient descent method is used to optimize the instance segmentation network until the network converges.

[0082] After the instance segmentation network is trained, when a to-be-detected image containing a target object is obtained, the target detection network is used to perform target detection on the to-be-detected image to obtain a target box and a target category corresponding to the to-be-detected image, and the instance segmentation network obtained by training is used to perform target segmentation on a region image corresponding to the target box to obtain a target mask image.

[0083] The training method of the segmentation network provided in this embodiment includes: performing feature extraction on an obtained sample image to obtain an image feature map; the sample image contains a target object; performing mask processing on the feature values of each pixel point in the image feature map to determine the generated label corresponding to each pixel point; performing target segmentation processing on the image feature map based on an instance segmentation network to obtain prediction information corresponding to each pixel point; and iteratively training the instance segmentation network based on the error value between the generated label and the prediction information corresponding to each pixel point. The present application does not need to spend a lot of manpower and material resources to label the boundary information of the sample image, determines the generated label corresponding to each pixel point according to the sensitivity of each pixel point in the feature map, performs target segmentation on the image feature map through the instance segmentation network to obtain prediction information corresponding to each pixel point, and trains the instance segmentation network based on the error value between the generated label and the prediction information of each pixel point.

[0084] Please refer to Figure 4 , Figure 4 ​is a framework schematic diagram of an embodiment of the terminal provided by the present application. The terminal 80 comprises a memory 81 and a processor 82 coupled with each other. The processor 82 is configured to execute program instructions stored in the memory 81 to implement the steps of any of the above-mentioned embodiment of the training method of the segmentation network. In a specific implementation scenario, the terminal 80 can include but is not limited to a microcomputer, a server, and in addition, the terminal 80 can also include a notebook computer, a tablet computer and other mobile devices, which are not limited here.

[0085] Specifically, the processor 82 is configured to control itself and the memory 81 to implement the steps of any of the above-mentioned embodiment of the training method of the segmentation network. The processor 82 can also be referred to as a CPU (Central Processing Unit). The processor 82 can be an integrated circuit chip with processing capability. The processor 82 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 82 can be implemented by an integrated circuit chip.

[0086] Please refer to Figure 5 , Figure 5 is a framework schematic diagram of an embodiment of the computer readable storage medium provided by the present application. The computer readable storage medium 90 stores program instructions 901 capable of being executed by the processor, and the program instructions 901 are configured to implement the steps of any of the above-mentioned embodiment of the training method of the segmentation network.

[0087] In some embodiments, the apparatus provided by the embodiments of the present application has functions or includes modules that can be used to execute the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0088] The above description of each embodiment tends to emphasize the differences between each embodiment, and the same or similar parts can be mutually referred to. For the sake of brevity, it will not be repeated here.

[0089] In several embodiments provided in the present application, it should be understood that the disclosed methods and apparatuses can be implemented in other manners. For example, the division of the apparatus embodiments is merely illustrative, and for example, the division of the modules or units can be changed, or some features can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0090] In addition, each function unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be implemented in the form of hardware or in the form of a software function unit.

[0091] If the integrated unit is implemented in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application essentially or substantially, or all or part of the technical solutions, can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the methods in the various embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various other media that can store program codes.

[0092] The above is merely an embodiment of the present application, and does not limit the patent protection scope of the present application. Any equivalent structure or equivalent process transformation based on the content of the present application specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A training method for an instance segmentation network, characterized in that, The training method includes: Feature extraction is performed on the acquired sample image to obtain an image feature map; the sample image contains a target object; The feature values ​​of each pixel in the image feature map are masked to determine the generated label corresponding to each pixel. The image feature map is segmented based on an instance segmentation network to obtain prediction information corresponding to each pixel. The instance segmentation network is iteratively trained based on the error value between the generated label and the predicted information corresponding to each pixel. The step of performing masking processing on the feature values ​​of each pixel in the image feature map to determine the generated label corresponding to each pixel includes: Traverse all the pixels in the image feature map, assign the feature value of the selected pixel to a preset value, and obtain the updated feature map; The updated feature map and the image feature map are processed by an object detection network to obtain first detection information corresponding to the image feature map and second detection information corresponding to the updated feature map. The first detection information includes the detection category, detection confidence, and detection location information of the target object. The second detection information includes the predicted category, prediction confidence, and predicted location information of the updated target. In response to the detection category of the target object being the same as the predicted category of the updated target, the generated label corresponding to the pixel is determined based on whether the error value between the detection confidence and the prediction confidence, and the error value between the detection location information and the prediction location information meet the corresponding preset requirements; the generated label includes foreground pixels and background pixels.

2. The training method for the instance segmentation network according to claim 1, characterized in that, The step of determining the generated label corresponding to the pixel based on whether the error value between the detection confidence level and the prediction confidence level, and the error value between the detection location information and the prediction location information meet the corresponding preset requirements includes: If the error between the detection confidence and the prediction confidence is greater than the maximum confidence threshold, or if the error between the detection location information and the prediction location information is greater than the maximum location variable threshold, then the generated label of the pixel is determined to be the foreground pixel. If the error between the detection confidence and the prediction confidence is less than the minimum confidence threshold, and the error between the detection location information and the prediction location information is less than the minimum location variable threshold, then the generated label of the pixel is determined to be the background pixel.

3. The training method for the instance segmentation network according to claim 1, characterized in that, The iterative training of the instance segmentation network based on the error value between the generated label and the predicted information corresponding to each pixel includes: The instance segmentation network is iteratively trained based on the error value between the generated labels of the pixels corresponding to the background pixels and the foreground pixels and the predicted information.

4. The training method for the instance segmentation network according to claim 1, characterized in that, The step of processing the updated feature map and the image feature map respectively through the object detection network to obtain the first detection information corresponding to the image feature map and the second detection information corresponding to the updated feature map includes: The target detection network performs target region detection on the feature map to obtain candidate target regions; Based on the candidate target region, feature extraction is performed on the feature map to obtain the target feature map; Based on the target feature map, the detection information corresponding to the feature map is determined, and the detection information includes target category, confidence level and location information.

5. The training method for the instance segmentation network according to claim 1 or 4, characterized in that, The training method for the object detection network includes: Multiple training images are acquired, each containing a target; the training images are associated with annotation category and annotation location information. The target detection network is used to detect targets in the training images to obtain the predicted category and predicted location information of the targets. The target detection network is trained based on the error values ​​between the labeled category and the predicted category corresponding to the training image, and the error values ​​between the labeled location information and the predicted location information.

6. The training method for the instance segmentation network according to claim 1, characterized in that, After the step of masking the feature values ​​of each pixel in the image feature map to determine the generated label corresponding to each pixel, the method further includes: The size of the image composed of the generated labels of the pixels is adjusted to a preset size using the nearest neighbor interpolation method.

7. A terminal, characterized in that, The terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor, the processor being configured to execute program data to implement the steps in the training method for the instance segmentation network as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the training method for the instance segmentation network as described in any one of claims 1 to 6.

Images