Image instance segmentation training method and related equipment thereof
By cutting the image into a cut image and using an instance segmentation model for pseudo-label annotation and training, the problem of insufficient recognition accuracy in existing X-ray security inspection technology is solved, and high-precision segmentation and identification of contraband is achieved.
Patent Information
- Application Number
- CN202510308860.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-14
- Publication Date
- 2025-08-01
AI Technical Summary
The existing X-ray security technology lacks the accuracy of identifying contrabands under complex backgrounds, overlapping occlusions and multi-scale changes, especially in public places with crowded flows.
By cutting the image into a cut map containing the instance object, using the first instance segmentation model for pseudo-label annotation, and training the second instance segmentation model with the first polygon annotation information and position information, realizing pixel-level accuracy segmentation of the instance object and improving the recognition accuracy.
Under complex backgrounds, overlapping occlusions and multi-scale changes, the identification accuracy of contraband is significantly improved, the cost of manual labeling is reduced, and high-precision segmentation of instance objects is achieved.
Smart Images
Figure CN120411495A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of computer intelligent image processing, and particularly to an instance segmentation training method for images and related devices thereof. Background Art
[0002] Public safety is the focus of attention of every country and society. Especially in crowded public places such as railway stations, stadiums, subways, airports, etc., it is particularly important to protect the safety of people and property. As an important security inspection means, X-ray security inspection technology can effectively detect and identify contraband such as guns and controlled knives in luggage.
[0003] However, in the existing X-ray security inspection technology, manual visual recognition is used to identify contraband in X-ray security inspection images, which has low recognition accuracy, such as problems of missed detection and misdetection. With the development of deep learning technology, although the X-ray security inspection image contraband recognition technology based on object detection algorithms has improved the recognition accuracy of contraband, in the current X-ray security inspection technology, the recognition ability of contraband instance objects in complex backgrounds, overlapping occlusions, and multi-scale changes is weak. Summary of the Invention
[0004] This application provides an instance segmentation training method for images and related devices thereof, which is used to improve the recognition accuracy of instance objects under complex backgrounds, overlapping occlusions, and multi-scale changes.
[0005] In a first aspect of this application, an instance segmentation training method for images is provided, which is applied to a security inspection scenario. The image includes an X-ray image collected by an X-ray acquisition device, and the method includes:
[0006] Input the cropped image into a first instance segmentation model to infer and predict the first pseudo-label annotation result of the instance object in the cropped image of the image. Among them, the first pseudo-label annotation result includes first polygon annotation information and the category of the instance object, and the first polygon annotation information represents the contour boundary coordinates of the instance object in the cropped image;
[0007] Determine the second polygon annotation information of the instance object fused in the image according to the first polygon annotation information and the first position information. Among them, the second polygon annotation information represents the contour boundary coordinates of the instance object in the image, and the first position information represents the position coordinates of the cropped image in the image;
[0008] Train a second instance segmentation model for inferring and predicting the instance segmentation result of the instance object in the target image according to a first training set. Among them, the first training set includes the second polygon annotation information, the category of the instance object, and the image.
[0009] In some implementation manners, the method further includes:
[0010] Input the target image into the second instance segmentation model;
[0011] Obtain the category of the instance object obtained by the second instance segmentation model through inference and prediction of the target image and the corresponding second polygon annotation information.
[0012] In some implementation manners, the method further includes:
[0013] Train the first instance segmentation model according to a second training set, where the second training set includes sample cut-out images of instance objects, the categories of the instance objects in the sample cut-out images, and the rectangular box annotation information of the sample cut-out images.
[0014] In some implementation manners, the determining the second polygon annotation information of the instance object fused in the image according to the first polygon annotation information and the first position information includes:
[0015] Calculate the sum value of each contour boundary coordinate of the first polygon annotation information and the position coordinate of the first position information to obtain the second polygon annotation information of the instance object in the image.
[0016] In some implementation manners, the first training set is a training set obtained by screening according to a first screening condition after the cut-out images of multiple images are inferred and predicted by the first instance segmentation model,
[0017] where the first screening condition includes IOU screening processed according to the second polygon annotation information and the rectangular box annotation information of the corresponding cut-out image, and the rectangular box annotation information includes the first position information and the second position information representing the position coordinates of the cut-out image in the image.
[0018] In some implementation manners, the first screening condition further includes confidence screening of the categories of the instance objects inferred and predicted by the first instance segmentation model, or, quantity consistency screening of the number of instance objects inferred and predicted by the first instance segmentation model and the actual number of instance objects in the corresponding cut-out image.
[0019] In some implementation manners, the process of the IOU screening includes:
[0020] Calculate the IOU according to the second polygon annotation information of the instance objects inferred and predicted for each respective image of the multiple images and the rectangular box annotation information of the corresponding cut-out image to obtain the IOU of the instance objects inferred and predicted for each respective image of the multiple images;
[0021] From the IOU of the instance objects inferred and predicted for each of the respective images among the multiple images, obtain multiple target IOUs that meet the IOU threshold condition, and further obtain the first training set associated with the multiple target IOUs.
[0022] In some implementation manners, calculating the IOU according to the second polygon annotation information of the instance objects inferred and predicted for each of the respective images among the multiple images and the rectangular box annotation information of the corresponding cutout image includes:
[0023] Obtain the maximum coordinates and minimum coordinates in the second polygon annotation information of the instance objects inferred and predicted for each of the respective images among the multiple images;
[0024] Calculate the IOU according to the maximum coordinates and minimum coordinates, and the rectangular box annotation information of the corresponding cutout image, where the difference between the first position information and the second position information represents the size of the cutout image.
[0025] A second aspect of the present application provides an image instance segmentation training device, including:
[0026] An input module, configured to input a cutout image of an image into a first instance segmentation model to infer and predict a first pseudo-label annotation result of an instance object in the cutout image of the image, where the first pseudo-label annotation result includes first polygon annotation information and the category of the instance object, and the first polygon annotation information represents the contour boundary coordinates of the instance object in the cutout image;
[0027] A determination module, configured to determine second polygon annotation information of the instance object fused in the image according to the first polygon annotation information and first position information, where the second polygon annotation information represents the contour boundary coordinates of the instance object in the image, and the first position information represents the position coordinates of the cutout image in the image;
[0028] A training module, configured to train a second instance segmentation model for inferring and predicting an instance segmentation result of an instance object in a target image according to a first training set, where the first training set includes the second polygon annotation information, the category of the instance object, and the image.
[0029] A third aspect of the present application provides an image instance segmentation inference and prediction device, including
[0030] An input module, configured to input a target image into the trained second instance segmentation model;
[0031] An acquisition module, configured to acquire the category of the instance object inferred and predicted by the second instance segmentation model for the target image and the corresponding second polygon annotation information.
[0032] In the fourth aspect of the present application, an electronic device is provided, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method according to any one of the foregoing first aspects.
[0033] In the fifth aspect of the present application, a computer-readable storage medium is provided, in which program instructions are stored, and the program instructions are executed by a processor to execute the method according to any one of the foregoing first aspects.
[0034] Compared with the prior art, the technical solution of the present application cuts an image containing instance objects into a cut image containing instance objects, and then inputs it into the first instance segmentation model for inference and prediction to obtain the first polygon annotation information and category of the instance objects in the cut image. Furthermore, the second polygon annotation information determined according to the first polygon annotation information and the first position information, the category of the instance objects associated with the inference and prediction, and the corresponding image are used to train the second instance segmentation model for inferring and predicting the instance segmentation results of the instance objects in the target image. Since the second instance segmentation model uses the first instance segmentation model based on the first polygon annotation information for inferring and predicting pixel-level accuracy to infer and predict the second polygon annotation information of the instance objects in the target image, it realizes directly inferring and predicting the pixel-level accuracy segmentation results of each instance object in the complete image in the form of the complete image, which improves the recognition accuracy of instance objects under complex backgrounds, overlapping occlusions, and multi-scale changes. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without any creative effort, other drawings can be obtained according to these drawings.
[0036] Figure 1 It is a schematic flowchart of a method for training an image segmentation instance model according to an embodiment of the present application;
[0037] Figure 2 It is a schematic flowchart of a process for training a first instance segmentation model according to an embodiment of the present application;
[0038] Figure 3 It is a schematic flowchart of a process for quickly obtaining the annotation information of an instance object in an image based on the annotation information of the instance object in a cut image and the position information of the cut image in the image;
[0039] Figure 4 It is a schematic flowchart of a confidence screening process;
[0040] Figure 5 It is a schematic diagram of an IOU screening process in an embodiment of the present application;
[0041] Figure 6 It is a schematic diagram of a quantity consistency screening process in an embodiment of the present application;
[0042] Figure 7 It is a schematic overview diagram of a method for training an image segmentation instance model in an embodiment of the present application;
[0043] Figure 8 It is a schematic diagram of an image instance segmentation training device in an embodiment of the present application;
[0044] Figure 9 It is a schematic diagram of an image instance segmentation inference and prediction device in an embodiment of the present application;
[0045] Figure 10 It is a schematic diagram of an electronic device in an embodiment of the present application. Detailed implementation manners
[0046] Next, in combination with the accompanying drawings of the specification, the solutions of the embodiments of the present application will be described in detail.
[0047] In the following description, specific details such as specific system structures, interfaces, and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the present application.
[0048] Next, in combination with the accompanying drawings in the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.
[0049] The terms "first", "second", and "third" in this application are only used for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features represented. Thus, features defined with "first", "second", and "third" may explicitly or implicitly include at least one of such features. In the description of this application, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise specifically defined. All directional indications (such as up, down, left, right, front, back...) in the embodiments of this application are only used to explain the relative positional relationship and movement conditions between components in a specific posture (as shown in the drawings). If the specific posture changes, then the directional indication also changes accordingly. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.
[0050] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.
[0051] Please refer to Figure 1 , Figure 1 shows the pseudo-label result of the instance object in the cut image formed from the complete image through the inference prediction of the first instance segmentation model. Then, according to the pseudo-label result and the complete image of the corresponding cut image, a training set is determined to train a second instance segmentation model for inferring and predicting the segmentation result of the instance object in the complete image, specifically including:
[0052] 101: Input the cut image of the image into the first instance segmentation model to infer and predict the first pseudo-label annotation result of the instance object in the cut image of the image. Among them, the first pseudo-label annotation result includes the first polygon annotation information and the category of the instance object, and the first polygon annotation information represents the contour boundary coordinates of the instance object in the cut image;
[0053] Form multiple cut images from the complete image distributed with multiple instance objects. Each cut image includes an instance object. Then, input the cut image into the trained first instance segmentation model for inferring and predicting the category and the first polygon annotation information of the instance object in the cut image to be predicted, so as to infer and predict the first pseudo-label annotation result of the instance object in the cut image.
[0054] Among them, the first pseudo-label annotation result includes the first polygon annotation information and the category of the instance object. The first polygon annotation information includes the contour boundary coordinates of the instance object in the cropped image.
[0055] In some embodiments, the instance object can be a prohibited item in a security inspection scenario, such as a knife, a gun, a pressure tank, a lighter, etc. Specifically, it is not limited here. Therefore, the first polygon annotation information includes the contour boundary coordinates formed by the dividing line between the corresponding prohibited item and its background in the cropped image. The image can be an X-ray image collected by an X-ray acquisition device.
[0056] It should be emphasized that the first polygon annotation information includes the contour boundary coordinates of the instance object in the cropped image, rather than the contour boundary coordinates in the corresponding complete image.
[0057] It can be understood that in this step, a complete image including multiple instance objects is formed into multiple cropped images including the instance objects, and then input into the trained first segmentation instance model to obtain the first pseudo-label annotation result. This reduces the manual annotation cost, and since the cropped image is a local image of the complete image, compared with the complete image, using the local cropped image can obtain more precise first polygon annotation information of the image accuracy.
[0058] 102: Determine the second polygon annotation information of the instance object fused in the image according to the first polygon annotation information and the first position information. Among them, the second polygon annotation information represents the contour boundary coordinates of the instance object in the image, and the first position information represents the position coordinates of the cropped image in the image;
[0059] Merge the first polygon annotation information and the first position information to obtain the second polygon annotation information of the instance object fused in the image. Among them, the first position information represents the position coordinates of the cropped image in the image, which is recorded and saved during the process of forming the cropped image from the image. The second polygon annotation information includes the contour boundary coordinates of the instance object in the image.
[0060] It can be understood that the contour boundary coordinates of the first polygon annotation information and the second polygon annotation information represent the coordinates of the instance object under different reference objects.
[0061] 103: Train a second instance segmentation model for inferring and predicting the instance segmentation result of the instance object in the target image according to the first training set. Among them, the first training set includes the second polygon annotation information, the category of the instance object, and the image.
[0062] Create the second polygon annotation information, the category of the instance object inferred and predicted, and the image as the first training set, and use the first training set to train a second instance segmentation model for inferring and predicting the instance segmentation result of the instance object in the target image. Among them, the input feature of the training set is the image, and the output label of the training set is the second polygon annotation information and the category of the instance object inferred and predicted by the first instance segmentation model.
[0063] Understandably, the training method of this application uses a more refined local shear map and the first instance segmentation model to infer and predict the category of the instance object in the local shear map and the first polygon annotation information. Then, based on the first polygon annotation information and the first position information, the second polygon annotation information including the contour boundary coordinates of the instance object in the complete image is determined. Finally, the second polygon annotation information, the category of the instance object obtained by the first instance segmentation model, and the complete image are used as the first training set to train the second instance segmentation model. Since the second instance segmentation model uses the complete image as the prediction input, the second instance segmentation model trained in this application can predict the pixel-level second polygon annotation information and its category of the instance objects distributed in the complete image, thus achieving the recognition accuracy of the instance objects under complex backgrounds, overlapping occlusions, and multi-scale changes.
[0064] In some embodiments, input the target image into the trained second instance segmentation model, and then obtain the category of the instance object inferred and predicted by the second instance segmentation model for the target image and the corresponding second polygon annotation information.
[0065] Please refer to Figure 2 , Figure 2 shows how to train the first instance segmentation model, including:
[0066] 201: Train the first instance segmentation model according to the second training set, where the second training set includes a sample shear map including an instance object, the category of the instance object in the sample shear map, and the rectangular box annotation information of the sample shear map.
[0067] First, for each of multiple images, obtain the shear map including the instance object cropped from it, and obtain the category of the instance object in the shear map and the rectangular box annotation information of the shear map. Then, use the associated shear map, the category of the instance object in the shear map, and the rectangular box annotation information of the shear map as the second training set to train the first instance segmentation model. The first instance segmentation model is used to infer and predict the category of the instance object in the to-be-predicted shear map and the first polygon annotation information including the contour boundary coordinates of the instance object in the shear map. The rectangular box annotation information includes the size information representing the shear map and the coordinate information of the edges of the shear map.
[0068] It should be noted that the cropped image in this application is a partial image of the image. The image in this application is obtained by splicing the first cropped image (abbreviated as the cropped image) including the instance object and the second cropped image not including the instance object. That is, the difference between the cropped image and the image is that the image is a complete image that includes multiple instance objects distributed at multiple positions in the image, while the cropped image includes a single instance object or multiple instance objects. Therefore, since the cropped image discards the interfering image content that interferes with the corresponding instance object compared to the image, that is, discards the aforementioned second cropped image, a first instance segmentation model for more accurate instance object segmentation that can exclude interfering image content is obtained.
[0069] It can be understood that there is an association relationship among the images, cropped images, and instance objects used in the training of this application. In some preferred embodiments, the number of cropped images is the same as the number of instance objects, that is, each cropped image includes a single instance object, and each image includes at least one pair of cropped images and instance objects. In some embodiments, the training of the first instance segmentation model can be obtained by the boxlevelset algorithm, which uses the rectangular box annotation information of the cropped image, that is, the contour boundary curve of the instance object in the cropped image is inferred and predicted from the edge coordinates of the cropped image through the level set evolution function, that is, the first polygon annotation information, thereby generating high-quality instance segmentation pseudo-labels.
[0070] Please refer to Figure 3 , Figure 3 Further, step 102 is shown, which is used to simply and quickly convert the coordinates of the contour boundary curve of the instance object in the cropped image to the coordinates of the contour boundary curve of the instance object in the original image, so as to achieve the simple and accurate fusion of the instance object inferred and predicted by the first instance segmentation model into the corresponding image, including:
[0071] 301: Calculate the sum value of each of the contour boundary coordinates of the first polygon annotation information and the position coordinates of the first position information to obtain the second polygon annotation information of the instance object in the image.
[0072] Based on the corresponding coordinate component dimension, calculate the sum value of each contour boundary coordinate of the first polygon annotation information and the position coordinates of the first position information, thereby obtaining the second polygon annotation information of the instance object in the image.
[0073] Preferably, the first position information further represents the position coordinates of the upper left corner of the cropped image in the image. Similarly, the first polygon annotation information forms the position coordinates of the instance object in the cropped image based on the upper left corner of the cropped image. In this way, when obtaining the second polygon annotation information, the computational resource pressure caused by complex calculations is avoided, and fast and simple coordinate conversion is achieved, realizing the simple and accurate regression fusion of the annotation information of the instance object in the cropped image into the annotation information of the image.
[0074] Exemplarily, the first polygon annotation information predicted and inferred by the first instance segmentation model is [(x1′, y1′), (x′2, y2′), …, (x′ n , y n ′)], and the first position information is (x1, y1). Then the second polygon annotation information is [(x1′ + x1, y1′ + y1), (x′2 + x1, y2′ + y1), …, (x′ n + x1, y n ′ + y1)].
[0075] In some embodiments, to obtain a more accurate second instance segmentation model, after the cutouts of multiple images are inferred and predicted by the first instance segmentation model, a first training set is filtered according to the first filtering condition, so as to reduce the situation where the training set for training the second instance segmentation model generated by the first instance segmentation model is inaccurate due to the performance error of the first instance segmentation model.
[0076] Among them, the first filtering condition includes confidence filtering for the categories of instance objects inferred and predicted by the first instance segmentation model, quantity consistency filtering for the number of instance objects inferred and predicted by the first instance segmentation model and the actual number of instance objects in the corresponding cutout, and IOU filtering processed according to the second polygon annotation information and the rectangular box annotation information of the corresponding cutout, where the rectangular box annotation information includes, in addition to the first position information, second position information indicating the position coordinates of the cutout in the image.
[0077] In some embodiments, the first filtering condition may be at least one filtering category of the combination of the above filtering categories.
[0078] Please refer to Figure 4 , Figure 4 which shows the process of the aforementioned confidence filtering, including:
[0079] 401: Obtain the first confidence of the categories of each instance object predicted by the first instance segmentation model for the cutouts of each image of multiple images;
[0080] The first instance segmentation model will infer and generate the first confidence during the inference and prediction process, which represents the probability of the category of the instance object in the cutout inferred and predicted by the first instance segmentation model. Therefore, the first confidence of the categories of the instance objects in the cutouts of each image of multiple images can be obtained.
[0081] 402: Compare the first confidence with the confidence threshold to obtain the first training set.
[0082] For each of the multiple first confidence levels, in response to the first confidence level being greater than or equal to the confidence threshold, determine the first confidence level as the target confidence level and determine the associated image, the second polygon annotation information, and the category of the instance object inferred and predicted by the first instance segmentation model as the first training set.
[0083] Preferably, the confidence threshold is preset to 0.9.
[0084] Please refer to Figure 5 , Figure 5 which shows the process of the aforementioned IOU screening, including:
[0085] 501: Calculate the IOU based on the second polygon annotation information of the instance objects inferred and predicted for each of the respective images of the multiple images, and the rectangular box annotation information of the corresponding clipped images, to obtain the IOU of the instance objects inferred and predicted for each of the respective images of the multiple images;
[0086] After obtaining the second polygon annotation information of the instance objects obtained by inferring and predicting the clipped images of each of the multiple images by the first instance segmentation model, and the rectangular box annotation information of the corresponding clipped images, further, obtain the maximum coordinates and minimum coordinates of the second polygon annotation information, and then, calculate the IOU based on the maximum coordinates and minimum coordinates, and the first position information and second position information of the rectangular box annotation information of the corresponding clipped images.
[0087] Among them, the difference between the first position information and the second position information of the rectangular box annotation information of the clipped image represents the size of the clipped image. Preferably, the first position information represents the position coordinates of the upper left corner of the clipped image in the image, and the second position information represents the position coordinates of the lower right corner of the clipped image in the image.
[0088] Exemplarily, the contour boundary coordinates of the instance object of the second polygon annotation information in the image are [(x1′ + x1, y1′ + y1), (x′2 + x1, y2′ + y1), …, (x′ n + x1, y n ′ + y1)], then the minimum coordinates and maximum coordinates are determined from them as [((x i ′ + x1) min , (y i ′ + y1) min ), ((x i ′ + x1) max , (y i ′ + y1)max].
[0089] 502: Obtain multiple target IOU values that meet the IOU threshold condition from the IOU values of the instance objects inferred and predicted from each of the multiple images, and further obtain the first training set associated with the multiple target IOU values.
[0090] Compare each of the calculated IOU values in step 501 with the IOU threshold to obtain target IOU values that are greater than or equal to the IOU threshold, and further determine that the images, the second polygon annotation information, and the categories of the instance objects inferred and predicted corresponding to the target IOU values are the first training set. Preferably, the IOU threshold is preset to 0.9.
[0091] Furthermore, in some embodiments, after the IOU screening, it is also determined whether the categories of the instance objects in the manually annotated cut-out images are consistent with the categories of the instance objects predicted by the first instance segmentation model, so as to exclude the inconsistent category cases in the first training set obtained by the IOU screening to obtain an updated first training set.
[0092] Please refer to Figure 6 , Figure 6 shows the quantity consistency screening process, which is used to reduce the probability of poor accuracy in training the second instance segmentation model in the later stage due to the performance error of the first instance segmentation model, resulting in the inconsistent quantity of the instance objects predicted by the first instance segmentation model and the actual quantity of the instance objects in the cut-out image, including:
[0093] 601: Obtain the number of instance objects after the cut-out images of each of the multiple images are inferred and predicted by the first instance segmentation model;
[0094] For the cut-out image of each of the multiple images, obtain the number of instance objects inferred and predicted by the first instance segmentation model.
[0095] 602; Compare the number of instance objects predicted by the first instance segmentation model with the actual number of instance objects in the corresponding cut-out image to obtain the first training set.
[0096] For the number of instance objects obtained by inference and prediction of each cut-out image, if it is equal to the actual number of instance objects in the corresponding cut-out image, then exclude the associated image in the multiple images to obtain the target image, and further determine the target image, the categories of the instance objects inferred and predicted by the first instance segmentation model corresponding to it, and the second polygon annotation information of the instance objects as the first training set.
[0097] Please refer to Figure 7 , Figure 7 shows the overview process of how the second instance segmentation model of the present application is trained, including:
[0098] 701: Train a first instance segmentation model for inferring and predicting the first polygon annotation information and category of instance objects in the cropped image according to the second training set, where the second training set includes the cropped image of the instance object, the category of the instance object, and the rectangular box annotation information of the cropped image;
[0099] This step is similar to the aforementioned step 201, and will not be elaborated here specifically.
[0100] 702: Input the cropped image into the first instance segmentation model to infer and predict the first pseudo-label annotation result of the instance object in the cropped image, where the first pseudo-label annotation result includes the category of the instance object and the first polygon annotation information;
[0101] This step is similar to the aforementioned step 101, and will not be elaborated here specifically.
[0102] 703: Determine the second polygon annotation information representing the position coordinates of the instance object in the original image according to the first polygon annotation information and the first position information representing the position coordinates of the cropped image in the original image;
[0103] This step is similar to the aforementioned step 102, and will not be elaborated here specifically.
[0104] 704: Determine the first training set according to the first screening condition, where the first training set includes the associated image, the second polygon annotation information, and the category of the instance object inferred and predicted by the first instance segmentation model.
[0105] This step is similar to the aforementioned Figure 4 , Figure 5 and Figure 6 embodiments, and will not be elaborated here specifically.
[0106] It can be understood that when the first screening condition is the aforementioned screening category combination situation, the first training set for training the second instance segmentation model is the intersection of the respective first training sets of the corresponding combined components.
[0107] In some embodiments, the first training set is obtained by successively performing Figure 4 , Figure 5 and Figure 6 embodiments.
[0108] 705: Train a second instance segmentation model for inferring and predicting the instance segmentation result of the instance object in the target image according to the first training set, where the instance segmentation result includes the category of the instance object in the target image and the second polygon annotation information.
[0109] After obtaining the first training set, that is, when the input feature is an image and the output labels are the second polygon annotation information of the instance object and the category of the instance object inferred and predicted by the first instance segmentation model, use it to train a preset second instance segmentation model to obtain a second instance segmentation model that meets the preset accuracy conditions.
[0110] In some embodiments, the second instance segmentation model is the YOLOv8-seg model. During its training process, after every 20 training epochs, use the currently trained model to infer the unlabeled data and regenerate new pseudo-label results. Then use the newly generated pseudo-label results as the next training set and continue the training for the next 20 epochs. Through the above multiple iterations, a second instance segmentation model with the preset accuracy is obtained.
[0111] It can be understood that in this application, the pseudo-label annotation results are generated by the first instance segmentation model, which reduces the manual annotation cost. And because the first instance segmentation model uses local cropped images including instance objects from the complete image for inference and prediction and predicts with non-complete images, it reduces the interference of multiple instance objects and complex background images in the complete image, thus generating more accurate pixel-level pseudo-label results. Further, perform a merging process of regressing the pseudo-label results of the local cropped images to the complete image to obtain the second polygon annotation information. Finally, train the second instance segmentation model with the associated complete image, the second polygon annotation information, and the category of the instance object inferred and predicted by the first instance segmentation model, realizing that the second instance segmentation model can directly infer and predict the categories of multiple instance objects distributed in the complete image including multiple instance objects and complex background interference image content and the second polygon annotation information, that is, improving the recognition accuracy of instance objects under complex backgrounds, overlapping occlusions, and multi-scale changes.
[0112] Please refer to Figure 8 , Figure 8 which provides a schematic diagram of an image instance segmentation training device, including:
[0113] A first input module 801, configured to input a cropped image of the image into the first instance segmentation model to infer and predict the first pseudo-label annotation result of the instance object in the cropped image of the image, where the first pseudo-label annotation result includes the first polygon annotation information and the category of the instance object, and the first polygon annotation information represents the contour boundary coordinates of the instance object in the cropped image;
[0114] A determination module 802, configured to determine second polygon annotation information of the instance object fused in the image according to the first polygon annotation information and the first position information, where the second polygon annotation information represents contour boundary coordinates of the instance object in the image, and the first position information represents position coordinates of the clipped image in the image;
[0115] A training module 803, configured to train a second instance segmentation model for inferring and predicting an instance segmentation result of an instance object in a target image according to a first training set, where the first training set includes the second polygon annotation information, the category of the instance object, and the image.
[0116] Please refer to Figure 9 , Figure 9 There is provided a schematic diagram of an image instance segmentation inference and prediction device, including:
[0117] A second input module 901, configured to input a target image to the second instance segmentation model;
[0118] An acquisition module 902, configured to acquire the category of the instance object and corresponding second polygon annotation information obtained by the second instance segmentation model through inference and prediction of the target image.
[0119] This application further includes a schematic diagram of an electronic device. Please refer to Figure 10 including: a memory 1001 and a processor 1002; wherein, the memory is used to store programs; the processor is used to execute the programs in the memory, including executing any one of the methods in the foregoing embodiments;
[0120] Optionally, the electronic device further includes a bus system for connecting the memory and the processor, so that the memory and the processor can communicate with each other.
[0121] The processor may be a Central Processing Unit (CPU), and the processor may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0122] The memory can be an internal storage unit of the cloud device in some embodiments, such as the hard disk or memory of the cloud device. The memory can also be an external storage device of the cloud device in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the cloud device. Further, the memory can also include both the internal storage unit and the external storage device of the cloud device. The memory is used to store an operating system, application programs, a BootLoader, data, and other programs, such as the program code of a computer program. The memory can also be used to temporarily store the data that has been output or will be output.
[0123] This application also includes a computer-readable storage medium, which stores program instructions internally, and the program instructions are executed by a processor to implement the method in any one of the foregoing embodiments.
[0124] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-described embodiment methods of this application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-described method embodiments can be implemented.
[0125] Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc, etc.
[0126] The above is only the embodiment of this application, and does not limit the patent protection scope of this application. All equivalent structural or equivalent process transformations made by using the content of the specification and drawings of this application, or directly or indirectly applied in other related technical fields, are equally included in the patent protection scope of this application.
Claims
1. An instance segmentation training method for images, characterized in that, Applied to the security inspection scenario, the image includes an X-ray image collected by an X-ray acquisition device, and the method includes: Inputting a cropped image into a first instance segmentation model to infer and predict a first pseudo-label annotation result of instance objects in the cropped image of the image. Wherein, the first pseudo-label annotation result includes first polygon annotation information and the category of the instance object, and the first polygon annotation information represents the contour boundary coordinates of the instance object in the cropped image; Determining second polygon annotation information of the instance object fused in the image according to the first polygon annotation information and first position information. Wherein, the second polygon annotation information represents the contour boundary coordinates of the instance object in the image, and the first position information represents the position coordinates of the cropped image in the image; Training a second instance segmentation model for inferring and predicting the instance segmentation result of instance objects in a target image according to a first training set. Wherein, the first training set includes the second polygon annotation information, the category of the instance object, and the image; 2. The method according to claim 1, characterized in that, The method further includes: Inputting a target image into the second instance segmentation model; Obtaining the category of the instance object of the target image inferred and predicted by the second instance segmentation model and the corresponding second polygon annotation information; 3. The method according to claim 1, characterized in that, The method further includes: Training the first instance segmentation model according to a second training set. Wherein, the second training set includes a sample cropped image of an instance object, the category of the instance object in the sample cropped image, and the rectangular box annotation information of the sample cropped image; 4. The method according to claim 1, wherein The determining the second polygon annotation information of the instance object fused in the image according to the first polygon annotation information and first position information includes: Calculating the sum value of each of the contour boundary coordinates of the first polygon annotation information and the position coordinates of the first position information to obtain the second polygon annotation information of the instance object in the image; 5. The method according to claim 1, wherein The first training set is a training set obtained by screening according to a first screening condition after the cropped images of multiple images are inferred and predicted by the first instance segmentation model; Wherein, the first screening condition includes IOU screening processed according to the second polygon annotation information and the rectangular box annotation information of the corresponding cropped image. The rectangular box annotation information includes the first position information and second position information representing the position coordinates of the cropped image in the image; 6. The method according to claim 5, wherein The first screening condition further includes confidence screening of the category of the instance object inferred and predicted by the first instance segmentation model, or, quantity consistency screening of the number of instance objects inferred and predicted by the first instance segmentation model and the number of actual instance objects in the corresponding cropped image; 7. The method according to claim 5, characterized in that, The process of the IOU screening includes: Calculating the IOU according to the second polygon annotation information of the instance objects inferred and predicted for each image of the multiple images and the rectangular box annotation information of the corresponding cropped image to obtain the IOU of the instance objects inferred and predicted for each image of the multiple images; From the IOU of the instance objects respectively inferred and predicted from each of the multiple images, obtain multiple target IOUs that meet the IOU threshold condition, and further obtain the first training set associated with the multiple target IOUs.
8. The method according to claim 7, wherein The calculating the IOU according to the second polygon annotation information of the instance objects respectively inferred and predicted from each of the multiple images and the rectangular box annotation information of the corresponding clipped images includes: Obtain the maximum coordinates and minimum coordinates in the second polygon annotation information of the instance objects respectively inferred and predicted from each of the multiple images; Calculate the IOU according to the maximum coordinates and minimum coordinates and the rectangular box annotation information of the corresponding clipped images, wherein the difference between the first position information and the second position information represents the size of the clipped image.
9. An electronic device, comprising a memory and a processor, characterized in that, A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that, Program instructions are stored therein, and the program instructions are executed by the processor to implement the method according to any one of claims 1 to 8.