Material sorting device and system based on instance segmentation

Abstract

The present invention discloses a material sorting device and system based on instance segmentation, which relates to the technical field of instance segmentation. The device includes: an instance segmentation module for performing instance segmentation on a material image to obtain a plurality of bounding boxes and corresponding segmentation masks, segmentation thresholds, and confidence scores, where the confidence score represents the confidence that the bounding box contains the material in the material image; a segmentation accuracy improvement module connected to the instance segmentation module for selecting a first target bounding box from the plurality of bounding boxes according to the segmentation masks, segmentation thresholds, and confidence scores corresponding to the respective bounding boxes; and a sorting module connected to the segmentation accuracy improvement module for sorting out non-compliant materials in the material image according to the first target bounding box and the target segmentation mask, where the target segmentation mask is the segmentation mask corresponding to the first target bounding box.

Description

Technical Field

[0001] The present invention relates to the technical field of instance segmentation, and particularly to a material sorting device and system based on instance segmentation. Background Art

[0002] In the production process of disposable wooden chopsticks and bamboo chopsticks, it is necessary to sort the chopsticks that meet the standard length, and the longer or shorter chopsticks need to be removed. However, the sorting technologies in the related art usually adopt manual sorting, which has a large workload and low efficiency. Summary of the Invention

[0003] The present invention aims to at least solve one of the technical problems in the related art to some extent. For this purpose, the first object of the present invention is to propose a material sorting device based on instance segmentation to improve the sorting efficiency.

[0004] The second object of the present invention is to propose a material sorting system based on instance segmentation.

[0005] To achieve the above object, an embodiment of the first aspect of the present invention proposes a material sorting device based on instance segmentation. The device includes: an instance segmentation module for performing instance segmentation on a material image to obtain a plurality of bounding boxes and corresponding segmentation masks, segmentation thresholds, and confidence scores, where the confidence score represents the confidence that the bounding box contains the material in the material image; a segmentation accuracy improvement module connected to the instance segmentation module for selecting a first target bounding box from the plurality of bounding boxes according to the segmentation masks, segmentation thresholds, and confidence scores corresponding to the respective bounding boxes; and a sorting module connected to the segmentation accuracy improvement module for sorting out non-compliant materials in the material image according to the first target bounding box and a target segmentation mask, where the target segmentation mask is the segmentation mask corresponding to the first target bounding box.

[0006] In addition, the material sorting device based on instance segmentation according to the embodiment of the present invention may further have the following additional technical features:

[0007] In an embodiment of the present invention, the instance segmentation module includes: a feature extraction sub-module for extracting features from the material image to obtain a feature map; and a segmentation sub-module connected to the feature extraction sub-module for performing instance segmentation on the feature map to obtain a plurality of the bounding boxes and corresponding segmentation masks, segmentation thresholds, and confidence scores.

[0008] In an embodiment of the present invention, the feature extraction sub-module includes a first pooling layer, a first convolutional layer, a second pooling layer, a second convolutional layer, a third pooling layer, a third convolutional layer, a fourth pooling layer, a fourth convolutional layer, a fifth pooling layer, a first upsampling layer, a second upsampling layer, a fifth convolutional layer, a sixth convolutional layer, and a first connection layer, which are connected in sequence. The feature extraction module further includes a second connection layer and a third connection layer. The output ends of the third pooling layer and the second upsampling layer are also connected to the input end of the third connection layer. The output ends of the fourth pooling layer and the fifth convolutional layer are also connected to the input end of the second connection layer. The output end of the fifth pooling layer is also connected to the input end of the first connection layer. Among them, the first connection layer is used to output a first feature map, the second connection layer is used to output a second feature map, and the third connection layer is used to output a third feature map.

[0009] In an embodiment of the present invention, the segmentation accuracy improvement module includes: a multiplication sub-module, which is used to multiply the segmentation threshold and the segmentation mask corresponding to each bounding box; a convolution and pooling sub-module, which is used to perform convolution and pooling processing on each multiplication result to obtain a first score value corresponding to the multiplication result, where M is a positive integer greater than 1; a first target bounding box selection sub-module, which is used to obtain a second score value according to each bounding box and a first preset box for each bounding box, and obtain a final score value according to the first score value, the second score value, and the target confidence score corresponding to the bounding box, and determine the bounding box as the first target bounding box when the final score value is greater than a first preset score threshold.

[0010] In an embodiment of the present invention, the first target bounding box selection sub-module is specifically used to: substitute the first score value, the second score value, and the target confidence score into a pre-designed formula to obtain the final score value.

[0011] In an embodiment of the present invention, the sorting module is specifically used to: obtain the size information of the target material according to the first target bounding box and the target segmentation mask, and determine that the target material is a non-compliant material when the size information is not within a preset size range, where the target material is the material in the material image that is included in the first target bounding box.

[0012] In an embodiment of the present invention, the device further includes: a comparison module, connected to the instance segmentation module, for comparing the confidence scores corresponding to each of the bounding boxes with a second preset score threshold, and taking the confidence scores greater than the second preset score threshold as first intermediate scores; a sorting module, connected to the comparison module, for arranging the first intermediate scores in descending order to obtain a sorted score sequence in descending order, and selecting the first preset number of scores from the score sequence as second intermediate scores; a selection module, connected to the sorting module and the segmentation accuracy improvement module, for obtaining the intersection over union (IoU) of each intermediate bounding box and a first preset box, and selecting at least one second target bounding box from the intermediate bounding boxes according to the IoU and the second intermediate scores, so that the segmentation accuracy improvement module selects the first target bounding box from the second target bounding boxes, where the intermediate bounding box is the bounding box corresponding to the second intermediate score.

[0013] In an embodiment of the present invention, the device further includes: an update module, for obtaining a first loss value according to the first target bounding box and the target segmentation mask, obtaining a second loss value according to the difference between the first loss value and a preset standard value, and updating at least one of the instance segmentation module, the segmentation accuracy improvement module, and the sorting module according to the second loss value.

[0014] In an embodiment of the present invention, the update module is specifically configured to: obtain a third loss value according to the first target bounding box and a third preset box, obtain a fourth loss value according to the target segmentation mask and a preset ground truth mask, and obtain the first loss value according to the third loss value and the fourth loss value.

[0015] To achieve the above object, an embodiment of the second aspect of the present invention provides a material sorting system based on instance segmentation. The system includes a track transmission subsystem, an imaging subsystem, and a jet ejection subsystem. The track transmission subsystem is used to move the material through the track. The imaging subsystem is used to photograph the material and sort out non-compliant materials according to the obtained material image. The jet ejection subsystem is used to jet out the non-compliant materials moving to a preset jet-off position from a preset material moving path, so as to achieve material sorting. Wherein, the imaging subsystem includes a camera and the above-mentioned material sorting device based on instance segmentation. The camera is used to photograph the material at a preset photographing position to obtain the material image. The preset photographing position is located on the preset material moving path and between the preset jet-off position and the starting moving position of the material on the track. The material sorting device based on instance segmentation is used to sort out the non-compliant materials according to the material image.

[0016] According to the material sorting device and system based on instance segmentation of the embodiments of the present invention, an instance segmentation module is provided for performing instance segmentation on a material image to obtain a plurality of bounding boxes and corresponding segmentation masks, segmentation thresholds, and confidence scores for the bounding boxes, where the confidence score represents the confidence that the bounding box contains the material in the material image; a segmentation accuracy improvement module is connected to the instance segmentation module for selecting a first target bounding box from the plurality of bounding boxes according to the segmentation masks, segmentation thresholds, and confidence scores corresponding to each bounding box; a sorting module is connected to the segmentation accuracy improvement module for sorting out non-compliant materials in the material image according to the first target bounding box and a target segmentation mask, where the target segmentation mask is the segmentation mask corresponding to the first target bounding box. Thus, automatic material sorting based on instance segmentation can be achieved, with high efficiency and low manual workload.

[0017] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a structural block diagram of a material sorting device based on instance segmentation according to an embodiment of the present invention;

[0019] Figure 2 is a structural schematic diagram of a material sorting device based on instance segmentation according to an example of the present invention;

[0020] Figure 3 is a structural schematic diagram of a material sorting device based on instance segmentation according to another example of the present invention;

[0021] Figure 4 is a structural schematic diagram of a material sorting device based on instance segmentation according to yet another example of the present invention;

[0022] Figure 5 is a structural schematic diagram of a material sorting device based on instance segmentation according to yet another example of the present invention;

[0023] Figure 6 is a structural schematic diagram of a material sorting device based on instance segmentation according to yet another example of the present invention;

[0024] Figure 7 is a working flow chart of a material sorting device based on instance segmentation according to an example of the present invention;

[0025] Figure 8 is a structural schematic diagram of a material sorting system based on instance segmentation according to an embodiment of the present invention;

[0026] Figure 9 is a structural block diagram of an imaging subsystem according to an embodiment of the present invention. Detailed implementation manners

[0027] The following describes an instance segmentation-based material sorting device and system according to an embodiment of the present invention with reference to the accompanying drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described with reference to the accompanying drawings are exemplary and should not be construed as limiting the present invention.

[0028] Figure 1 It is a structural block diagram of an instance segmentation-based material sorting device according to an embodiment of the present invention.

[0029] As Figure 1 shown, the instance segmentation-based material sorting device 400 includes: an instance segmentation module 500, configured to perform instance segmentation on a material image to obtain a plurality of bounding boxes and corresponding segmentation masks, segmentation thresholds, and confidence scores, where the confidence score represents the confidence that the bounding box contains the material in the material image; a segmentation accuracy improvement module 600, connected to the instance segmentation module 500, configured to select a first target bounding box from the plurality of bounding boxes according to the segmentation masks, segmentation thresholds, and confidence scores corresponding to the respective bounding boxes; and a sorting module 700, connected to the segmentation accuracy improvement module 600, configured to sort out non-compliant materials in the material image according to the first target bounding box and a target segmentation mask, where the target segmentation mask is the segmentation mask corresponding to the first target bounding box.

[0030] Specifically, after obtaining the material image, in order to sort out non-compliant materials according to the material image, the above-mentioned instance segmentation-based material sorting device 400 is provided to include an instance segmentation module 500, a segmentation accuracy improvement module 600, and a sorting module 700.

[0031] The above-mentioned instance segmentation module 500 is configured to perform instance segmentation on the material image. Specifically, since the material image contains at least one material, the instance segmentation module 500 performs instance segmentation on the material image, and a plurality of bounding boxes and corresponding segmentation masks, segmentation thresholds, and confidence scores will be obtained. Furthermore, each bounding box and the corresponding segmentation mask, segmentation threshold, and confidence score thereof can be regarded as an instance segmentation result. Each of the above-mentioned bounding boxes has a corresponding segmentation mask, segmentation threshold, and confidence score, and the confidence score represents the confidence that the bounding box corresponding to the score contains the material.

[0032] In practical applications, it may occur that the instance segmentation yields bounding boxes that cannot be used for sorting non-compliant materials. Therefore, it is also necessary to set up a segmentation accuracy improvement module 600. After obtaining multiple instance segmentation results, the instance segmentation results are input into the segmentation accuracy improvement module 600, so that the segmentation accuracy improvement module 600 selects the first target bounding box from multiple bounding boxes according to the segmentation mask, segmentation threshold, and confidence score corresponding to each bounding box. That is to say, after obtaining multiple bounding boxes, the segmentation accuracy improvement module 600 needs to select the bounding boxes that meet the requirements from them, use them as the first target bounding boxes, and discard the bounding boxes that do not meet the requirements. For example, although theoretically each bounding box contains materials, in practical applications, there may be bounding boxes that do not contain materials. At this time, the bounding box that does not contain materials is the non-compliant bounding box that needs to be discarded.

[0033] After obtaining the first target bounding box, the first target bounding box and the target segmentation mask corresponding to the first target bounding box are input into the sorting module 700. The sorting module 700 obtains the information of the materials contained in each first target bounding box according to the first target bounding box and the target segmentation mask, and thus judges whether the material is a non-compliant material according to the information of the material.

[0034] Thus, it is possible to perform instance segmentation on the material image, and then sort out non-compliant materials according to the instance segmentation results.

[0035] In some embodiments of the present invention, the instance segmentation module 500 includes: a feature extraction sub-module for extracting features from the material image to obtain a feature map; a segmentation sub-module connected to the feature extraction sub-module for performing instance segmentation on the feature map to obtain multiple bounding boxes and the segmentation masks, segmentation thresholds, and confidence scores corresponding to the bounding boxes.

[0036] Specifically, the instance segmentation module 500 is set to include a feature extraction sub-module and a segmentation sub-module. After obtaining the material image, the instance segmentation module 500 performs instance segmentation on the material image.

[0037] The specific instance segmentation process is as follows: the feature extraction sub-module in the instance segmentation module 500 extracts features from the material image to obtain a feature map, which contains information such as semantic information, spatial information, and detail information. After obtaining the feature map, the segmentation sub-module can segment the feature map to obtain multiple bounding boxes and the segmentation masks, segmentation thresholds, and confidence scores corresponding to each bounding box.

[0038] In some embodiments of the present invention, the feature extraction sub-module includes a first pooling layer, a first convolutional layer, a second pooling layer, a second convolutional layer, a third pooling layer, a third convolutional layer, a fourth pooling layer, a fourth convolutional layer, a fifth pooling layer, a first upsampling layer, a second upsampling layer, a fifth convolutional layer, a sixth convolutional layer, and a first connection layer connected in sequence. The feature extraction module further includes a second connection layer and a third connection layer; the output ends of the third pooling layer and the second upsampling layer are also connected to the input end of the third connection layer, the output ends of the fourth pooling layer and the fifth convolutional layer are also connected to the input end of the second connection layer, and the output end of the fifth pooling layer is also connected to the input end of the first connection layer; wherein, the first connection layer is used to output a first feature map, the second connection layer is used to output a second feature map, and the third connection layer is used to output a third feature map.

[0039] The following will be specifically described with reference to Figure 2 the examples shown below.

[0040] In Figure 2 , C2f1 is the first pooling layer, Conv1 is the first convolutional layer, C2f2 is the second pooling layer, Conv2 is the second convolutional layer, C2f3 is the third pooling layer, Conv3 is the third convolutional layer, C2f4 is the fourth pooling layer, Conv4 is the fourth convolutional layer, C2f5 is the fifth pooling layer, Upsample1 is the first upsampling layer, Upsample2 is the second upsampling layer, Conv5 is the fifth convolutional layer, Conv6 is the sixth convolutional layer, Concat1 is the first connection layer, Concat2 is the second connection layer, and Concat3 is the third connection layer.

[0041] In this example, the size and depth of the output result of the above-mentioned first pooling layer are set to 640×640×3, that is, the size is 640×640 and the depth is 3. The first convolutional layer is 320×320×32, the second pooling layer is 320×320×64, the second convolutional layer is 160×160×128, the third pooling layer is 160×160×128, the third convolutional layer is 80×80×256, the fourth pooling layer is 80×80×256, the fourth convolutional layer is 40×40×512, the fifth pooling layer is 40×40×512, the first upsampling layer is 80×80×256, the second upsampling layer is 160×160×128, the fifth convolutional layer is 80×80×128, the sixth convolutional layer is 40×40×256, the first connection layer is 40×40×762, the second connection layer is 80×80×384, and the third connection layer is 160×160×192. Of course, in actual applications, it is not limited to this and can be adjusted according to actual needs.

[0042] It can be seen that after obtaining the material image with a size of 3600×2400, the material image is first scaled and filled to obtain an image with a size of 640×640, and then the 640×640-sized image is input into the network. The feature extraction sub-module in the network includes a first pooling layer, a first convolutional layer, a second pooling layer, a second convolutional layer, a third pooling layer, a third convolutional layer, a fourth pooling layer, a fourth convolutional layer, a fifth pooling layer, a first upsampling layer, a second upsampling layer, a fifth convolutional layer, a sixth convolutional layer, and a first connection layer connected in sequence. It also includes a second connection layer and a third connection layer. After the above 640×640-sized image is processed by the convolutional layer, pooling layer, and upsampling layer, it is then input into the first connection layer, second connection layer, and third connection layer. The first connection layer, second connection layer, and third connection layer respectively output a first feature map, a second feature map, and a third feature map.

[0043] Continue to refer to Figure 3 the example shown to illustrate the above segmentation sub-module.

[0044] It can be seen that after Concat1, Concat2, and Concat3, a first device Detaction, a second device Mask Confidence, and a third device Proto Struct are also set. Detaction, Mask Confidence, and Proto Struct together constitute the above segmentation sub-module, and each device is used to generate corresponding information according to the first feature map, second feature map, and third feature map. Specifically, Detaction is used to generate a bounding box and a confidence score corresponding to the bounding box according to the first feature map, second feature map, and third feature map. Mask Confidence is used to generate a segmentation threshold according to the first feature map, second feature map, and third feature map. Proto Struct is used to generate a segmentation mask according to the first feature map, second feature map, and third feature map. That is to say, every time Detaction generates a bounding box, Detaction also needs to generate a confidence score corresponding to the bounding box, and Mask Confidence needs to generate a segmentation threshold corresponding to the bounding box, and ProtoStruct needs to generate a segmentation mask corresponding to the bounding box.

[0045] In some embodiments of the present invention, the segmentation accuracy improvement module 600 includes: a multiplication sub-module for multiplying the segmentation threshold corresponding to each bounding box by the segmentation mask for each bounding box; a convolution and pooling sub-module for performing convolution and pooling processing on each multiplication result to obtain a first score value corresponding to the multiplication result, where M is a positive integer greater than 1; a first target bounding box selection sub-module for obtaining a second score value according to each bounding box and a first preset box for each bounding box, and obtaining a final score value according to the first score value, the second score value and the target confidence score corresponding to the bounding box, and determining the bounding box as the first target bounding box when the final score value is greater than the first preset score threshold.

[0046] Specifically, for each bounding box, the segmentation accuracy improvement module 600 needs to obtain the final score value corresponding to the bounding box according to the confidence score, the segmentation threshold and the segmentation mask corresponding to the bounding box.

[0047] The segmentation accuracy improvement module 600 needs to multiply the segmentation mask corresponding to each bounding box by the segmentation threshold for each bounding box, and pass the multiplication result through the convolution and pooling sub-module to obtain a first score value. Moreover, it is also necessary to obtain a second score value according to the differences between the center point, width, height and diagonal length of each bounding box and the first preset box.

[0048] After obtaining the first score value and the second score value, for each bounding box, the final score value can be obtained according to the first score value, the second score value and the confidence score corresponding to the bounding box.

[0049] After obtaining the final score value corresponding to each bounding box, the final score value can be compared with the first preset score threshold to obtain the final score value greater than the first preset score threshold, and the bounding box corresponding to the final score value greater than the first preset score threshold is used as the first target bounding box, and then it is judged whether the material contained in the first target bounding box is non-compliant material according to the first target bounding box and the target segmentation mask.

[0050] The above multiplication sub-module can be implemented by a multiplier. The above convolution and pooling sub-module can be set to include M convolutional layers connected in series and a max pooling layer, where M is a positive integer greater than 1. The above first target bounding box selection sub-module can be implemented by an electronic device storing a computer program, pre-set software, digital circuits, etc., as long as it can implement the disclosed content of the above positive sample selector.

[0051] The following is described in conjunction with Figure 4 the example shown.

[0052] In Figure 4Among them, 3 is the above-mentioned multiplication sub-module, 501 is the above-mentioned first target bounding box selection sub-module, and Output includes the above-mentioned first target bounding box and the confidence score, segmentation threshold, and segmentation mask corresponding to the first target bounding box. In this example, the above-mentioned multiplication sub-module is implemented using a multiplier.

[0053] Origin Mask is the operation result of multiplier 3. That is to say, in Figure 4 the specific example shown, it is set that after multiplier 3 performs the operation, it will output an output result of N×160×160, that is, it has N channels and a size of 160×160. Of course, in actual applications, the size and channels of the output result of multiplier 3 can be adjusted according to the actual situation and are not limited to this.

[0054] Figure 4 The multiple Convs in [[ ]] are the above-mentioned M consecutively connected convolutional layers. The M consecutively connected convolutional layers can be set to be the same or different. The M consecutively connected convolutional layers can use the same convolutional layers as the above-mentioned first, second, third, fourth, fifth, and sixth convolutional layers, or different convolutional layers.

[0055] Figure 4 Relu in [[ ]] is the activation function, 4 is the confidence score, 5 is the second score value, and 6 is the first score value. MaxPooling is the above-mentioned max pooling layer.

[0056] That is to say, in Figure 3 the above-mentioned convolution and pooling sub-module includes M Convs, M Relus, and one MaxPooling.

[0057] In some embodiments of the present invention, the first target bounding box selection sub-module is specifically used for: substituting the first score value, the second score value, and the target confidence score into a pre-designed formula to obtain a final score value.

[0058] Specifically, the above-mentioned pre-designed formula can be the following calculation formula:

[0059] final_score = Score1α × Score2 β × Score3γ,

[0060] where final_score is the final score value, Score1 is the target confidence score, Score2 is the second score value, Score3 is the first score value, α is the first preset exponential coefficient, β is the second preset exponential coefficient, and γ is the third preset exponential coefficient.

[0061] After calculating the final score value final_score, the closer the final score value is to 1, the higher the matching degree and the more likely it is a positive sample. Therefore, for each first target bounding box, compare its corresponding final score value with the second preset score threshold. When the final score value corresponding to the first target bounding box is greater than the second preset score threshold, determine that the first target bounding box is a positive sample.

[0062] In some embodiments of the present invention, the sorting module 700 is specifically configured to: obtain the size information of the target material according to the first target bounding box and the target segmentation mask. When the size information is not within the preset size range, determine that the target material is a non-compliant material, where the target material is the material in the material image that is included in the second target bounding box. For example, if the material is chopsticks, the size information of the chopsticks included in the second target bounding box can be obtained according to the second target bounding box and the segmentation mask corresponding to the second target bounding box, so as to obtain the length of the chopsticks according to the size information. If the length of a certain chopstick is significantly too short, the chopstick can be used as a non-compliant chopstick.

[0063] In some embodiments of the present invention, the material sorting device 400 based on instance segmentation further includes: a comparison module, connected to the instance segmentation module 500, for comparing the confidence scores corresponding to each bounding box with the second preset score threshold, and taking the confidence scores greater than the second preset score threshold as the first intermediate scores; a sorting module, connected to the comparison module, for arranging the first intermediate scores in descending order to obtain a descending score sequence, and selecting the first preset number of scores from the score sequence as the second intermediate scores; a selection module, connected to the sorting module and the segmentation accuracy improvement module 600, for obtaining the intersection over union of each intermediate bounding box and the first preset box, and selecting at least one second target bounding box from the intermediate bounding boxes according to the intersection over union and the second intermediate scores, so that the segmentation accuracy improvement module 600 selects the first target bounding box from the second target bounding boxes, where the intermediate bounding box is the bounding box corresponding to the second intermediate score.

[0064] Specifically, a comparison module, a sorting module, and a selection module are further set. The output result of the instance segmentation module 500 first passes through the processing of the comparison module, the sorting module, and the selection module to obtain the second target bounding box, so that the segmentation accuracy improvement module 600 selects the first target bounding box from the second target bounding boxes. That is to say, the output result of the instance segmentation module 500 is processed by the comparison module, the sorting module, and the selection module, and then processed by the segmentation accuracy improvement module 600.

[0065] After generating the segmentation mask, confidence score, and segmentation threshold corresponding to each bounding box, the comparison module is used to traverse all the confidence scores generated by the segmentation sub-module, compare the confidence scores with the second preset score threshold, filter out the confidence scores less than the second preset score threshold, and use the remaining confidence scores as the first intermediate scores.

[0066] After filtering the confidence scores, the sorting module sorts the first intermediate scores in descending order according to the specific score values, and then selects the top_k first intermediate scores to obtain the second intermediate scores. This top_k is the above-mentioned preset number, that is, the number of the second intermediate scores is top_k.

[0067] After obtaining the top_k second intermediate scores, the corresponding bounding boxes of the second intermediate scores can be used as the intermediate bounding boxes.

[0068] It should be noted that the above sorting of the first intermediate scores in descending order according to the specific score values is carried out by category. Specifically, since there may be more than one type of material in the material image, after obtaining the first intermediate scores, it is necessary to sort all the first intermediate scores of each classification in descending order according to the score values, and select the top_k first intermediate scores of this classification. This classification refers to the category of the material. For example, if the material includes chopsticks, spoons, and diamonds, then chopsticks are one classification, spoons are one classification, and diamonds are one classification. Taking chopsticks and spoons as an example, the sorting module needs to sort all the first intermediate scores related to chopsticks in descending order and select the top_k first intermediate scores as the second intermediate scores under the classification of chopsticks. The sorting module also needs to sort all the first intermediate scores related to spoons in descending order and select the top_k first intermediate scores as the second intermediate scores under the classification of spoons. That is to say, under each classification, there are top_k intermediate bounding boxes.

[0069] After obtaining the second intermediate scores, the selection module determines the intermediate bounding boxes according to the second intermediate scores, calculates the intersection over union (IoU) of each intermediate bounding box and the first preset box, and thus selects at least one second target bounding box from the intermediate bounding boxes according to the IoU and the confidence scores corresponding to the intermediate bounding boxes.

[0070] As an example, the above selection module can work in the following way:

[0071] The selection module first calculates the ratio between the intersection and the union of each intermediate bounding box and the first preset box to obtain the IoU of this intermediate bounding box and the first preset box, and then performs an f-function calculation for the IoU corresponding to each intermediate bounding box.

[0072] After calculating the f function for each intersection over union (IoU), selection is made according to the following formula:

[0073]

[0074] Among them, means that only the IoU corresponding to the above first intermediate score will be carried out, s k is the confidence score corresponding to the k-th bounding box, s i is the second preset score threshold for the i-th classification, means all, f(IoU k，i ) refers to the result obtained by substituting IoU k，i into the above f function calculation, IoU k，i is the IoU corresponding to the k-th intermediate bounding box under the i-th classification. That is to say, after calculating the IoU corresponding to each intermediate bounding box, the minimum value among them is selected, and f(IoU .，i ) is the minimum f function calculation result obtained under the i-th classification, IoU .，i means selection is made according to the IoUs under the i-th classification.

[0075] After calculating f(IoU .，i ), calculation is carried out according to the following formula:

[0076]

[0077] That is to say, after calculating f(IoU .，i ), each eligible f(IoU i.j ) needs to be divided by f(IoU .，i ), and the minimum value decay i of the calculation results is selected.

[0078] Among them, IoU i，j is the IoU corresponding to the j-th sample under the i-th classification. For example, assuming the classification is chopsticks and the image taken of the chopsticks contains n chopsticks, then the value range of j is 1 to n, and f(IoU i，j ) is the calculation result corresponding to the corresponding IoU i，j .

[0079] After selecting the minimum calculation result decay i , substitute this decay i into the following formula for calculation:

[0080] sCores Iou = scores top_k * decay i ,

[0081] Among them, sCores top_k are the above top_k first intermediate scores, that is, the above second intermediate scores. That is to say, for each second intermediate score, its product with decay i needs to be calculated to obtain the IOU score sCores Iou corresponding to this score.

[0082] After calculating the IOU scores scores corresponding to the above top_k second intermediate scores Iou , filter out the values of the IOU scores greater than the preset IOU score threshold, and use the remaining intermediate bounding boxes corresponding to the values as the second target bounding boxes.

[0083] Among them, the f function calculation for the intersection over union corresponding to each intermediate bounding box is: f(iou i，j ) = 1 - iou i，j . Of course, in this step, the iou i，j used for calculation are all the intersection over unions corresponding to the above first intermediate scores.

[0084] It should be noted that the above selection module can be implemented by an electronic device storing a computer program, a preset software, a digital circuit, etc., as long as it can implement the disclosed content of the above first selection sub-module.

[0085] After obtaining at least one second target bounding box, the segmentation accuracy improvement module 600 can select the first target bounding box from the second target bounding boxes. Through the segmentation accuracy improvement module 600, the improvement of the segmentation accuracy can be realized.

[0086] In some embodiments of the present invention, the material sorting device 400 based on instance segmentation further includes: an update module, configured to obtain a first loss value according to the first target bounding box and the target segmentation mask, obtain a second loss value according to the difference between the first loss value and a preset standard value, and update at least one of the instance segmentation module 500, the segmentation accuracy improvement module 600, and the sorting module 700 according to the second loss value.

[0087] Specifically, it is set that the material sorting device further includes an update module, and the update module is configured to substitute positive samples into a preset loss function for calculation, and update at least one of the instance segmentation module 500, the segmentation accuracy improvement module 600, and the sorting module 700 according to the calculation result.

[0088] The update module will be described below in conjunction with a specific embodiment. In this specific example, whether it is the second target bounding box or the third preset box, the upper left corner of it is used as its preset point. Of course, in practical applications, it is not limited to this.

[0089] In this specific embodiment, in order to obtain the first loss value based on the first target bounding box and the target segmentation mask, an update module may be set to obtain a third loss value based on the first target bounding box and a third preset box, obtain a fourth loss value based on the target segmentation mask and a preset true mask, and obtain the first loss value based on the third loss value and the fourth loss value.

[0090] The above-mentioned obtaining the third loss value based on the first target bounding box and the third preset box may be calculated by the following formula:

[0091]

[0092] Where Loss_detect is the above-mentioned third loss value, N is the number of materials in a batch, and the above-mentioned x i is the abscissa of the preset point on the first target bounding box corresponding to the i-th material, and y i is the ordinate of the preset point on the above-mentioned first target bounding box, w i is the horizontal width of the above-mentioned first target bounding box, and h i is the vertical width of the above-mentioned first target bounding box. is the abscissa of the preset point on the above-mentioned third preset box, is the ordinate of the preset point on the above-mentioned third preset box, is the horizontal width of the above-mentioned third preset box, is the vertical width of the above-mentioned third preset box.

[0093] The above-mentioned p ij is the classification value of the material in the first target bounding box by the material sorting device, is the manually marked classification value of the material in the first target bounding box. That is to say, if the update module is to update at least one of the instance segmentation module 500, the segmentation accuracy improvement module 600, and the sorting module 700, it is necessary to first perform manual coordinates on the materials in the first target bounding box, and it is necessary to pre-determine the classification values corresponding to different material classifications.

[0094] The above-mentioned C represents the number of classifications, and λ 1 , λ 2 are two preset weights, and the preset weights can be determined according to the weights of the bounding box loss and the classification loss in the third loss value.

[0095] The above-mentioned obtaining the fourth loss value based on the target segmentation mask and the preset true mask may be calculated by the following formula:

[0096]

[0097] Where Segment_Loss is the fourth loss value, N1 is the number of pixels in the target segmentation mask, and Mj is the true mask value of the j-th pixel, is the above-mentioned preset true mask.

[0098] The above-mentioned obtaining the first loss value based on the third loss value and the fourth loss value can be calculated by the following formula:

[0099] Loss1 = 0.5 × Loss_detect + 0.75 × Segment_Loss,

[0100] where Loss1 is the first loss value.

[0101] After calculating the first loss value, the error between the first loss value and the preset standard value can be compared, and at least one of the instance segmentation module 500, the segmentation accuracy improvement module 600, and the sorting module 700 can be updated according to the error. The above-mentioned updating at least one of the instance segmentation module 500, the segmentation accuracy improvement module 600, and the sorting module 700 can be to update the weights of the convolution kernels included therein. Of course, the updating module can also update the above-mentioned comparison module, sorting module, and selection module.

[0102] The following combines Figure 5 and Figure 6 shown examples for specific illustration.

[0103] In Figure 5 , Labell is the label image, which is a standard image that has been pre-processed with manual instance segmentation and annotation. Furthermore, the above-mentioned preset standard value can be obtained according to this label image, and 500 is the above-mentioned segmentation accuracy improvement module 600. In Figure 5 , Detection Loss is the first operation module, Segment Loss is the second operation module, and Loss is the third operation module. The first operation module is used to calculate the above-mentioned third loss value, the second operation module is used to calculate the above-mentioned fourth loss value, and the third operation module is used to calculate the above-mentioned first loss value and the second loss value. The updating module includes the first operation module, the second operation module, and the third operation module.

[0104] See Figure 6 , Detection Loss calculates the above-mentioned third loss value according to the output of the above-mentioned first target bounding box selection sub-module. Segment Loss calculates the above-mentioned fourth loss value according to the output of the above-mentioned first target bounding box selection sub-module and the output of the above-mentioned multiplication sub-module. For example, it can be set to obtain the true mask value through the output of the multiplication sub-module. It should be noted that Figure 6 is only a specific example, and in practical applications, it is not limited to this.

[0105] That is to say, after obtaining the material image, the material image is first input into the instance segmentation module 500, and the feature extraction submodule in the instance segmentation module 500 performs feature extraction on the material image to obtain a first feature map, a second feature map and a third feature map. The segmentation submodule obtains multiple bounding boxes according to the first feature map, the second feature map and the third feature map, as well as a confidence score, a segmentation threshold and a segmentation mask corresponding to each bounding box.

[0106] Furthermore, each bounding box and the confidence score, segmentation threshold, and segmentation mask corresponding to each bounding box are input into a comparison module, and the comparison module compares each confidence score with a second preset score threshold to obtain a first intermediate score, and then the sorting module sorts each first intermediate score and selects a second intermediate score. The number of the second intermediate scores can be multiple.

[0107] After obtaining the second intermediate scores, since there are multiple second intermediate scores, there are also multiple intermediate bounding boxes corresponding to the second intermediate scores. At this time, the selection module and the segmentation accuracy improvement module 600 can select the first target bounding box from the multiple intermediate bounding boxes. By continuously performing bounding box selection by the selection module and the segmentation accuracy improvement module 600, the segmentation accuracy of instance segmentation can be further improved.

[0108] After obtaining the first target bounding box, the sorting module 700 sorts out non-compliant materials from the materials contained in the first target bounding box according to the first target bounding box and its corresponding target segmentation mask.

[0109] Moreover, after obtaining the first target bounding box, an update module may be used to update at least one of the segmentation module, the segmentation accuracy improvement module 600 , the sorting module 700 and other modules according to the first target bounding box and its corresponding target segmentation mask.

[0110] The above-mentioned sorting module 700 and updating module can be implemented by electronic devices storing computer programs, preset software, digital circuits, etc., as long as the public contents of the above-mentioned sorting module 700 and updating module can be implemented.

[0111] See below Figure 7 The example shown is used for illustration.

[0112] Specifically, first initialize the software and hardware, load the SortNet model, which is the model used to sort out non-compliant materials according to material images included in the material sorting device 400 based on instance segmentation, and then start unloading, and turn on the dual-camera, which is a camera used to capture material images. Of course, a single-camera camera or other camera can also be used to capture the material images.

[0113] If the dual camera fails to turn on, an error message is reported.

[0114] If the dual camera turns on successfully, the front and rear 4096×256 images are taken. That is to say, after the camera is turned on, images are continuously captured and images with a size of 4096×256 are continuously output.

[0115] Furthermore, the captured images are input into the above model, non-compliant materials are inferred, the spray valve delay is calculated, and a spray valve command is sent when the delay time arrives.

[0116] In summary, the material sorting device based on instance segmentation according to the embodiment of the present invention is provided with an instance segmentation module for performing instance segmentation on a material image to obtain a plurality of bounding boxes and corresponding segmentation masks, segmentation thresholds, and confidence scores. Among them, the confidence score represents the confidence that the bounding box contains the material in the material image; a segmentation accuracy improvement module, connected to the instance segmentation module, for selecting a first target bounding box from a plurality of bounding boxes according to the segmentation masks, segmentation thresholds, and confidence scores corresponding to each bounding box; a sorting module, connected to the segmentation accuracy improvement module, for sorting out non-compliant materials in the material image according to the first target bounding box and the target segmentation mask, where the target segmentation mask is the segmentation mask corresponding to the first target bounding box. Thus, automatic material sorting based on instance segmentation can be realized, with high efficiency and low manual workload.

[0117] Furthermore, the present invention proposes a material sorting system based on instance segmentation.

[0118] Figure 8 It is a schematic structural diagram of a material sorting system based on instance segmentation according to an embodiment of the present invention.

[0119] As Figure 8 shown, the material sorting system 100 based on instance segmentation includes a crawler transmission subsystem 200, an imaging subsystem 300, and a jet ejection subsystem 800. The crawler transmission subsystem 200 is used to move the material through the crawler. The imaging subsystem 300 is used to capture the material and sort out non-compliant materials according to the captured material image. The jet ejection subsystem 800 is used to jet out the non-compliant materials that move to a preset jet-off position from a preset material movement path, thereby realizing material sorting.

[0120] Among them, referring to Figure 9, the imaging subsystem 300 includes a camera 401 and the instance segmentation-based material sorting device 400 of the above embodiment. The camera 401 is used to capture the material at a preset shooting position to obtain a material image. The preset shooting position is located on a preset material movement path and between the preset ejection position and the starting movement position of the material on the conveyor belt. The instance segmentation-based material sorting device 400 of the above embodiment is used to sort out non-compliant materials according to the material image.

[0121] The above materials can be chopsticks, spoons, etc.

[0122] The above preset material movement path can be the movement path of the material on the conveyor belt or can also include the movement path of the material on the conveyor belt. Specifically, when the conveyor belt drives the material to move, this movement path can be set as the preset material movement path; or the preset material movement path can also be set to include at least two parts. The first part is that the conveyor belt drives the material to move, and the second part is that the material freely falls after moving to the end of the conveyor belt. At this time, the preset material movement path can be composed of these two parts. Of course, other ways of driving the material to move can also be additionally set after the material completes free fall according to actual needs, so that the preset material movement path is composed of at least three parts.

[0123] The above preset ejection position can be a position on the conveyor belt or other positions. Similarly, the above preset shooting position can also be a position on the conveyor belt or other positions, as long as when the material moves on the preset material movement path, it first passes through the preset shooting position and then passes through the preset ejection position.

[0124] That is to say, Figure 8 It is only a specific embodiment. In this embodiment, the preset material movement path is composed of two parts, namely the path of the conveyor belt driving the material to move and the free fall of the material after moving to the end of the conveyor belt. The above preset shooting position is located on the conveyor belt, and the above preset ejection position is located on the preset material movement path and in the free fall part of the material.

[0125] At this time, the conveyor belt in the conveyor belt transmission subsystem 200 drives the material to move, the camera 401 captures the material image, and the above instance segmentation-based material sorting device 400 sorts out non-compliant materials according to the material image. When the non-compliant materials move to the preset ejection position, the jet ejection subsystem 800 jets the non-compliant materials out of the preset material movement path. In Figure 1 In the shown embodiment, the path of the compliant materials is path 1, and the path of the non-compliant materials is path 2. Since path 1 is included in the preset material movement path, it can be seen that the non-compliant materials are jetted out of the preset material movement path.

[0126] Among them, to determine whether the non-compliant material reaches the preset ejection position, it can be determined by the imaging subsystem 300, by the jet ejection subsystem 800, or by an additional controller.

[0127] The following is illustrated with a specific example.

[0128] In this example, the preset material movement path consists of two parts, namely the path of the track belt driving the material to move and the free fall of the material after it reaches the end of the track. The above-mentioned preset shooting position is located at the end of the track. That is, after the material passes through the preset shooting position, it starts to free fall. The above-mentioned preset ejection position is located on the preset material movement path and in the free fall part of the material. Determining whether the non-compliant material reaches the preset ejection position is performed by the imaging subsystem 300.

[0129] Step 1: According to the application scenario of the material, a track with a width of 60 cm and a transmission speed of 2 m / s is designed; and a camera 401 is placed above the track. The camera 401 in this example is a line scan camera with a resolution of 3600×1, which is used to scan the image features of the material.

[0130] Among them, the above-mentioned line scan camera can be set to generate a picture after accumulating 2400 lines. The picture size is 3600×2400, that is, a material image with a size of 3600×2400 is obtained.

[0131] Moreover, after generating the material image with a size of 3600×2400, the image can also be scaled. As an example, the material image with a size of 3600×2400 is scaled proportionally to 640×427, and 0 is filled in the short side, that is, the short side is filled with black, so as to obtain a material image of 640×640. At this time, the material sorting device can sort out non-compliant materials according to the 640×640 material image. Moreover, the image can also be normalized into a feature vector, so as to obtain an input matrix with a size of 640×640×3.

[0132] Step 2: At a distance from the camera detection point (the above-mentioned preset shooting position) a spray valve is placed. This spray valve is the component for jetting in the above-mentioned jet ejection subsystem 800. The spray valve has a total of 64 jet outlets, and on average each spray valve controls an area of 0.9375 cm. The calculation formula is as follows:

[0133]

[0134] where t 2 is the time when the material reaches the spray valve port, t 1 is the time when the material reaches the camera detection point, and g is the acceleration due to gravity.

[0135] Step 3: First, initialize the software and hardware. The software system starts running. Click the blanking button on the UI interface, and the line scan camera captures images.

[0136] Step 4: Determine which materials are non-compliant materials that need to be removed by the spray valve, and calculate how long it takes to send a spray valve command according to the following formula:

[0137]

[0138] where T 0 is the free movement time from the camera detection point to the spray valve, is the height between the crawler plane and the spray valve, t detector is the model inference time, is the time required to send a spray valve command.

[0139] Step 5: Finally, when reaching send a spray valve command through the serial port between the imaging subsystem 300 and the jetting rejection subsystem 800 to instruct the spray valve to jet air.

[0140] Thus, automatic material sorting based on instance segmentation can be realized, with high efficiency and low manual workload.

[0141] In summary, for the material sorting system based on instance segmentation in the embodiments of the present invention, through the material sorting device based on instance segmentation in the above embodiments, an instance segmentation module is provided for performing instance segmentation on the material image to obtain a plurality of bounding boxes and the segmentation masks, segmentation thresholds, and confidence scores corresponding to the bounding boxes. Among them, the confidence score represents the confidence that the bounding box contains the material in the material image; a segmentation accuracy improvement module is connected to the instance segmentation module for selecting a first target bounding box from the plurality of bounding boxes according to the segmentation masks, segmentation thresholds, and confidence scores corresponding to the respective bounding boxes; a sorting module is connected to the segmentation accuracy improvement module for sorting out the non-compliant materials in the material image according to the first target bounding box and the target segmentation mask, where the target segmentation mask is the segmentation mask corresponding to the first target bounding box. Thus, automatic material sorting based on instance segmentation can be realized, with high efficiency and low manual workload.

[0142] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein can be considered as a definite sequence list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0143] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or a combination of the following techniques known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0144] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0145] In the description of this specification, the orientation or positional relationship indicated by terms such as "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. is based on the orientation or positional relationship shown in the drawings, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and should not be construed as a limitation on the present invention.

[0146] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of the present invention, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise specifically and clearly defined.

[0147] In the description of this specification, unless otherwise stated, terms such as "installation", "connection", "connection", "fixation", etc. should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or integrated; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the communication inside two elements or the interaction relationship between two elements, unless otherwise clearly defined. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0148] In the present invention, unless otherwise clearly specified and limited, the first feature being "on" or "under" the second feature may be that the first and second features are in direct contact, or the first and second features are indirectly in contact through an intermediate medium. Moreover, the first feature being "above", "over" and "on top of" the second feature may be that the first feature is directly above or obliquely above the second feature, or merely indicates that the first feature has a higher horizontal height than the second feature. The first feature being "under", "below" and "beneath" the second feature may be that the first feature is directly below or obliquely below the second feature, or merely indicates that the first feature has a lower horizontal height than the second feature.

[0149] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as a limitation on the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A material sorting device based on instance segmentation, characterized in that: The device comprises: An instance segmentation module, used to perform instance segmentation on the material image to obtain a plurality of bounding boxes and segmentation masks, segmentation thresholds, and confidence scores corresponding to the bounding boxes, wherein the confidence score indicates the confidence that the bounding box contains the material in the material image; A segmentation accuracy improvement module, connected to the instance segmentation module, configured to select a first target bounding box from the plurality of bounding boxes according to the segmentation mask, the segmentation threshold and the confidence score corresponding to each of the bounding boxes; a sorting module, connected to the segmentation accuracy improvement module, and configured to sort out non-compliant materials in the material image according to the first target bounding box and a target segmentation mask, wherein the target segmentation mask is a segmentation mask corresponding to the first target bounding box; The segmentation accuracy improvement module includes: A multiplication submodule, configured to multiply the segmentation threshold and the segmentation mask corresponding to each of the bounding boxes; A convolution and pooling submodule, configured to perform convolution and pooling processing on each multiplication result to obtain a first fractional value corresponding to the multiplication result, wherein M is a positive integer greater than 1; The first target bounding box selection submodule is used to obtain a second score value for each of the bounding boxes according to the bounding box and the first preset box, and obtain a final score value according to the first score value, the second score value and the target confidence score corresponding to the bounding box, and determine that the bounding box is the first target bounding box when the final score value is greater than a first preset score threshold.

2. The material sorting device based on instance segmentation according to claim 1 is characterized in that: The instance segmentation module comprises: A feature extraction submodule is used to extract features from the material image to obtain a feature map; The segmentation submodule is connected to the feature extraction submodule and is used to perform instance segmentation on the feature map to obtain a plurality of the bounding boxes and segmentation masks, segmentation thresholds, and confidence scores corresponding to the bounding boxes.

3. The material sorting device based on instance segmentation according to claim 2 is characterized in that: The feature extraction submodule includes a first pooling layer, a first convolutional layer, a second pooling layer, a second convolutional layer, a third pooling layer, a third convolutional layer, a fourth pooling layer, a fourth convolutional layer, a fifth pooling layer, a first upsampling layer, a second upsampling layer, a fifth convolutional layer, a sixth convolutional layer, and a first connection layer, which are connected in sequence. The feature extraction submodule also includes a second connection layer and a third connection layer. The output ends of the third pooling layer and the second upsampling layer are also connected to the input end of the third connection layer, the output ends of the fourth pooling layer and the fifth convolutional layer are also connected to the input end of the second connection layer, and the output end of the fifth pooling layer is also connected to the input end of the first connection layer; The first connection layer is used to output a first feature map, the second connection layer is used to output a second feature map, and the third connection layer is used to output a third feature map.

4. The material sorting device based on instance segmentation according to claim 1, characterized in that: The first target bounding box selection submodule is specifically used for: Substitute the first score value, the second score value, and the target confidence score into a preset calculation formula to obtain the final score value.

5. The material sorting device based on instance segmentation according to claim 1, characterized in that: The sorting module is specifically used for: The size information of the target material is obtained according to the first target bounding box and the target segmentation mask. When the size information is not within a preset size range, the target material is determined to be a non-compliant material, wherein the target material is a material contained in the first target bounding box in the material image.

6. The material sorting device based on instance segmentation according to claim 1, characterized in that: The device also includes: a comparison module, connected to the instance segmentation module, configured to compare the confidence scores corresponding to the bounding boxes with a second preset score threshold, and use the confidence scores greater than the second preset score threshold as the first intermediate scores; a sorting module, connected to the comparison module, configured to arrange the first intermediate scores in descending order to obtain a score sequence arranged in descending order, and select a preset number of scores from the score sequence as second intermediate scores; a selection module connected to the sorting module and the segmentation accuracy improvement module, and configured to obtain an intersection-and-union ratio between each intermediate bounding box and a first preset box, and select at least one second target bounding box from the intermediate bounding box according to the intersection-and-union ratio and the second intermediate score, so that the segmentation accuracy improvement module selects the first target bounding box from the second target bounding box, wherein the intermediate bounding box is a bounding box corresponding to the second intermediate score.

7. The material sorting device based on instance segmentation according to claim 1, characterized in that: The device also includes: An updating module is used to obtain a first loss value according to the first target bounding box and the target segmentation mask, and to obtain a second loss value according to the difference between the first loss value and a preset standard value, and to update at least one of the instance segmentation module, the segmentation accuracy improvement module, and the sorting module according to the second loss value.

8. The material sorting device based on instance segmentation according to claim 7 is characterized in that: The update module is specifically used for: A third loss value is obtained according to the first target bounding box and the third preset box, a fourth loss value is obtained according to the target segmentation mask and the preset true mask, and the first loss value is obtained according to the third loss value and the fourth loss value.

9. A material sorting system based on instance segmentation, characterized in that: The system includes a crawler transmission subsystem, an imaging subsystem and an air jet rejection subsystem. The crawler transmission subsystem is used to realize the movement of materials by crawlers, the imaging subsystem is used to photograph the materials and sort out non-compliant materials according to the photographed material images, and the air jet rejection subsystem is used to spray the non-compliant materials moved to the preset spraying position away from the preset material movement path, thereby realizing material sorting; Wherein, the imaging subsystem includes a camera and a material sorting device based on instance segmentation according to any one of claims 1-8, the camera is used to shoot the material at a preset shooting position to obtain the material image, the preset shooting position is located on the preset material movement path, and is located between the preset spraying position and the starting movement position of the material on the crawler, and the material sorting device based on instance segmentation is used to sort out the non-compliant material according to the material image.

Images