Material sorting device and system based on instance segmentation
Through the material sorting device and system based on example segmentation, the disposable wood and bamboo chopsticks are automatically sorted, which solves the problem of low sorting efficiency in the existing technology and realizes efficient and accurate material sorting.
Patent Information
- Application Number
- CN202411167657.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-23
AI Technical Summary
In the prior art, the material sorting efficiency is low in the production process of disposable wooden chopsticks and bamboo chopsticks, and manual sorting is required, resulting in large workload and low efficiency.
Using a material sorting device and system based on instance segmentation, the material image is divided by the instance segmentation module to obtain bounding box, segmentation mask, segmentation threshold and confidence score, and automatically segmented with the segmentation accuracy improvement module and sorting module. Select non-compliant materials.
It realizes the automation of material sorting, improves sorting efficiency, reduces manual workload, and improves the accuracy and efficiency of sorting.
Smart Images

Figure CN119068000A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of instance segmentation, and in particular to a material sorting device and system based on instance segmentation. Background Art
[0002] In the production process of disposable wooden and bamboo chopsticks, chopsticks that meet the standard length need to be sorted, and longer or shorter chopsticks need to be discarded. However, the technology used for sorting in the related art usually adopts manual sorting, which is labor-intensive and inefficient. Summary of the invention
[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent. To this end, the first object of the present invention is to provide a material sorting device based on instance segmentation to improve sorting efficiency.
[0004] The second object of the present invention is to propose a material sorting system based on instance segmentation.
[0005] To achieve the above-mentioned purpose, the first aspect of the present invention proposes a material sorting device based on instance segmentation, and the device includes: an instance segmentation module, which is used to perform instance segmentation on a material image to obtain multiple bounding boxes and segmentation masks, segmentation thresholds, and confidence scores corresponding to the bounding boxes, wherein the confidence score represents the confidence that the material in the material image is contained in the bounding box; a segmentation accuracy improvement module, which is connected to the instance segmentation module, and is used to select a first target bounding box from the multiple bounding boxes according to the segmentation masks, segmentation thresholds, and confidence scores corresponding to each of the bounding boxes; a sorting module, which is connected to the segmentation accuracy improvement module, and is used to sort out non-compliant materials in the material image according to the first target bounding box and the target segmentation mask, wherein the target segmentation mask is a segmentation mask corresponding to the first target bounding box.
[0006] In addition, the material sorting device based on instance segmentation according to the embodiment of the present invention may also have the following additional technical features:
[0007] In one embodiment of the present invention, the instance segmentation module includes: a feature extraction submodule, used to extract features from the material image to obtain a feature map; a segmentation submodule, connected to the feature extraction submodule, used to perform instance segmentation on the feature map to obtain multiple bounding boxes and segmentation masks, segmentation thresholds, and confidence scores corresponding to the bounding boxes.
[0008] In one embodiment of the present invention, the feature extraction submodule includes a first pooling layer, a first convolutional layer, a second pooling layer, a second convolutional layer, a third pooling layer, a third convolutional layer, a fourth pooling layer, a fourth convolutional layer, a fifth pooling layer, a first upsampling layer, a second upsampling layer, a fifth convolutional layer, a sixth convolutional layer, and a first connection layer connected in sequence, and the feature extraction module also includes a second connection layer and a third connection layer; the output ends of the third pooling layer and the second upsampling layer are also connected to the input end of the third connection layer, the output ends of the fourth pooling layer and the fifth convolutional layer are also connected to the input end of the second connection layer, and the output end of the fifth pooling layer is also connected to the input end of the first connection layer; wherein the first connection layer is used to output a first feature map, the second connection layer is used to output a second feature map, and the third connection layer is used to output a third feature map.
[0009] In one embodiment of the present invention, the segmentation accuracy improvement module includes: a multiplication submodule, which is used to multiply the segmentation threshold and the segmentation mask corresponding to each of the bounding boxes; a convolution and pooling submodule, which is used to perform convolution and pooling on each multiplication result to obtain a first score value corresponding to the multiplication result, wherein M is a positive integer greater than 1; a first target bounding box selection submodule, which is used to obtain a second score value according to the bounding box and the first preset box for each of the bounding boxes, and obtain a final score value according to the first score value, the second score value and the target confidence score corresponding to the bounding box, and when the final score value is greater than the first preset score threshold, determine that the bounding box is the first target bounding box.
[0010] In one embodiment of the present invention, the first target bounding box selection submodule is specifically used to: substitute the first score value, the second score value and the target confidence score into a preset calculation formula to obtain the final score value.
[0011] In one embodiment of the present invention, the sorting module is specifically used to: obtain size information of the target material based on the first target bounding box and the target segmentation mask, and when the size information is not within a preset size range, determine that the target material is a non-compliant material, wherein the target material is the material contained in the first target bounding box in the material image.
[0012] In one embodiment of the present invention, the device also includes: a comparison module, connected to the instance segmentation module, for comparing the confidence score corresponding to each of the bounding boxes with a second preset score threshold, and taking the confidence score greater than the second preset score threshold as the first intermediate score; a sorting module, connected to the comparison module, for arranging the first intermediate scores in descending order to obtain a score sequence after descending order, and selecting a preset number of scores from the score sequence as the second intermediate scores; a selection module, connected to the sorting module and the segmentation accuracy improvement module, for obtaining the intersection-and-union ratio of each intermediate bounding box to the first preset box, and selecting at least one second target bounding box from the intermediate bounding box according to the intersection-and-union ratio and the second intermediate score, so that the segmentation accuracy improvement module selects the first target bounding box from the second target bounding box, wherein the intermediate bounding box is the bounding box corresponding to the second intermediate score.
[0013] In one embodiment of the present invention, the device also includes: an updating module, used to obtain a first loss value based on the first target bounding box and the target segmentation mask, and obtain a second loss value based on the difference between the first loss value and a preset standard value, and update at least one of the instance segmentation module, the segmentation accuracy improvement module, and the sorting module based on the second loss value.
[0014] In one embodiment of the present invention, the update module is specifically used to: obtain a third loss value based on the first target bounding box and the third preset box, obtain a fourth loss value based on the target segmentation mask and the preset true mask, and obtain the first loss value based on the third loss value and the fourth loss value.
[0015] To achieve the above-mentioned purpose, the second aspect of the embodiment of the present invention proposes a material sorting system based on instance segmentation, the system comprising a crawler transmission subsystem, an imaging subsystem and a jet rejection subsystem, the crawler transmission subsystem is used to realize the movement of materials by crawlers, the imaging subsystem is used to photograph the materials, and sort out non-compliant materials according to the material images obtained by photographing, and the jet rejection subsystem is used to spray non-compliant materials moved to a preset spray-off position away from the preset material movement path, thereby realizing material sorting; wherein, the imaging subsystem comprises a camera and the above-mentioned material sorting device based on instance segmentation, the camera is used to photograph the material at a preset shooting position to obtain the material image, the preset shooting position is located on the preset material movement path, and is located between the preset spray-off position and the starting movement position of the material on the crawler, and the material sorting device based on instance segmentation is used to sort out the non-compliant materials according to the material image.
[0016] According to the material sorting device and system based on instance segmentation of the embodiment of the present invention, an instance segmentation module is provided for performing instance segmentation on the material image to obtain multiple bounding boxes and segmentation masks, segmentation thresholds, and confidence scores corresponding to the bounding boxes, wherein the confidence score indicates the confidence of the material in the material image contained in the bounding box; a segmentation accuracy improvement module is connected to the instance segmentation module and is used to select a first target bounding box from multiple bounding boxes according to the segmentation mask, segmentation threshold, and confidence score corresponding to each bounding box; a sorting module is connected to the segmentation accuracy improvement module and is used to sort out non-compliant materials in the material image according to the first target bounding box and the target segmentation mask, wherein the target segmentation mask is the segmentation mask corresponding to the first target bounding box. Thus, automatic material sorting based on instance segmentation can be achieved with high efficiency and low manual workload.
[0017] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 is a structural block diagram of a material sorting device based on instance segmentation according to an embodiment of the present invention;
[0019] Figure 2 is a schematic structural diagram of a material sorting device based on instance segmentation according to an example of the present invention;
[0020] Figure 3 is a structural schematic diagram of a material sorting device based on instance segmentation according to another example of the present invention;
[0021] Figure 4 is a structural schematic diagram of a material sorting device based on instance segmentation according to another example of the present invention;
[0022] Figure 5 is a structural schematic diagram of a material sorting device based on instance segmentation according to another example of the present invention;
[0023] Figure 6 is a structural schematic diagram of a material sorting device based on instance segmentation according to another example of the present invention;
[0024] Figure 7 It is a work flow chart of a material sorting device based on instance segmentation according to an example of the present invention;
[0025] Figure 8 is a schematic structural diagram of a material sorting system based on instance segmentation according to an embodiment of the present invention;
[0026] Fig. 9 4 is a structural block diagram of an imaging subsystem according to an embodiment of the present invention. DETAILED DESCRIPTION
[0027] The following describes a material sorting device and system based on example segmentation according to an embodiment of the present invention with reference to the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described with reference to the accompanying drawings are exemplary and should not be construed as limiting the present invention.
[0028] Figure 1 It is a structural block diagram of a material sorting device based on instance segmentation according to an embodiment of the present invention.
[0029] like Figure 1 As shown, the material sorting device 400 based on instance segmentation includes: an instance segmentation module 500, which is used to perform instance segmentation on the material image to obtain multiple bounding boxes and segmentation masks, segmentation thresholds, and confidence scores corresponding to the bounding boxes, wherein the confidence score represents the confidence that the material in the material image is contained in the bounding box; a segmentation accuracy improvement module 600, which is connected to the instance segmentation module 500, and is used to select a first target bounding box from the multiple bounding boxes according to the segmentation mask, segmentation threshold and confidence score corresponding to each bounding box; a sorting module 700, which is connected to the segmentation accuracy improvement module 600, and is used to sort out non-compliant materials in the material image according to the first target bounding box and the target segmentation mask, wherein the target segmentation mask is the segmentation mask corresponding to the first target bounding box.
[0030] Specifically, after obtaining the material image, in order to sort out non-compliant materials according to the material image, the above-mentioned material sorting device 400 based on instance segmentation is set to include an instance segmentation module 500, a segmentation accuracy improvement module 600, and a sorting module 700.
[0031] The instance segmentation module 500 is used to perform instance segmentation on the material image. Specifically, since the material image contains at least one material, the instance segmentation module 500 performs instance segmentation on the material image, and obtains multiple bounding boxes and segmentation masks, segmentation thresholds, and confidence scores corresponding to the bounding boxes, and then each bounding box and the segmentation mask, segmentation threshold, and confidence score corresponding to the bounding box can be regarded as an instance segmentation result. Each of the above bounding boxes has a segmentation mask, segmentation threshold, and confidence score corresponding to the bounding box, and the confidence score indicates the confidence that the bounding box corresponding to the score contains the material.
[0032] In actual applications, instances may segment out bounding boxes that cannot be used to sort non-compliant materials. Therefore, a segmentation accuracy improvement module 600 needs to be set up. After obtaining multiple instance segmentation results, the instance segmentation results are input into the segmentation accuracy improvement module 600, so that the segmentation accuracy improvement module 600 selects the first target bounding box from the multiple bounding boxes according to the segmentation mask, segmentation threshold, and confidence score corresponding to each bounding box. In other words, after obtaining multiple bounding boxes, the segmentation accuracy improvement module 600 needs to select a bounding box that meets the requirements and use it as the first target bounding box, and discard the bounding boxes that do not meet the requirements. For example, although each bounding box theoretically contains materials, in actual applications, there may be bounding boxes that do not contain materials. At this time, the bounding box that does not contain materials is the non-compliant bounding box that needs to be discarded.
[0033] After obtaining the first target bounding box, the first target bounding box and the target segmentation mask corresponding to the first target bounding box are input into the sorting module 700. The sorting module 700 obtains information about the materials contained in each first target bounding box according to the first target bounding box and the target segmentation mask, thereby determining whether the material is a non-compliant material according to the information about the material.
[0034] In this way, instance segmentation of material images can be achieved, and then non-compliant materials can be sorted out based on the instance segmentation results.
[0035] In some embodiments of the present invention, the instance segmentation module 500 includes: a feature extraction submodule, which is used to extract features from the material image to obtain a feature map; a segmentation submodule, which is connected to the feature extraction submodule, and is used to perform instance segmentation on the feature map to obtain multiple bounding boxes and segmentation masks, segmentation thresholds, and confidence scores corresponding to the bounding boxes.
[0036] Specifically, the instance segmentation module 500 includes a feature extraction submodule and a segmentation submodule. After obtaining a material image, the instance segmentation module 500 performs instance segmentation on the material image.
[0037] The specific instance segmentation process is that the feature extraction submodule in the instance segmentation module 500 extracts features from the material image to obtain a feature map, which contains semantic information, spatial information, detail information, etc. After obtaining the feature map, the segmentation submodule can segment the feature map to obtain multiple bounding boxes and a segmentation mask, segmentation threshold, and confidence score corresponding to each bounding box.
[0038] In some embodiments of the present invention, the feature extraction submodule includes a first pooling layer, a first convolutional layer, a second pooling layer, a second convolutional layer, a third pooling layer, a third convolutional layer, a fourth pooling layer, a fourth convolutional layer, a fifth pooling layer, a first upsampling layer, a second upsampling layer, a fifth convolutional layer, a sixth convolutional layer, and a first connection layer connected in sequence, and the feature extraction module also includes a second connection layer and a third connection layer; the output ends of the third pooling layer and the second upsampling layer are also connected to the input end of the third connection layer, the output ends of the fourth pooling layer and the fifth convolutional layer are also connected to the input end of the second connection layer, and the output end of the fifth pooling layer is also connected to the input end of the first connection layer; wherein the first connection layer is used to output the first feature map, the second connection layer is used to output the second feature map, and the third connection layer is used to output the third feature map.
[0039] Combine the following Figure 2 The example shown is specifically described.
[0040] exist Figure 2 In the figure, C2f1 is the first pooling layer, Conv1 is the first convolutional layer, C2f2 is the second pooling layer, Conv2 is the second convolutional layer, C2f3 is the third pooling layer, Conv3 is the third convolutional layer, C2f4 is the fourth pooling layer, Conv4 is the fourth convolutional layer, C2f5 is the fifth pooling layer, Upsample1 is the first upsampling layer, Upsample2 is the second upsampling layer, Conv5 is the fifth convolutional layer, Conv6 is the sixth convolutional layer, Concat1 is the first connection layer, Concat2 is the second connection layer, and Concat3 is the third connection layer.
[0041] In this example, the size and depth of the output result of the first pooling layer are set to 640×640×3, that is, the size is 640×640, the depth is 3, the first convolutional layer is 320×320×32, the second pooling layer is 320×320×64, the second convolutional layer is 160×160×128, the third pooling layer is 160×160×128, the third convolutional layer is 80×80×256, and the fourth pooling layer is 80×80 ×256, the fourth convolution layer is 40×40×512, the fifth pooling layer is 40×40×512, the first upsampling layer is 80×80×256, the second upsampling layer is 160×160×128, the fifth convolution layer is 80×80×128, the sixth convolution layer is 40×40×256, the first connection layer is 40×40×762, the second connection layer is 80×80×384, and the third connection layer is 160×160×192. Of course, in practical applications, it is not limited to this and can be adjusted according to actual needs.
[0042] It can be seen that after obtaining a material image of 3600×2400, the material image is first scaled and padded to obtain an image of 640×640, and then the image of 640×640 is input into the network. The feature extraction submodule in the network includes a first pooling layer, a first convolutional layer, a second pooling layer, a second convolutional layer, a third pooling layer, a third convolutional layer, a fourth pooling layer, a fourth convolutional layer, a fifth pooling layer, a first upsampling layer, a second upsampling layer, a fifth convolutional layer, a sixth convolutional layer, and a first connection layer, which are connected in sequence, and also include a second connection layer and a third connection layer. After the above-mentioned image of 640×640 is processed by the convolutional layer, the pooling layer, and the upsampling layer, it is input into the first connection layer, the second connection layer, and the third connection layer. The first connection layer, the second connection layer, and the third connection layer output the first feature map, the second feature map, and the third feature map, respectively.
[0043] Continue to see Figure 3 The example shown illustrates the segmentation submodule described above.
[0044] It can be seen that after Concat1, Concat2, and Concat3, the first device Detaction, the second device Mask Confidence, and the third device Proto Struct are also set. Detaction, Mask Confidence, and Proto Struct together constitute the above-mentioned segmentation submodule, and each device is used to generate corresponding information according to the first feature map, the second feature map, and the third feature map. Specifically, Detaction is used to generate a bounding box and a confidence score corresponding to the bounding box according to the first feature map, the second feature map, and the third feature map, Mask Confidence is used to generate a segmentation threshold according to the first feature map, the second feature map, and the third feature map, and Proto Struct is used to generate a segmentation mask according to the first feature map, the feature map, and the third feature map. In other words, every time Detaction generates a bounding box, Detaction also needs to generate a confidence score corresponding to the bounding box, and Mask Confidence needs to generate a segmentation threshold corresponding to the bounding box, and ProtoStruct needs to generate a segmentation mask corresponding to the bounding box.
[0045] In some embodiments of the present invention, the segmentation accuracy improvement module 600 includes: a multiplication submodule, which is used to multiply the segmentation threshold and the segmentation mask corresponding to each bounding box for each bounding box; a convolution and pooling submodule, which is used to perform convolution and pooling on each multiplication result to obtain a first score value corresponding to the multiplication result, wherein M is a positive integer greater than 1; a first target bounding box selection submodule, which is used to obtain a second score value for each bounding box based on the bounding box and the first preset box, and obtain a final score value based on the first score value, the second score value and the target confidence score corresponding to the bounding box, and when the final score value is greater than the first preset score threshold, determine that the bounding box is the first target bounding box.
[0046] Specifically, for each bounding box, the segmentation accuracy improvement module 600 needs to obtain a final score value corresponding to the bounding box according to the confidence score, segmentation threshold and segmentation mask corresponding to the bounding box.
[0047] The segmentation accuracy improvement module 600 needs to multiply the segmentation mask and the segmentation threshold corresponding to each bounding box, and pass the multiplication result through the convolution and pooling submodules to obtain a first score value. In addition, it is also necessary to obtain a second score value based on the difference between the center point, width, height and diagonal length of each bounding box and the first preset box.
[0048] After the first score value and the second score value are obtained, for each bounding box, a final score value can be obtained according to the first score value, the second score value and the confidence score corresponding to the bounding box.
[0049] After obtaining the final score value corresponding to each bounding box, the final score value can be compared with the first preset score threshold to obtain a final score value greater than the first preset score threshold, and the bounding box corresponding to the final score value greater than the first preset score threshold is used as the first target bounding box, and then according to the first target bounding box and the target segmentation mask, it is determined whether the material contained in the first target bounding box is a non-compliant material.
[0050] The multiplication submodule can be implemented by a multiplier. The convolution and pooling submodule can be configured to include M convolution layers connected in series and a maximum pooling layer, where M is a positive integer greater than 1. The first target bounding box selection submodule can be implemented by an electronic device storing a computer program, preset software, a digital circuit, etc., as long as the disclosed content of the positive sample selector can be implemented.
[0051] Combine the following Figure 4 The example shown is used for illustration.
[0052] exist Figure 4, 3 is the multiplication submodule, 501 is the first target bounding box selection submodule, and the Output includes the first target bounding box and the confidence score, segmentation threshold, and segmentation mask corresponding to the first target bounding box. In this example, the multiplication submodule is implemented by a multiplier.
[0053] Origin Mask is the result of multiplier 3. Figure 4 In the specific example shown, after the multiplier 3 performs the operation, it is set to output an output result of N×160×160, that is, it has N channels and the size is 160×160. Of course, in practical applications, the size and channels of the output result of the multiplier 3 can be adjusted according to actual conditions, and are not limited thereto.
[0054] Figure 4 The multiple Conv in are the M convolutional layers connected in sequence, and the M convolutional layers connected in sequence can be set to be the same or different. The M convolutional layers connected in sequence can use the same convolutional layers as the first, second, third, fourth, fifth, and sixth convolutional layers, or different convolutional layers.
[0055] Figure 4 Relu in is the activation function, 4 is the confidence score, 5 is the second score value, and 6 is the first score value. MaxPooling is the maximum pooling layer mentioned above.
[0056] That is to say, Figure 3 In the above convolution and pooling submodule, M Conv, M Relu and one MaxPooling.
[0057] In some embodiments of the present invention, the first target bounding box selection submodule is specifically used to: substitute the first score value, the second score value and the target confidence score into a preset calculation formula to obtain a final score value.
[0058] Specifically, the above preset calculation formula may be the following calculation formula:
[0059] final_Score=Score1 α ×Score2 β ×Score3 γ ,
[0060] Among them, final_score is the final score value, Score1 is the target confidence score, Score2 is the second score value, Score3 is the first score value, α is the first preset exponential coefficient, β is the second preset exponential coefficient, and γ is the third preset exponential coefficient.
[0061] After calculating the final score value final_score, the closer the final score value is to 1, the higher the matching degree is, and the more likely it is a positive sample. Therefore, for each first target bounding box, the corresponding final score value is compared with the second preset score threshold. When the final score value corresponding to the first target bounding box is greater than the second preset score threshold, the first target bounding box is determined to be a positive sample.
[0062] In some embodiments of the present invention, the sorting module 700 is specifically used to: obtain the size information of the target material according to the first target bounding box and the target segmentation mask, and when the size information is not within the preset size range, determine that the target material is a non-compliant material, wherein the target material is a material contained in the second target bounding box in the material image. For example, if the material is chopsticks, the size information of the chopsticks contained in the second target bounding box can be obtained according to the second target bounding box and the segmentation mask corresponding to the second target bounding box, so as to obtain the length of the chopsticks according to the size information. If the length of a certain chopstick is obviously too short, the chopstick can be regarded as a non-compliant chopstick.
[0063] In some embodiments of the present invention, the material sorting device 400 based on instance segmentation further includes: a comparison module, connected to the instance segmentation module 500, for comparing the confidence score corresponding to each bounding box with a second preset score threshold, and taking the confidence score greater than the second preset score threshold as the first intermediate score; a sorting module, connected to the comparison module, for arranging the first intermediate score in descending order to obtain a score sequence after descending order, and selecting a preset number of scores from the score sequence as the second intermediate score; a selection module, connected to the sorting module and the segmentation accuracy improvement module 600, for obtaining the intersection and union ratio of each intermediate bounding box with the first preset box, and selecting at least one second target bounding box from the intermediate bounding box according to the intersection and union ratio and the second intermediate score, so that the segmentation accuracy improvement module 600 selects the first target bounding box from the second target bounding box, wherein the intermediate bounding box is the bounding box corresponding to the second intermediate score.
[0064] Specifically, a comparison module, a sorting module and a selection module are further provided, and the output result of the instance segmentation module 500 is first processed by the comparison module, the sorting module and the selection module to obtain a second target bounding box, so that the segmentation accuracy improvement module 600 selects the first target bounding box from the second target bounding box. In other words, the output result of the instance segmentation module 500 is processed by the segmentation accuracy improvement module 600 after being processed by the comparison module, the sorting module and the selection module.
[0065] After generating the segmentation mask, confidence score and segmentation threshold corresponding to each bounding box, the comparison module is used to traverse all confidence scores generated by the segmentation submodule, compare the confidence scores with the second preset score threshold, filter out the confidence scores less than the second preset score threshold, and use the remaining confidence scores as the first intermediate scores.
[0066] After filtering the confidence scores, the sorting module arranges the first intermediate scores in descending order according to the specific score values, and then selects the top_k first intermediate scores to obtain the second intermediate scores. The top_k is the above-mentioned preset number, that is, the number of the second intermediate scores is top_k.
[0067] After obtaining the top_k second intermediate scores, the bounding boxes corresponding to the second intermediate scores may be used as the intermediate bounding boxes.
[0068] It should be noted that the above-mentioned arrangement of the first intermediate scores in descending order according to the size of the specific score value is carried out according to the category. Specifically, since there may be more than one type of material in the material image, after obtaining the first intermediate score, it is necessary to arrange all the first intermediate scores of each category in descending order according to the size of the score value, and select the top_k first intermediate scores of the category. The category refers to the category of the material. For example, if the material includes chopsticks, spoons, and diamonds, chopsticks are a category, spoons are a category, and diamonds are a category. Taking chopsticks and spoons as examples, the sorting module needs to arrange all the first intermediate scores related to chopsticks in descending order, select the top_k first intermediate scores, as the second intermediate scores under the category of chopsticks, and the sorting module also needs to arrange all the first intermediate scores related to spoons in descending order, and select the top_k first intermediate scores as the second intermediate scores under the category of spoons. That is to say, under each category, there are top_k intermediate bounding boxes.
[0069] After obtaining the second intermediate score, the selection module determines the intermediate bounding box according to the second intermediate score, and calculates the intersection-and-union ratio of each intermediate bounding box with the first preset box, thereby selecting at least one second target bounding box from the intermediate bounding box according to the intersection-and-union ratio and the confidence score corresponding to the intermediate bounding box.
[0070] As an example, the above selection module can work as follows:
[0071] The selection module first calculates the ratio between the intersection and union of each intermediate bounding box and the first preset box, obtains the intersection-union ratio of the intermediate bounding box and the first preset box, and then performs f function calculation on the intersection-union ratio corresponding to each intermediate bounding box.
[0072] After calculating the f function for each intersection-and-union ratio, select it according to the following formula:
[0073]
[0074] in, This means that only the intersection and union ratio corresponding to the first intermediate score above will be performed, s k is the confidence score corresponding to the kth bounding box, s i is the second preset score threshold under the i-th category, means all, f(iou k,i ) means iou k,i Substitute the result calculated by the above f function, iou k,i is the intersection-and-union ratio of the kth intermediate bounding box under the ith category. That is, after calculating the intersection-and-union ratios of each intermediate bounding box, the minimum value is selected, f(iou .,i ) is the minimum f function calculation result under the selected i-th category, iou .,i It means to select according to the intersection and union ratio under the i-th category.
[0075] After calculating f(iou .,i ), and then calculate according to the following formula:
[0076]
[0077] That is to say, after calculating f(iou .,i ), each qualified f(iou i.j ) are divided by f(iou .,i ) and select the minimum decay value of the calculation result i .
[0078] Among them, iou i,j is the intersection-over-union ratio corresponding to the jth sample under the ith category. For example, assuming that the category is chopsticks, and the image taken for chopsticks contains n chopsticks, then the value of j ranges from 1 to n, and f(iou i,j ) is the corresponding iou i,j The calculation result of .
[0079] In the selection to get the minimum calculation result decay i After that, the decay i Substitute the following formula to calculate:
[0080] scores Iou =scores top_k *decay i ,
[0081] Among them, scores top_k is the top_k first intermediate scores, that is, the second intermediate scores. That is, for each second intermediate score, its decay i The product of the score is used to get the IOU score corresponding to the score. Iou .
[0082] After calculating the IOU scores corresponding to the top_k second intermediate scores above, Iou Finally, the values whose IOU scores are greater than the preset IOU score threshold are filtered out, and the intermediate bounding boxes corresponding to the remaining values are used as the second target bounding box.
[0083] Among them, the f function calculation for the intersection and union ratio corresponding to each intermediate bounding box is: f(iou i,j )=1-iou i,j Of course, in this step, the iou used for calculation i,j are all the intersection and union ratios corresponding to the first intermediate scores mentioned above.
[0084] It should be noted that the above-mentioned selection module can be implemented by electronic devices storing computer programs, preset software, digital circuits, etc., as long as the public content of the above-mentioned first selection sub-module can be implemented.
[0085] After obtaining at least one second target bounding box, the first target bounding box can be selected from the second target bounding boxes by the segmentation accuracy improvement module 600. The segmentation accuracy can be improved by the segmentation accuracy improvement module 600.
[0086] In some embodiments of the present invention, the material sorting device 400 based on instance segmentation further includes: an updating module, used to obtain a first loss value based on the first target bounding box and the target segmentation mask, and to obtain a second loss value based on the difference between the first loss value and a preset standard value, and to update at least one of the instance segmentation module 500, the segmentation accuracy improvement module 600, and the sorting module 700 based on the second loss value.
[0087] Specifically, the material sorting device also includes an update module, which is used to substitute positive samples into a preset loss function for calculation, and update at least one of the instance segmentation module 500, the segmentation accuracy improvement module 600, and the sorting module 700 according to the calculation results.
[0088] The updating module is described below in conjunction with a specific embodiment. In this specific example, whether it is the second target bounding box or the third preset box, the upper left corner thereof is used as the preset point thereof. Of course, it is not limited to this in practical applications.
[0089] In this specific embodiment, in order to obtain a first loss value based on the first target bounding box and the target segmentation mask, the update module can be set to obtain a third loss value based on the first target bounding box and the third preset box, and to obtain a fourth loss value based on the target segmentation mask and the preset true mask, and to obtain the first loss value based on the third loss value and the fourth loss value.
[0090] The third loss value obtained according to the first target bounding box and the third preset box can be calculated by the following formula:
[0091]
[0092] Among them, Loss_detect is the third loss value mentioned above, N is the number of materials in a batch, and the above x i is the horizontal coordinate of the preset point on the first target bounding box corresponding to the i-th material, y i is the ordinate of the preset point on the first target bounding box, w i is the horizontal width of the first target bounding box, h i is the vertical width of the first target bounding box, is the horizontal coordinate of the preset point on the third preset frame, 1 is the vertical coordinate of the preset point on the third preset frame, is the horizontal width of the third preset frame, It is the vertical width of the third preset frame.
[0093] The above ij is the classification value of the material in the first target boundary box by the material sorting device, Manually annotate the classification values of the materials in the first target boundary box. That is, if at least one of the instance segmentation module 500, the segmentation accuracy improvement module 600, and the sorting module 700 is to be updated through the update module, the materials in the first target boundary box need to be manually coordinated first, and the classification values corresponding to different material classifications need to be determined in advance.
[0094] The above C represents the number of classifications, λ1 and λ2 are two preset weights, and the preset weights can be determined according to the weights of the bounding box loss and the classification loss in the third loss value.
[0095] The fourth loss value obtained according to the target segmentation mask and the preset true mask can be calculated by the following formula:
[0096]
[0097] Among them, Segment_Loss is the fourth loss value, N1 is the number of pixels in the target segmentation mask, M jis the true mask value of the j-th pixel, Preset real mask for the above.
[0098] The first loss value obtained according to the third loss value and the fourth loss value can be calculated by the following formula:
[0099] Loss1=0.5×Loss_detect+0.75×Segment_Loss,
[0100] Among them, Loss1 is the first loss value.
[0101] After the first loss value is calculated, the error between the first loss value and the preset standard value can be compared, and at least one of the instance segmentation module 500, the segmentation accuracy improvement module 600, and the sorting module 700 can be updated according to the error. The above-mentioned updating of at least one of the instance segmentation module 500, the segmentation accuracy improvement module 600, and the sorting module 700 can be to update the weight of the convolution kernel contained therein. Of course, the update module can also update the above-mentioned comparison module, sorting module, and selection module.
[0102] Combine the following Figure 5 and Figure 6 The example shown is specifically described.
[0103] exist Figure 5 In , Label1 is a label image, which is a standard image that has been pre-segmented and annotated with artificial instances, and the above-mentioned preset standard value can be obtained according to the label image, and 500 is the above-mentioned segmentation accuracy improvement module 600. Figure 5 In the example, Detection Loss is the first operation module, Segment Loss is the second operation module, and Loss is the third operation module. The first operation module is used to calculate the third loss value, the second operation module is used to calculate the fourth loss value, and the third operation module is used to calculate the first loss value and the second loss value. The update module includes the first operation module, the second operation module, and the third operation module.
[0104] See also Figure 6 , Detection Loss calculates the third loss value based on the output of the first target bounding box selection submodule. Segment Loss calculates the fourth loss value based on the output of the first target bounding box selection submodule and the output of the multiplication submodule. For example, the real mask value can be obtained by the output of the multiplication submodule. It should be noted that Figure 6 This is just a specific example and is not limited to this in actual applications.
[0105] That is to say, after obtaining the material image, the material image is first input into the instance segmentation module 500, and the feature extraction submodule in the instance segmentation module 500 performs feature extraction on the material image to obtain a first feature map, a second feature map and a third feature map. The segmentation submodule obtains multiple bounding boxes according to the first feature map, the second feature map and the third feature map, as well as a confidence score, a segmentation threshold and a segmentation mask corresponding to each bounding box.
[0106] Furthermore, each bounding box and the confidence score, segmentation threshold, and segmentation mask corresponding to each bounding box are input into a comparison module, and the comparison module compares each confidence score with a second preset score threshold to obtain a first intermediate score, and then the sorting module sorts each first intermediate score and selects a second intermediate score. The number of the second intermediate scores can be multiple.
[0107] After obtaining the second intermediate scores, since there are multiple second intermediate scores, there are also multiple intermediate bounding boxes corresponding to the second intermediate scores. At this time, the selection module and the segmentation accuracy improvement module 600 can select the first target bounding box from the multiple intermediate bounding boxes. By continuously performing bounding box selection by the selection module and the segmentation accuracy improvement module 600, the segmentation accuracy of instance segmentation can be further improved.
[0108] After obtaining the first target bounding box, the sorting module 700 sorts out non-compliant materials from the materials contained in the first target bounding box according to the first target bounding box and its corresponding target segmentation mask.
[0109] Moreover, after obtaining the first target bounding box, an update module may be used to update at least one of the segmentation module, the segmentation accuracy improvement module 600 , the sorting module 700 and other modules according to the first target bounding box and its corresponding target segmentation mask.
[0110] The above-mentioned sorting module 700 and updating module can be implemented by electronic devices storing computer programs, preset software, digital circuits, etc., as long as the public contents of the above-mentioned sorting module 700 and updating module can be implemented.
[0111] See below Figure 7 The example shown is used for illustration.
[0112] Specifically, first initialize the software and hardware, load the SortNet model, which is the model used to sort out non-compliant materials according to material images included in the material sorting device 400 based on instance segmentation, and then start unloading, and turn on the dual-camera, which is a camera used to capture material images. Of course, a single-camera camera or other camera can also be used to capture the material images.
[0113] If the dual camera fails to be turned on, an error is reported.
[0114] If the dual camera is turned on successfully, the front and back 4096×256 images are taken. In other words, after the camera is turned on, it continuously takes images and continuously outputs images of size 4096×256.
[0115] Furthermore, the captured image is input into the above-mentioned model to infer non-compliant materials, thereby calculating the spray valve delay, and sending a spray valve instruction when the delay time expires.
[0116] In summary, the material sorting device based on instance segmentation of the embodiment of the present invention is provided with an instance segmentation module for performing instance segmentation on the material image to obtain multiple bounding boxes and segmentation masks, segmentation thresholds, and confidence scores corresponding to the bounding boxes, wherein the confidence score indicates the confidence of the material in the material image contained in the bounding box; a segmentation accuracy improvement module is connected to the instance segmentation module, and is used to select a first target bounding box from multiple bounding boxes according to the segmentation mask, segmentation threshold, and confidence score corresponding to each bounding box; a sorting module is connected to the segmentation accuracy improvement module, and is used to sort out non-compliant materials in the material image according to the first target bounding box and the target segmentation mask, wherein the target segmentation mask is the segmentation mask corresponding to the first target bounding box. In this way, automatic material sorting based on instance segmentation can be achieved with high efficiency and low manual workload.
[0117] Furthermore, the present invention proposes a material sorting system based on instance segmentation.
[0118] Figure 8 It is a structural schematic diagram of a material sorting system based on instance segmentation according to an embodiment of the present invention.
[0119] like Figure 8 As shown, the material sorting system 100 based on instance segmentation includes a track transmission subsystem 200, an imaging subsystem 300 and a jet rejection subsystem 800. The track transmission subsystem 200 is used to realize the movement of materials by tracks, the imaging subsystem 300 is used to photograph the materials and sort out non-compliant materials based on the photographed material images, and the jet rejection subsystem 800 is used to spray the non-compliant materials moved to a preset spraying position away from the preset material movement path, thereby realizing material sorting.
[0120] Among them, see Fig. 9The imaging subsystem 300 includes a camera 401 and a material sorting device 400 based on instance segmentation of the above embodiment. The camera 401 is used to shoot the material at a preset shooting position to obtain a material image. The preset shooting position is located on a preset material moving path and is located between a preset spraying position and a starting moving position of the material on the track. The material sorting device 400 based on instance segmentation of the above embodiment is used to sort out non-compliant materials according to material images.
[0121] The above-mentioned materials can be chopsticks, spoons, etc.
[0122] The above-mentioned preset material movement path can be the movement path of the material on the crawler, and can also include the movement path of the material on the crawler. Specifically, the crawler drives the material to move, and this movement path can be set as the preset material movement path; or the preset material movement path can be set to include at least two parts, the first part, the crawler drives the material to move, and the second part, the material moves to the end of the crawler and then falls freely. At this time, the preset material movement path can be composed of these two parts. Of course, according to actual needs, other ways to drive the material to move can be additionally set after the material completes free fall, so that the preset material movement path is composed of at least three parts.
[0123] The preset spraying position can be a position on the crawler or other positions. Similarly, the preset shooting position can also be a position on the crawler or other positions, as long as the material first passes through the preset shooting position and then passes through the preset spraying position when moving on the preset material moving path.
[0124] That is to say, Figure 8 This is just a specific embodiment. In this embodiment, the preset material movement path consists of two parts, namely the path where the material is moved by the track and the free fall of the material after it moves to the end of the track. The above-mentioned preset shooting position is located on the track, and the above-mentioned preset spraying position is located on the preset material movement path and is located in the free fall part of the material.
[0125] At this time, the crawler belt in the crawler belt transmission subsystem 200 drives the material to move, and the camera 401 captures the material image. The material sorting device 400 based on the instance segmentation sorts out the non-compliant material according to the material image. When the non-compliant material moves to the preset ejection position, the jet ejection subsystem 800 ejects the non-compliant material from the preset material movement path. Figure 1 In the illustrated embodiment, the path of the compliant material is path 1, and the path of the non-compliant material is path 2. Since path 1 is included in the preset material movement path, it can be seen that the non-compliant material is sprayed away from the preset material movement path.
[0126] The determination of whether the non-compliant material has reached the preset ejection position may be made by the imaging subsystem 300, by the jet ejection subsystem 400, or by an additionally provided controller.
[0127] The following is an explanation with a specific example.
[0128] In this example, the preset material movement path consists of two parts, namely, the path of the material driven by the crawler and the free fall of the material after it moves to the end of the crawler. The preset shooting position is located at the end of the crawler, that is, the material starts to fall freely after passing the preset shooting position. The preset spraying position is located on the preset material movement path and is located in the free fall part of the material. The imaging subsystem 300 determines whether the non-compliant material reaches the preset spraying position.
[0129] Step 1: According to the application scenario of the material, a crawler with a width of 60cm and a transmission speed of 2m / s is designed; and a camera 401 is placed above the crawler. The camera 401 in this example is a line scan camera with a resolution of 3600×1, which is used to scan the image features of the material.
[0130] The line scan camera can be set to generate a picture after accumulating 2400 lines, and the picture size is 3600×2400, that is, a material image with a size of 3600×2400 is obtained.
[0131] Moreover, after generating a material image of size 3600×2400, the image can be scaled. As an example, the material image of size 3600×2400 is proportionally scaled to 640×427, and the short sides are filled with 0, that is, the short sides are filled with black, so as to obtain a material image of 640×640. At this time, the material sorting device can sort out non-compliant materials according to the material image of 640×640. Moreover, the image can be normalized into a feature vector, so as to obtain an input matrix of size 640×640×3.
[0132] Step 2: At the distance camera detection point (the above preset shooting position) A spray valve is placed at the position of the nozzle. The spray valve is a component of the jet ejection subsystem 800. The spray valve has 64 jet outlets. Each spray valve controls an area of 0.9375 cm on average. The calculation formula is as follows:
[0133]
[0134] Among them, t2 is the time when the material reaches the spray valve port, t1 is the time when the material reaches the camera detection point, and g is the acceleration of gravity.
[0135] Step 3: First, initialize the software and hardware, start the software system, click the unloading button on the UI interface, and the line scan camera will capture images.
[0136] Step 4: Determine which materials are non-compliant materials that need to be rejected by the spray valve, and calculate how long it takes to send the spray valve instruction according to the following formula:
[0137]
[0138] Among them, T0 is the free movement time from the camera detection point to the spray valve, is the height between the track plane and the spray valve, t detector is the model inference time, The time required to send the spray valve command.
[0139] Step 5: Finally, when you reach When the jet ejection subsystem 800 is used, a jet valve instruction is sent through the serial port between the imaging subsystem 300 and the jet ejection subsystem 800 to instruct the jet valve to eject the jet.
[0140] In this way, automatic material sorting based on instance segmentation can be achieved with high efficiency and low manual workload.
[0141] In summary, the material sorting system based on instance segmentation of the embodiment of the present invention, through the material sorting device based on instance segmentation of the above embodiment, is provided with an instance segmentation module for performing instance segmentation on the material image to obtain multiple bounding boxes and segmentation masks, segmentation thresholds, and confidence scores corresponding to the bounding boxes, wherein the confidence score represents the confidence of the material in the material image contained in the bounding box; a segmentation accuracy improvement module, connected to the instance segmentation module, is used to select a first target bounding box from multiple bounding boxes according to the segmentation mask, segmentation threshold, and confidence score corresponding to each bounding box; a sorting module, connected to the segmentation accuracy improvement module, is used to sort out non-compliant materials in the material image according to the first target bounding box and the target segmentation mask, wherein the target segmentation mask is the segmentation mask corresponding to the first target bounding box. In this way, automatic material sorting based on instance segmentation can be achieved with high efficiency and low manual workload.
[0142] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.
[0143] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0144] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0145] In the description of this specification, the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and cannot be understood as a limitation on the present invention.
[0146] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0147] In the description of this specification, unless otherwise specified, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0148] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.
[0149] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. A material sorting device based on instance segmentation, characterized in that: The device comprises: An instance segmentation module, used to perform instance segmentation on the material image to obtain a plurality of bounding boxes and segmentation masks, segmentation thresholds, and confidence scores corresponding to the bounding boxes, wherein the confidence score indicates the confidence that the bounding box contains the material in the material image; A segmentation accuracy improvement module, connected to the instance segmentation module, configured to select a first target bounding box from the plurality of bounding boxes according to the segmentation mask, the segmentation threshold and the confidence score corresponding to each of the bounding boxes; A sorting module is connected to the segmentation accuracy improvement module, and is used to sort out non-compliant materials in the material image according to the first target bounding box and the target segmentation mask, wherein the target segmentation mask is a segmentation mask corresponding to the first target bounding box.
2. The material sorting device based on instance segmentation according to claim 1 is characterized in that: The instance segmentation module comprises: A feature extraction submodule is used to extract features from the material image to obtain a feature map; The segmentation submodule is connected to the feature extraction submodule and is used to perform instance segmentation on the feature map to obtain a plurality of the bounding boxes and segmentation masks, segmentation thresholds, and confidence scores corresponding to the bounding boxes.
3. The material sorting device based on instance segmentation according to claim 2 is characterized in that: The feature extraction submodule includes a first pooling layer, a first convolutional layer, a second pooling layer, a second convolutional layer, a third pooling layer, a third convolutional layer, a fourth pooling layer, a fourth convolutional layer, a fifth pooling layer, a first upsampling layer, a second upsampling layer, a fifth convolutional layer, a sixth convolutional layer, and a first connection layer, which are connected in sequence. The feature extraction module also includes a second connection layer and a third connection layer. The output ends of the third pooling layer and the second upsampling layer are also connected to the input end of the third connection layer, the output ends of the fourth pooling layer and the fifth convolutional layer are also connected to the input end of the second connection layer, and the output end of the fifth pooling layer is also connected to the input end of the first connection layer; The first connection layer is used to output a first feature map, the second connection layer is used to output a second feature map, and the third connection layer is used to output a third feature map.
4. The material sorting device based on instance segmentation according to claim 1, characterized in that: The segmentation accuracy improvement module includes: A multiplication submodule, configured to multiply the segmentation threshold and the segmentation mask corresponding to each of the bounding boxes; A convolution and pooling submodule, configured to perform convolution and pooling processing on each multiplication result to obtain a first fractional value corresponding to the multiplication result, wherein M is a positive integer greater than 1; The first target bounding box selection submodule is used to obtain a second score value for each of the bounding boxes according to the bounding box and the first preset box, and obtain a final score value according to the first score value, the second score value and the target confidence score corresponding to the bounding box, and determine that the bounding box is the first target bounding box when the final score value is greater than a first preset score threshold.
5. The material sorting device based on instance segmentation according to claim 4 is characterized in that: The first target bounding box selection submodule is specifically used for: Substitute the first score value, the second score value, and the target confidence score into a preset calculation formula to obtain the final score value.
6. The material sorting device based on instance segmentation according to claim 1, characterized in that: The sorting module is specifically used for: The size information of the target material is obtained according to the first target bounding box and the target segmentation mask. When the size information is not within a preset size range, the target material is determined to be a non-compliant material, wherein the target material is a material contained in the first target bounding box in the material image.
7. The material sorting device based on instance segmentation according to claim 1, characterized in that: The device also includes: a comparison module, connected to the instance segmentation module, configured to compare the confidence scores corresponding to the bounding boxes with a second preset score threshold, and use the confidence scores greater than the second preset score threshold as the first intermediate scores; a sorting module, connected to the comparison module, configured to arrange the first intermediate scores in descending order to obtain a score sequence arranged in descending order, and select a preset number of scores from the score sequence as second intermediate scores; a selection module connected to the sorting module and the segmentation accuracy improvement module, and configured to obtain an intersection-and-union ratio between each intermediate bounding box and a first preset box, and select at least one second target bounding box from the intermediate bounding box according to the intersection-and-union ratio and the second intermediate score, so that the segmentation accuracy improvement module selects the first target bounding box from the second target bounding box, wherein the intermediate bounding box is a bounding box corresponding to the second intermediate score.
8. The material sorting device based on instance segmentation according to claim 1, characterized in that: The device also includes: An updating module is used to obtain a first loss value according to the first target bounding box and the target segmentation mask, and to obtain a second loss value according to the difference between the first loss value and a preset standard value, and to update at least one of the instance segmentation module, the segmentation accuracy improvement module, and the sorting module according to the second loss value.
9. The material sorting device based on instance segmentation according to claim 8, characterized in that: The update module is specifically used for: A third loss value is obtained according to the first target bounding box and the third preset box, a fourth loss value is obtained according to the target segmentation mask and the preset true mask, and the first loss value is obtained according to the third loss value and the fourth loss value.
10. A material sorting system based on instance segmentation, characterized in that: The system includes a crawler transmission subsystem, an imaging subsystem and an air jet rejection subsystem. The crawler transmission subsystem is used to realize the movement of materials by crawlers, the imaging subsystem is used to photograph the materials and sort out non-compliant materials according to the photographed material images, and the air jet rejection subsystem is used to spray the non-compliant materials moved to the preset spraying position away from the preset material movement path, thereby realizing material sorting; Wherein, the imaging subsystem includes a camera and a material sorting device based on instance segmentation according to any one of claims 1-9, the camera is used to shoot the material at a preset shooting position to obtain the material image, the preset shooting position is located on the preset material movement path, and is located between the preset spraying position and the starting movement position of the material on the crawler, and the material sorting device based on instance segmentation is used to sort out the non-compliant material according to the material image.
Citation Information
Patent Citations
Method for multi-target task instance segmentation
CN111986125A
Head model for instance segmentation, instance segmentation model, image segmentation method and device
CN112508027A
Method and device for instance segmentation, equipment and storage medium
CN114998592A
Biological aeration detection and identification method based on YOLOv5-Segment instance segmentation
CN117115517A
Header Model For Instance Segmentation, Instance Segmentation Model, Image Segmentation Method and Apparatus
US20210312240A1
