Scrap classification method based on small target data enhancement and multi-view collaborative reasoning

Abstract

The application discloses a kind of based on small target data enhancement and multi-view collaborative inference scrap classification method, it is related to metal recovery technical field.The application discloses a kind of based on small target data enhancement and multi-view collaborative inference scrap classification method, it is related to metal recovery technical field.The application discloses a kind of based on small target data enhancement and multi-view collaborative inference scrap classification method, it is related to metal recovery technical field.The application discloses a kind of based on small target data enhancement and multi-view collaborative inference scrap classification method, it is related to metal recovery technical field.The application discloses a kind of based on small target data enhancement and multi-view collaborative inference scrap classification method, it is related to metal recovery technical field.The application discloses a kind of based on small target data enhancement and multi-view collaborative inference scrap classification method, it is related to metal recovery technical field.The application discloses a kind of based on small target data enhancement and multi-view collaborative inference scrap classification method, it is related to metal recovery technical field.The application discloses a kind of based on small target data enhancement and multi-view collaborative inference scrap classification method, it is related to metal recovery technical field.The application discloses a kind of based on small target data enhancement and multi-view collaborative inference scrap classification method, it is related to metal recovery technical field.The application discloses a kind of based on small target data enhancement and multi-view collaborative inference scrap classification method, it is related to metal recovery technical field.The application discloses a kind of based on small target data enhancement and multi-view collaborative inference scrap classification method, it is related to metal recovery technical field.The application discloses a kind of based on small target data enhancement and multi-view collaborative inference scrap classification method, it is related to metal recovery technical field.The application discloses a kind of based on small target data enhancement and multi-view collaborative inference scrap classification method, it is related to metal recovery technical field.The application discloses a kind

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of metal recycling, in particular to a waste steel classification method based on small target data enhancement and multi-view collaborative reasoning. BACKGROUND

[0002] In the current steel production process, in order to reduce costs and improve smelting efficiency, waste steel is usually recycled for remelting and reuse. Due to the large amount of waste steel used, mixed loading of multiple types of materials, and the phenomenon of waste steel adulteration, in order to ensure product quality and increase steel production, high-precision grading and classification of waste steel is required. The traditional waste steel grading is greatly influenced by human subjective factors, and requires high personnel requirements. Generally, it requires familiarity with standards and rich experience to judge, and there are differences in judgment by each person. Fatigue, mood, etc. may affect the evaluation results. There is no quantitative evaluation conclusion, and it is difficult to form good data analysis and make suppliers convinced. At the same time, the waste steel grading environment is relatively poor, and the quality inspector needs to climb four or five meters to the top of the truck each time to observe the waste steel in the truck at close range. The labor intensity is high, and the operation risk is high. How to solve the many problems existing in the traditional waste steel grading and realize the intelligent waste steel grading meeting the requirements of the new industrial revolution is the focus of steel enterprises.

[0003] Under the background of rapid development of deep learning, some waste steel intelligent grading algorithms based on deep learning have appeared. However, small waste steel is difficult to detect intelligently due to its small size, limited available features, and easy to be blocked by other waste steel, resulting in low grading accuracy and large errors. In view of this, a waste steel classification method based on small target data enhancement and multi-view collaborative reasoning is proposed. SUMMARY

[0004] (I) Technical problems solved

[0005] In view of the deficiencies of the prior art, the present application provides a waste steel classification method based on small target data enhancement and multi-view collaborative reasoning, which solves the problems mentioned in the background art.

[0006] (II) Technical solutions

[0007] To achieve the above purpose, the present application is realized by the following technical solutions: a waste steel classification method based on small target data enhancement and multi-view collaborative reasoning, comprising the following steps:

[0008] S1: Establish a multi-view waste steel picture data set to store multi-view waste steel pictures at the same time.

[0009] S2: Design a clipping strategy for high-resolution images, clip the input picture and enhance the image.

[0010] S3: using a pre-trained ResNet50 to extract scrap steel picture features to generate a feature map.

[0011] S4: inputting the feature map generated in S3 into a multi-view collaborative network to extract consistency and complementary feature information between scrap steel pictures of different views.

[0012] S5: weighting the consistency and complementary information extracted in S4 and fusing it with the original feature map to form a new feature map, and using the consistency and complementary information for classification.

[0013] S6: combining the new feature map in S5, inputting it into the RPN network and performing RoiAlign operation, inputting it into Mask-Rcnnhead to realize classification, detection and segmentation of small scrap steel.

[0014] S7: predicting the input picture to obtain classification, detection and segmentation results.

[0015] Preferably, the step S4 specifically comprises:

[0016] S41: dividing the multi-view feature map into a main view feature map m and two auxiliary view feature maps a1 and a2.

[0017] S42: splicing the two auxiliary view feature maps a1 and a2 to form an auxiliary view feature map a.

[0018] S43: splicing the main view feature map m in S42 and the auxiliary view feature map a, inputting them into a softmax function after passing through a fully connected layer to obtain a correlation score vector corresponding to each dimension of the main view feature map, which is expressed as follows:

[0019] s m =softmax(W h ·[m,a])

[0020] S44: splicing the auxiliary view feature map a in S42 and the main view feature map m, inputting them into a softmax function after passing through a fully connected layer to obtain a correlation score vector corresponding to each dimension of the auxiliary view feature map, which is expressed as follows:

[0021] s a =softmax(W a ·[a,m])

[0022] S45: using the correlation score vectors in steps S43 and S44 to distinguish the consistency and complementary features of different views, and using a sigmoid function to make them continuous, which is expressed as follows:

[0023]

[0024] S46: By taking the inner product of the four weight vectors in S45 and the main and auxiliary view feature maps m and a, the consistency and complementarity information in the main and auxiliary view feature maps can be obtained, and the formula is as follows:

[0025]

[0026]

[0027]

[0028]

[0029] S47: The consistent main vector v m and the consistent auxiliary vector v a are spliced, input into a neural network and an activation function, and the enhanced consistent vector information is obtained, and the formula is as follows:

[0030]

[0031] S48: Calculate the divergence loss of KL, and the formula is as follows:

[0032] p o = softmax(U o ·v o )

[0033]

[0034] Where p o is the possibility distribution of the scrap steel category, and U o is a trainable weight matrix.

[0035] Preferably, the step S1 specifically comprises:

[0036] S11: Define all scrap steel categories that can be contained in the scene.

[0037] S12: Name the scrap steel pictures, and determine the different view pictures belonging to the same moment from the name.

[0038] S13: Use the labelImg software to segment and label the scrap steel pictures to obtain a label file storing the labels.

[0039] Preferably, the step S2 specifically comprises:

[0040] S21: Data augmentation is performed on the data set, and the pictures are cropped, and several cropping strategies are proposed.

[0041] S22: Random sliding window cutting strategy.

[0042] Preferably, the step S3 specifically comprises:

[0043] S31: Use the Resnet50 model pre-trained on the dataset ImageNet as the original model for scrap steel picture feature extraction.

[0044] S32: Input the scrap steel picture into the Resnet50 model for feature extraction, and obtain the feature map output by the model.

[0045] Preferably, the step S5 specifically comprises:

[0046] S51: Input the consistency complementary information vector with comprehensive information into a neural network and an activation function to obtain the weighted consistency complementary information vector x ma , and the formula is as follows:

[0047]

[0048] S52: Aggregate the weighted consistency complementary information vector with the main view feature map m and the two auxiliary view feature maps a1 and a2 in step S41 respectively to form the fused main view feature map m' and auxiliary view feature maps a1' and a2'.

[0049] 7. The scrap steel classification method based on small target data enhancement and multi-view collaborative inference according to claim 6, wherein the step S6 specifically comprises:

[0050] S61: Input the main view feature map m' and the auxiliary view feature maps a1' and a2' obtained in step S5 into the RPN network respectively to obtain the candidate region.

[0051] S62: According to the candidate region, perform RoiAlign operation on the feature map to obtain the fixed size feature map.

[0052] S63: Input the fixed size feature map into Mask-Rcnn head to perform scrap steel classification, detection and segmentation, and obtain the classification loss L cls , the detection loss L bbox and the segmentation loss L mask .

[0053] S64: Add the KL divergence loss L KL obtained in step S4 to the three losses in step S63 to obtain the final loss L final .

[0054] Lfinal = L cls + L bbox + L mask + L KL

[0055] S65: At this point, the model training is completed.

[0056] Preferably, the step S7 specifically comprises:

[0057] S71: Input three pictures of different perspectives at the same time, and crop the three pictures according to the operation in step S21, and input the cropped sub-pictures into the model to obtain classification, detection and segmentation results.

[0058] S72: Merge the classification, detection and segmentation results of the sub-pictures belonging to the same original picture according to the positions of the sub-pictures in the original picture, and visualize in the original picture.

[0059] S73: At this point, the model prediction is completed.

[0060] (Three) beneficial effects

[0061] The application provides a scrap steel classification method based on small target data enhancement and multi-perspective collaborative reasoning. The following beneficial effects are provided:

[0062] (1) The scrap steel classification method based on small target data enhancement and multi-perspective collaborative reasoning uses a ResNet50 model to extract features from the cropped sub-pictures of different perspectives, inputs the feature maps of different perspectives into a multi-perspective collaborative network to obtain consistent and complementary feature information, fuses the feature information with the original feature map, and inputs the fused feature information into a subsequent Mask-Rcnnhead for reasoning and gives classification, detection and segmentation information. A cropping strategy is designed for high-resolution images, the image is enhanced by cropping operation, the area ratio of small scrap steel in each picture is improved, and sufficient features are extracted for recognition, which can improve the detection rate and classification accuracy of small scrap steel.

[0063] (2) The scrap steel classification method based on small target data enhancement and multi-perspective collaborative reasoning proposes a multi-perspective collaborative network, which assigns a relationship score to each dimension of the input feature, thereby dividing the feature into consistent and complementary features. The network can fully utilize the information provided by different perspectives, divide the information of different perspectives into consistent and complementary feature information, and use the information of different perspectives at the same time to optimize classification, so that the accuracy of the obtained results is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 The present application is a general flowchart;

[0065] Figure 2 The flowchart of step S2 of the present application is shown in the figure;

[0066] Figure 3 The flowchart of step S4 of the present application is shown in the figure;

[0067] Figure 4 The flowchart of step S5 of the present application is shown in the figure;

[0068] Figure 5 The flowchart of step S6 of the present application is shown in the figure. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0070] Please refer to Figures 1-5 The present application provides a technical solution: a scrap steel classification method based on small target data enhancement and multi-view collaborative reasoning, comprising the following steps:

[0071] S1: Establish a multi-view scrap steel picture data set to store multi-view scrap steel pictures at the same time.

[0072] S2: Design a clipping strategy for high-resolution images, clip the input picture and enhance the image.

[0073] S3: Use a pre-trained ResNet50 to extract scrap steel picture features and generate a feature map.

[0074] S4: Input the feature map generated in S3 into a multi-view collaborative network to extract consistent and complementary feature information between different view scrap steel pictures.

[0075] S5: Weight the consistent and complementary information extracted in S4 and fuse it with the original feature map to form a new feature map, and use the consistent and complementary information for classification.

[0076] S6: Combine the new feature map in S5, input it into the RPN network, perform RoiAlign operation, input Mask-Rcnnhead, and realize classification, detection and segmentation of small scrap steel;

[0077] S7: Predict the input picture to obtain classification, detection and segmentation results;

[0078] In the embodiment, step S1 specifically comprises:

[0079] S11: define all scrap steel categories that can be contained in this scene.

[0080] S12: name the scrap steel pictures, from which it can be determined that different perspective pictures belong to the same moment.

[0081] S13: use the labelImg software to segment and label the scrap steel pictures, and obtain a label file storing the labels.

[0082] In this embodiment, step S2 specifically comprises:

[0083] S21: perform data enhancement on the data set, and crop the pictures.

[0084] S22: random sliding window cutting strategy: the size of the original picture to be cut is h: 3840, w: 2160, the cutting size is 640*640, the overlap ratio is 0.2, and the step is 512. The cutting is sequentially slid from left to right and from top to bottom. The center cutting strategy is to traverse each bounding box, taking the bounding box as the center coordinates of the cutting. Considering that the center cutting introduces prior information, that is, the center of each cutting will have a bounding box, randomness is introduced. When traversing each bounding box, random coordinates of the bounding box in the cutting coordinate system are generated, so as to realize the random cutting strategy.

[0085] S23: on the basis of the random sliding window cutting strategy, sampling rules are added considering various situations, and a regular sliding window cutting strategy is proposed. The sampling standards for any cutting mainly include the following rules:

[0086] (1) the target upper left corner is in the cutting, and the target lower right corner is in the cutting: the target is complete in the cutting, the cutting and the target are retained, and it is judged whether the following rules are followed.

[0087] (2) the target upper left corner is above the cutting, and the target lower right corner is in the cutting: the lower half of the target in the cutting is retained, and in order to ensure that the target is as complete as possible, if the target size is not less than 200*200, it is determined to be effective sampling, the cutting and the target are retained, and it is judged whether the following rules are followed.

[0088] (3) the target upper left corner is above the cutting, and the target lower right corner is on the right side of the cutting area: the left lower part of the target in the cutting is retained, and in order to ensure that the target is as complete as possible, if the target size is not less than 200*200, it is determined to be effective sampling, the cutting and the target are retained, and it is judged whether the following rules are followed.

[0089] (4) The upper left corner of the target is on the left side of the cutout, and the lower right corner of the target is in the cutout: the right half of the target in the cutout is retained, to ensure that the target is as complete as possible, if the size of the target is not less than 200*200, it is determined that the sampling is effective, the cutout and the target are retained, and whether the following rules are followed is determined.

[0090] (5) The upper left corner of the target is on the left side of the cutout, and the lower right corner of the target is on the lower side of the cutout: the upper right part of the target in the cutout is retained, to ensure that the target is as complete as possible, if the size of the target is not less than 200*200, it is determined that the sampling is effective, the cutout and the target are retained, and whether the following rules are followed is determined.

[0091] (6) The upper left corner of the target is on the upper left side of the cutout, and the lower right corner of the target is in the cutout: the lower right part of the target in the cutout is retained, to ensure that the target is as complete as possible, if the size of the target is not less than 200*200, it is determined that the sampling is effective, the cutout and the target are retained, and whether the following rules are followed is determined.

[0092] (7) The upper left corner of the target is in the cutout, and the lower right corner of the target is on the lower right side of the cutout: the upper left part of the target in the cutout is retained, to ensure that the target is as complete as possible, if the size of the target is not less than 200*200, it is determined that the sampling is effective, the cutout and the target are retained, and whether the following rules are followed is determined.

[0093] (8) The upper left corner of the target is in the cutout, and the lower right corner of the target is on the lower side of the cutout: the upper half of the target in the cutout is retained, to ensure that the target is as complete as possible, if the size of the target is not less than 200*200, it is determined that the sampling is effective, the cutout and the target are retained, and whether the following rules are followed is determined.

[0094] (9) The upper left corner of the target is in the cutout, and the lower right corner of the target is on the right side of the cutout: the left half of the target in the cutout is retained, to ensure that the target is as complete as possible, if the size of the target is not less than 200*200, it is determined that the sampling is effective, the cutout and the target are retained, otherwise it is determined that there is no corresponding target in the cutout to add to the negative sample candidate set.

[0095] In order to balance the proportion of positive and negative samples, a negative sample collection strategy is proposed, negative samples are introduced, and the false detection rate is reduced. The negative sample candidate set is composed of cutouts that are determined to have no corresponding target in the rule sliding window cutout strategy. The actually collected negative samples are selected from 20% of the negative sample candidate set according to the random principle.

[0096] In this embodiment, step S3 specifically comprises:

[0097] S31: using a Resnet50 model pre-trained on a data set ImageNet as an original model for feature extraction of scrap steel pictures.

[0098] S32: input the scrap steel picture into the Resnet50 model to perform feature extraction and obtain a feature map output by the model.

[0099] In this embodiment, step S4 specifically includes:

[0100] S41: divide the multi-view feature map into a main-view feature map m and two auxiliary-view feature maps a1 and a2.

[0101] S42: splice the two auxiliary-view feature maps a1 and a2 to form an auxiliary-view feature map a.

[0102] S43: splice the main-view feature map m and the auxiliary-view feature map a in S42, input the spliced result into a softmax function after passing through a full connection layer, and obtain a correlation score vector corresponding to each dimension of the main-view feature map, which is expressed by the following formula:

[0103] s m =softmax(W h ·[m,a])

[0104] wherein W m represents a learnable weight matrix, D m represents the dimension of the main-view feature map, D represents the dimension of the full connection vector, and s a represents the correlation score vector corresponding to each dimension of the main-view feature map.

[0105] S44: splice the auxiliary-view feature map a and the main-view feature map m in S42, input the spliced result into a softmax function after passing through a full connection layer, and obtain a correlation score vector corresponding to each dimension of the auxiliary-view feature map, which is expressed by the following formula:

[0106] s a =softmax(W a ·[a,m])

[0107] wherein W a represents a learnable weight matrix, D o represents the dimension of the auxiliary-view feature map, D represents the dimension of the full connection vector, and s o represents the correlation score vector corresponding to each dimension of the auxiliary-view feature map.

[0108] S45: use the correlation score vectors in steps S43 and S44 to distinguish the consistency and complementarity features of different views; set a threshold value ξ o for training, wherein o∈{m,a}; the threshold value can divide the correlation score vector into a consistency weight vector and a complementarity weight vector, which are distinguished by the following formula:

[0109]

[0110] where γ o [i] is the consistency weight vector γ o is the value of the i-th dimension, indicating the degree of consistency.

[0111]

[0112] where δ o [i] is the complement weight vector δ o is the value of the i-th dimension, indicating the degree of complement.

[0113] Since the above two formulas are discontinuous, a sigmoid function is used to make them continuous, as follows:

[0114]

[0115] where w is a trainable weight that measures the difference between s o [i] and ξ o , making γ o [i] as close to 0 or s o [i] as possible; after this step, two consistency weight vectors γ m , γ a and two complement weight vectors δ m , δ a are obtained.

[0116] S46: By taking the inner product of the four weight vectors in S45 with the main-view feature map and the auxiliary-view feature map m and a, the consistency and complement information in the main-view feature map and the auxiliary-view feature map can be obtained, as follows:

[0117]

[0118]

[0119]

[0120]

[0121] where μ m is the complement main vector, indicating the complement information in the main-view feature map; μ a is the complement auxiliary vector, indicating the complement information in the auxiliary-view feature map, ν m is the consistent main vector, indicating the consistent information in the main-view feature map; ν a is the consistent auxiliary vector, indicating the consistent information in the auxiliary-view feature map.

[0122] S47: In order to better utilize the relationship between consistent information, the consistent principal vector v m is spliced with the consistent auxiliary vector v a , input into a neural network and an activation function, and the enhanced consistent vector information is obtained, and the formula is as follows:

[0123]

[0124] where W ν is a trainable weight matrix, is an activation function, is the enhanced consistent vector information; in order to better combine complementary information and consistent information, the complementary principal vector μ m , the enhanced consistent vector and the complementary auxiliary vector μ a are spliced to obtain the consistent complementary information vector with comprehensive information The formula is

[0125] S48: In order to make the consistent principal vector v m and the consistent auxiliary vector v a as similar as possible in the scrap steel category prediction distribution, the KL divergence loss between them needs to be calculated, and the formula is as follows:

[0126] p o = softmax(U o · v o )

[0127]

[0128] where p o is the possibility distribution of the scrap steel category, U o is a trainable weight matrix, and L KL is the calculated KL divergence loss.

[0129] In this embodiment, step S5 specifically comprises:

[0130] S51: The consistent complementary information vector with comprehensive information is input into a neural network and an activation function, and the weighted consistent complementary information vector x ma is obtained, and the formula is as follows:

[0131]

[0132] where W is a trainable weight matrix, is the consistent complementary information vector with comprehensive information, and x mais a weighted consistency complementary information vector.

[0133] S52: the weighted consistency complementary information vector and the two auxiliary view feature maps a1 and a2 in step S41 are aggregated respectively to form the fused main view feature map m' and auxiliary view feature maps a1' and a2'.

[0134] In this embodiment, step S6 specifically includes:

[0135] S61: input the main view feature map m' and auxiliary view feature maps a1' and a2' obtained in step S5 into the RPN network respectively to obtain the candidate region.

[0136] S62: according to the candidate region, perform RoiAlign operation on the feature map to obtain a fixed size feature map.

[0137] S63: input the fixed size feature map into Mask-Rcnn head to perform classification, detection and segmentation of scrap steel, and obtain classification loss L cls , detection loss L bbox and segmentation loss L mask .

[0138] S64: add the KL divergence loss L KL obtained in step S4 to the three losses in step S63 to obtain the final loss L final .

[0139] L final = L cls + L bbox + L mask + L KL

[0140] S65: at this point, the model training is completed.

[0141] In this embodiment, step S7 specifically includes:

[0142] S71: input three pictures of different views at the same time, and crop the three pictures according to the operation in step s21, and input the cropped sub-pictures into the model to obtain classification, detection and segmentation results.

[0143] S72: merge the classification, detection and segmentation results of the sub-pictures belonging to the same original picture according to the positions of the sub-pictures in the original picture, and visualize in the original picture.

[0144] S73: at this point, the model prediction is completed.

[0145] The scrap steel classification method based on small target data enhancement and multi-view collaborative reasoning assigns a relationship score to each dimension of the input feature, thereby classifying the feature into consistent features and complementary features. The network can fully utilize the information provided by different views, classify the information of different views into consistent feature information and complementary feature information, use the information of different views at the same time to optimize the classification, and further improve the accuracy of the obtained results.

[0146] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for scrap classification based on small target data augmentation and multi-view collaborative inference, characterized in that: The method comprises the following steps: S1: establishing a multi-view scrap steel picture dataset, and storing multi-view scrap steel pictures at the same time; S2: designing a clipping strategy for a high-resolution image, clipping the input picture, and performing image enhancement; S3: extracting scrap steel picture features by using a pre-trained ResNet50 model, and generating a feature map; S4: inputting the feature map generated in S3 into a multi-view collaborative network to extract consistency and complementary feature information between different view scrap steel pictures; S5: weighting the consistency and complementary information extracted in S4, fusing the original feature map, forming a new feature map, and classifying by using the consistency and complementary information; S6: combining the new feature map in S5, inputting the RPN network, performing RoiAlign operation, inputting the Mask-Rcnn head, and realizing classification, detection and segmentation of small scrap steel; S7: predicting the input picture to obtain classification, detection and segmentation results; The step S4 specifically comprises: S41: divide the multi-view feature map into one main-view feature map and two auxiliary-view feature maps and ; S42: splice the two auxiliary-view feature maps and to form an auxiliary-view feature map ; S43: concatenate the main-view feature map in S42 with the auxiliary-view feature map and the auxiliary-view feature map , pass through a fully connected layer, and input into a softmax function to obtain a correlation score vector corresponding to each dimension of the main-view feature map, which is expressed by the following formula: ， wherein represents a learnable weight matrix, wherein represents the dimension of the main-view feature map, represents the dimension of the comprehensive vector, represents the associated score vector corresponding to each dimension of the main-view feature map; S44: splice the auxiliary-view feature map in S42 with the main-view feature map After a full connection layer, input the softmax function to obtain the correlation score vector of each dimension of the corresponding auxiliary-view feature map, which is expressed as follows: ， wherein represents a learnable weight matrix, wherein represents the dimension of the auxiliary-view feature map, represents the dimension of the comprehensive vector, represents the associated score vector for each dimension of the auxiliary-view feature map; S45: Using the correlation score vector in steps S43 and S44, the consistency and complementarity features of different perspectives can be distinguished; set the threshold value for training wherein ; the threshold value can divide the correlation score vector into: consistency weight vector and complementarity weight vector, which is specifically distinguished by the following formula: ， wherein points to a consistency weight vector first dimension, representing the degree of consistency; ， wherein pointing complementary weight vector first dimension, representing the degree of complementarity; Since the above two formulas are discontinuous, a sigmoid function is used to make them continuous, and the formula is as follows: ， wherein is a weight that can be trained to measure the difference between and 0 or as close as possible; through this step, two consistency weight vectors and two complementary weight vectors are obtained; S46: obtain the consistency and complementary information in the main-view feature map and the auxiliary-view feature map by the 4 weight vectors in S45 and inner product, to obtain the consistency and complementary information in the main-view feature map and the auxiliary-view feature map, and the formula is as follows: ， ， ， ， wherein is a complementary principal vector representing complementary information in the principal view feature map; is a complementary auxiliary vector representing complementary information in the auxiliary view feature map, is a consistent principal vector representing consistent information in the principal view feature map; is a consistent auxiliary vector representing consistent information in the auxiliary view feature map; S47: Concatenate the consistent principal vector with the consistent auxiliary vector to input a neural network and an activation function to obtain the enhanced consistent vector information, as follows: ， wherein is a trainable weight matrix, is an activation function, is the enhanced consistent vector information; in order to better combine the complementary information and the consistent information, the complementary principal vector , the enhanced consistent vector , the complementary auxiliary vector are spliced to obtain the consistent complementary information vector with comprehensive information , and the formula is ; S48: calculating the divergence loss of KL, and the formula is as follows: ， ， wherein, is a likelihood distribution of scrap steel classes, is a trainable weight matrix, is the computed KL divergence loss; The step S5 specifically comprises: S51: obtaining a consistency complementarity information vector with comprehensive information inputting a neural network and an activation function to obtain a weighted consistency complementarity information vector , as follows: ， wherein, is a trainable weight matrix, is a consistency complementary information vector with full information; S52: aggregate the weighted consistency complementarity information vectors with one main-view feature map in step S41 with two auxiliary-view feature maps and respectively to form fused main-view feature maps with auxiliary-view feature maps and .

2. The method of claim 1, wherein the method is based on small target data augmentation and multi-view collaborative inference. The step S1 specifically comprises: S11: defining all scrap steel categories that can be contained in the scene; S12: naming the scrap steel pictures, and determining different view pictures belonging to the same time from the name; S13: using labelImg software to perform segmentation annotation on the scrap steel pictures to obtain an annotation file storing labels.

3. The method of claim 2, wherein the method is based on small target data augmentation and multi-view collaborative inference. The step S2 specifically comprises: S21: performing data enhancement on the dataset, clipping the picture, and proposing several clipping strategies; S22: a random sliding window cutting strategy.

4. The method of claim 3, wherein the method is based on small target data augmentation and multi-view collaborative inference. The step S3 specifically comprises: S31: using a Resnet50 model pre-trained on a dataset ImageNet as an original model for scrap steel picture feature extraction; S32: inputting the scrap steel picture into the Resnet50 model to perform feature extraction, and obtaining a feature map output by the model.

5. The method of claim 3, wherein the method is characterized by: The step S6 specifically comprises: S61: Transfer the main view feature map obtained in step S5 Feature map with auxiliary view and Input the data into the RPN network to obtain candidate regions; S62: performing RoiAlign operation on the feature map according to the candidate region to obtain a fixed-size feature map; S63: input the fixed-size feature map into the Mask-Rcnn head to perform classification, detection and segmentation of the scrap steel, and obtain a classification loss , a detection loss , and a segmentation loss ​ S64: add the KL divergence loss obtained in step S4 with the three losses in step S63 to obtain the final loss ; = + + + ， S65: at this time, the model training is completed.

6. The method of claim 5, wherein the method is based on small target data augmentation and multi-view collaborative inference. The step S7 specifically comprises: S71: inputting three pictures of different views at the same time, clipping the three pictures according to the operation in step S21, and inputting the clipped sub-pictures into the model to obtain classification, detection and segmentation results; S72: combining the classification, detection and segmentation results of the sub-pictures belonging to the same original picture according to the positions of the sub-pictures in the original picture, and visualizing in the original picture; S73: at this time, the model prediction is completed.

Images