Integrated scrap steel grading and foreign body early warning method based on multi-scale image analysis

Abstract

The present invention discloses an integrated scrap steel grading and foreign matter warning method based on multi-scale image analysis, which relates to the field of metal processing technology. The overall information of unloading vehicles and comparative learning methods are used to build a priori knowledge base, and the scrap steel images captured at each layer are divided into major categories. Combined with the prior knowledge and the division results, the quality inspection process is guided in a targeted manner to call different quality inspection models for category identification. At the same time, the introduction of the hole convolution strategy can expand the visual receptive field, and based on this, an encoder and decoder structure that can extract multi-scale information is constructed. The image is convolved and features are extracted through the FPN network, and then the feature maps at different levels are enhanced using a repeated attention mechanism based on channel attention and spatial attention to highlight the foreign matter and impurity areas in the image.

Description

Technical Field

[0001] The present invention relates to the technical field of metal processing, and in particular to an integrated scrap steel grading and foreign matter early warning method based on multi-scale image analysis. Background Art

[0002] In recent years, the academic community has proposed a variety of methods to improve the accuracy of semantic segmentation and target detection tasks. Dilated convolution is one of the most widely used technologies. In deep networks, dilated convolution can increase the receptive field while maintaining the original resolution and making the positioning of the target more accurate. This technology is relatively mature in academia, but it is lacking in the industry. In addition, in the data collection work of actual scenes, there will be a large amount of unused unlabeled data. Compared with manually annotated data, unlabeled data is easy to obtain and large in scale, and has a higher utilization value. In order to utilize these unlabeled data, contrastive learning came into being. The idea of ​​contrastive learning is widely used in model training tasks with a small number of labels. By learning the discriminative information between different categories, full utilization of unlabeled data is achieved.

[0003] Existing automated scrap steel quality inspection and grading systems often only focus on obtaining grading basis directly from images, while ignoring the collection, acquisition and utilization of prior knowledge including raw material sources and overall characteristics, resulting in high manpower investment costs, long data accumulation cycles, single grading basis and insufficient information utilization during system construction. In addition, the scale span between different target instances in the semantic segmentation of scrap steel images is large, and the sample scale is unbalanced and the data volume is small for foreign body warning and impurity identification. In view of this, we propose an integrated scrap steel grading and foreign body warning method based on multi-scale image analysis. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the shortcomings of the prior art, the present invention provides an integrated scrap steel grading and foreign matter early warning method based on multi-scale image analysis, which solves the problems mentioned in the above background technology.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: an integrated scrap steel grading and foreign body early warning method based on multi-scale image analysis, comprising the following steps:

[0008] S1. Construct a priori knowledge base based on the characteristics of scrap steel types at the source of raw material procurement, and collect preliminary information such as the weight of scrap steel loaded on the scrap steel unloading vehicle and the total volume of the carriage.

[0009] S2: Train the recognition model for unloading triggering, complete the automatic capture of new layers in the video surveillance unloading process, and classify the new layers into major categories.

[0010] S3, the quality-guided selector calls the model based on prior knowledge and major category classification results.

[0011] S4. Perform reasoning work based on application scenarios, including detailed processes of quality classification, foreign body identification and impurity detection.

[0012] S5. Complete the output of the vehicle classification result based on the classification and warning information, and complete the push of information to the central control system.

[0013] S6. Set up remote monitoring and human intervention in the quality inspection process to complete the distribution and joint confirmation of grading information to the supplier side.

[0014] Preferably, the step S1 specifically includes:

[0015] S11. According to the distribution of scrap steel purchases and enterprise recycling standards, establish the major classification and sub-classification of scrap steel identification.

[0016] S12. Set up a camera monitoring environment at the unloading site.

[0017] S13. Complete the construction of positive and negative samples required by prior knowledge.

[0018] S14. Perform feature encoding on each category image of each pair of positive sample data through the feature extractor constructed by ResNet.

[0019] S15. Define similarity measurement function .

[0020] S16. The image feature encoder trained in steps S13-S15 is retained as prior knowledge in a prior knowledge base.

[0021] S17. Weigh and measure the volume of the unloading vehicles entering the site, and calculate the density information of the loaded scrap steel.

[0022] Preferably, the step S2 specifically includes:

[0023] S21, based on the labeled signal image dataset obtained in step S12.

[0024] S22. The unloading information will be monitored in real time. After the unloading vehicle enters the site and swipes the card, a card swiping signal will be sent and the grade judgment service will be triggered.

[0025] S23. Triggering an extraction action for the unloading action signal identified by the detection frame.

[0026] S24. Using the manually classified large-category image data obtained in step S12, a single-image classification model based on the ResNet101 network is trained.

[0027] S25. Input each layer unloading image obtained in S23 into the category classifier constructed in step S24 to obtain the major category information contained in the scrap steel image.

[0028] Preferably, the step S3 specifically includes:

[0029] S31. Annotate the major classification images obtained in S12, including scrap steel type annotations for segmentation and quality inspection and detection annotations for foreign body warning and impurity identification.

[0030] S32. The model guide integrates the material type classification information of each layer of the image in steps S24-S25.

[0031] S33, for each layer of image captured in step S23, a foreign body recognition model and an impurity detection model are called to perform target detection.

[0032] Preferably, step S4 comprises:

[0033] S41. After a specific large-category quality inspection model is selected in step S3, the multi-layer scrap steel image is input into the corresponding large-category quality inspection model.

[0034] S42: Perform multi-scale dilated convolution and global image pooling operations on the features obtained in step S41.

[0035] S43, connect the four convolutional layers in step S42 and the global features of the image, and obtain the output of the encoder through a 1×1 ordinary convolution.

[0036] S44, upsample the output of the encoder in S43 by 4 times and concatenate it with the low-level features of the same size as step S41.

[0037] Preferably, the step S4 further comprises:

[0038] S45. Determine the proportion of different types of scrap steel according to the result of image semantic segmentation, and finally determine the category of the scrap steel through dynamic threshold classification.

[0039] S46. In the foreign body recognition and impurity detection part, the data image is first padded to fix its aspect ratio, and finally the image size is adjusted through a multi-resolution image input device for multi-scale training.

[0040] S47. Use the FPN network to extract features from the image to obtain features at different levels.

[0041] S48. Use the output of step S47 as the input of the ROI pooling layer to obtain a feature map of the target level.

[0042] S49, performing area percentage statistics on the pixel-level scrap steel type determination output obtained in step S45 to obtain the number of each small category and the proportion of each large category.

[0043] Preferably, the step S5 specifically includes:

[0044] S51. Summarize the scrap steel type output, foreign matter information and doping status obtained in step S49, and present the visualization results, including highlighting dangerous foreign matter, building a scrap steel category information list for the entire vehicle, and push the grading results to the remote central console.

[0045] S52, completing the settlement of the vehicle grading process corresponding to the unloading vehicle.

[0046] Preferably, the step S6 specifically includes:

[0047] S61. In the scrap steel quality inspection process, the communication between the unloading vehicle information and the prior knowledge base and the real-time monitoring of the scrap steel unloading scene are realized.

[0048] (III) Beneficial effects

[0049] The present invention provides an integrated scrap steel grading and foreign body early warning method based on multi-scale image analysis. It has the following beneficial effects:

[0050] (1) The integrated scrap steel grading and foreign body warning method based on multi-scale image analysis can autonomously output useful results such as scrap steel type determination diagram, foreign body warning that may cause dangerous accidents and scrap steel impurity weight deduction guidance in real time based on video stream information, thereby freeing up manpower and improving production efficiency.

[0051] (2) This integrated scrap steel grading and foreign body warning method based on multi-scale image analysis can make full use of the prior knowledge of scrap steel types and unloading vehicles, rather than taking the image recognition results as the only basis for quality inspection.

[0052] (3) This integrated scrap steel grading and foreign body early warning method based on multi-scale image analysis provides targeted model calling guidance according to different scrap steel categories, achieving a more accurate and fine-grained recognition effect. This method can be changed according to different categories, with high flexibility and recognition coverage.

[0053] (4) The integrated scrap steel grading and foreign body warning method based on multi-scale image analysis is helpful to solve multiple difficulties in scrap steel grading and identification by setting up a model algorithm, and the model algorithm introduces contrastive learning, void convolution and repeated attention mechanism, so as to output more accurate grading and warning results. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] See also Figure 1 The present invention provides a technical solution: an integrated scrap steel grading and foreign body early warning method based on multi-scale image analysis, comprising the following steps:

[0057] S1. Construct a priori knowledge base based on the characteristics of scrap steel types at the source of raw material procurement, and collect prior information such as the weight of scrap steel loaded on the scrap steel unloading vehicle and the total volume of the carriage, including:

[0058] S11. According to the type distribution of scrap steel purchases and the enterprise recycling standards, establish the major category division and sub-category refinement of scrap steel identification. The major category division in this step will serve as the basis for estimating the major category distribution of the whole vehicle and the major category division basis for the subsequent single-image material type classifier. At the same time, the density of the major categories of scrap steel is measured and counted to obtain the density distribution range of each major category of scrap steel, and this information is retained in the prior knowledge base.

[0059] S12. Build a camera monitoring environment at the unloading site. The installed cameras will be used to collect data and obtain the video stream source required for the subsequent quality inspection and early warning system. Monitor the unloading process, obtain a certain amount of video data, extract multi-layer images from the data for subsequent quality inspection and early warning model training, and form a quality inspection and early warning data set to be labeled. The specific extraction rules are to retain the new layer image after each scrap steel in the carriage is grabbed or absorbed, and divide these images into major categories for the subsequent training of single-image material type classifiers; at the same time, the images with suction cups or grippers are used as the unloading signal detection model’s unloading action signal to be labeled. The signal labeling data will be used to train the subsequent unloading signal detection model.

[0060] S13, complete the construction of positive and negative samples required by prior knowledge. All scrap steel in the quality inspection and warning data set obtained in S12 is labeled by category, and image regions of the same category are cropped and extracted to obtain image sets of different categories. , for any set of a specific category, traverse the set, for any picture in it , where i=1, 2, ..., n, n is the number of images in the set, and any other image in the set is selected as the corresponding positive sample data , and these two images constitute a positive sample data pair , for any , the included The pictures are constructed in this way, and a total of For positive sample pairs. At the same time, will and except All images other than compose negative sample data pairs.

[0061] S14, for each category image of each pair of positive sample data, feature encoding is performed through the feature extractor constructed by ResNet, and a vector representing the image feature information is obtained through convolutional neural network CNN conversion. and , and then through the nonlinear transformation structure, the dense feature vector is obtained and , the negative sample data is processed similarly, and the feature vector is obtained and .

[0062] S15. Define similarity measurement function , using the L2 regularized form of Cosine similarity, that is ,in represents the transpose operation, Represents a vector Perform L2 regularization, that is, vector All components of are squared and then squared. The InfoNCE contrast loss function is introduced, and the contrast loss function formula is as follows:

[0063]

[0064] in is a hyperparameter, represents the exponential operation with the natural constant e as the base, The number of images representing the data set reduces the contrast loss of positive samples and increases the contrast loss of negative samples on the representation vector obtained in S12, so that the distance between positive sample pairs is shortened and the distance between negative sample pairs is expanded at the representation space level.

[0065] S16. The image feature encoder trained in steps S13-S15 is retained as prior knowledge in the prior knowledge base for use in downstream semantic segmentation tasks.

[0066] S17, weigh and measure the volume of the unloading vehicles entering the yard, calculate the density information of the loaded scrap steel, and estimate the type distribution of the density information. The above information is retained in the prior knowledge base for training the subsequent model selection guide.

[0067] S2: training the recognition model for unloading triggering, completing the automatic capture of new layers in the video surveillance unloading process, and classifying the new layers into major categories, including:

[0068] S21. Based on the labeled signal image dataset obtained in step S12, a YOLO target detection model is used for training, so that the detection model can complete the recognition of the unloading action signal.

[0069] S22, monitor the unloading information in real time. After the unloading vehicle enters the site and swipes the card, it will send a card swiping signal and trigger the classification service. It will receive the RTSP stream data of the monitoring video, complete the video stream decoding work, and extract the detection frame suspected of containing the unloading action signal at an appropriate time interval, and then send the detection frame to the model trained in S21 for detection.

[0070] S23, triggering an extraction action for the unloading action signal identified by the detection frame to obtain a quality inspection frame for identifying the type of scrap steel, specifically, extracting frames of the required layered images through the OPENCV graphics processing library to obtain multi-layer images after each unloading.

[0071] S24. Using the manually divided large-category image data obtained in step S12, a single-image material type classification model based on the ResNet101 network is trained. This model will be used for subsequent single-image material type classification reasoning.

[0072] S25: Input each layer of unloading images obtained in S23 into the category classifier constructed in step S24 to obtain the major category information contained in the scrap steel image, and complete the positioning of the major category composition of each layer of images. The obtained results will be used in the subsequent model selector to guide the call of the quality inspection model used for this layer of images.

[0073] S3, the quality-guided selector calls the model based on prior knowledge and the results of major classification, including:

[0074] S31. Label the major classification images obtained in S12, including the scrap steel type labels for segmentation and quality inspection and the detection labels for foreign body warning and impurity identification. Use the labeled data to train the semantic segmentation model and the target detection model, and store them in the model guide. The classification standard of each major category of the model guide is the same as the output domain value of the category classifier in S2, that is, for different scrap steel categories, there is a unique corresponding quality inspection model. , and several subcategories contained in this major category can be identified. This classification method will enable the model to identify scrap steel types more specifically.

[0075] S32, the model guide will integrate step S24 and step S25, the material type classification information of each layer of the picture, and the major category distribution information estimated in step S17 to select the corresponding scrap steel category inspection model, so as to perform semantic segmentation on the picture. For example, in step S17, it is estimated that the scrap steel type belongs to the major category between and The model is called.

[0076] S33: For each layer of image captured in step S23, the foreign body recognition model and the impurity detection model are called to perform target detection to obtain information about dangerous foreign bodies and provide guidance on impurity weight deduction.

[0077] S4. Perform reasoning work based on application scenarios, including the detailed process of quality classification, foreign body identification and impurity detection, including:

[0078] S41. After a specific major category quality inspection model is selected in step S3, the multi-layer scrap steel image is input into the corresponding major category quality inspection model. The quality inspection model first uses the image feature encoder retained by the prior knowledge base in S1 to extract a feature map containing prior category distribution information from the image.

[0079] S42, perform multi-scale dilated convolution and image global pooling operations on the features obtained in step S41. Multi-scale dilated convolution includes 1×1 ordinary convolution, 3×3 dilated convolution with a dilation rate of 6, 3×3 dilated convolution with a dilation rate of 12, and 3×3 dilated convolution with a dilation rate of 18. Convolution kernels with different sampling rates and receptive fields can capture objects from multiple scales to achieve the purpose of extracting features at multiple resolutions.

[0080] S43, connect the four convolutional layers in step S42 and the global features of the image, and obtain the output of the encoder through a 1×1 ordinary convolution. The encoder gradually reduces the size of the feature map in the above steps, so as to extract high-level semantic information.

[0081] S44, upsample the output of the encoder in S43 by 4 times and concatenate it with the low-level features of the same size in step S41. The low-level features in step S41 are first reduced in dimension by 1×1 convolution before concatenation. The connected feature map is convolved by 3×3 ordinary convolution and upsampled by 4 times to obtain the decoder output. The decoder gradually restores the spatial and detail information, and obtains the pixel-level classification of the scrap steel image through the output layer.

[0082] S45. Determine the proportion of different types of scrap steel according to the result of image semantic segmentation, and finally determine the category of the scrap steel through dynamic threshold classification.

[0083] S46. In the foreign body recognition and impurity detection part, the data image is first padded to fix its aspect ratio, and finally the image size is adjusted through a multi-resolution image input device for multi-scale training.

[0084] S47. Use the FPN network to extract features from the image to obtain features at different levels. Use the repeated attention mechanism to enhance features at different levels to obtain more robust enhanced features. Fuse the enhanced features output by the repeated attention mechanism and use them as input to the RPN module to generate the initial target candidate box.

[0085] S48: Use the output of step S47 as the input of the ROI pooling layer to obtain the feature map of the target level. Perform linear regression on the feature map of the target level and output the category and target box. Finally, perform post-processing by adjusting the confidence, IOU and other thresholds to filter out irrelevant redundant targets.

[0086] S49: Perform area percentage statistics on the pixel-level scrap steel type determination output obtained in step S45 to obtain the number of each small category and the proportion of each large category. At the same time, the dangerous foreign matter and doping detected in step S48 are archived.

[0087] S5: Output the vehicle classification results based on the classification and warning information, and push the information to the central control system, including:

[0088] The scrap steel type output, foreign matter information and doping status obtained in step S49 are summarized to present the visual results, including eye-catching reminders of dangerous foreign matter, building a scrap steel category information list for the entire vehicle, etc., and completing the push of the grading results to the remote central console.

[0089] S52, completing the settlement of the vehicle grading process corresponding to the unloading vehicle, stopping the quality inspection process of the grading algorithm, releasing equipment resources such as cameras and computing servers, and writing the grading results into the quality inspection database and other operations.

[0090] S6: Remote monitoring and human intervention of the quality inspection process are set up to complete the distribution and joint confirmation of grading information to the supplier side, including:

[0091] S61. In the scrap steel quality inspection process, the communication between the unloading vehicle information and the prior knowledge base and the real-time monitoring of the scrap steel unloading scene are realized. The synchronization of supplier information and the visual presentation of results are completed, and the grading results obtained after the unloading is completed are reviewed.

[0092] The prior knowledge base is constructed by using the overall information of the unloading vehicle and the contrastive learning method. The scrap steel images captured at each layer are divided into major categories. Combined with the prior knowledge and the division results, the quality inspection process is guided in a targeted manner to call different quality inspection models for category identification. At the same time, the introduction of the hole convolution strategy can expand the visual receptive field, and based on this, the encoder and decoder structure that can extract multi-scale information is constructed. The image is convolved and features are extracted through the FPN network, and then the feature maps at different levels are enhanced using the repeated attention mechanism based on channel attention and spatial attention to highlight the foreign matter and impurity areas in the image. Through the above steps, intelligent scrap steel grading and foreign matter warning information are obtained.

[0093] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An integrated scrap steel grading and foreign body early warning method based on multi-scale image analysis, characterized in that: The following steps are involved: S1. Construct a priori knowledge base based on the characteristics of scrap steel types at the source of raw material procurement, and collect prior information such as the weight of scrap steel loaded on the scrap steel unloading vehicle and the total volume of the carriage; S2: training the recognition model for unloading triggering, completing the automatic capture of new layers in the video surveillance unloading process, and classifying the new layers into major categories; S3, the model director calls the model according to the prior knowledge and the classification results; S4: Perform reasoning work based on application scenarios, including detailed processes of quality classification, foreign body identification and impurity detection; S5. Output the vehicle classification results based on the classification and warning information, and push the information to the central control system; S6. Remote monitoring and human intervention are set up for the quality inspection process, and the distribution and joint confirmation of grading information are completed to the supplier side; The step S4 comprises: S41, after a specific large-category quality inspection model is selected in step S3, the multi-layer scrap steel image is input into the corresponding large-category quality inspection model; S42, performing multi-scale hole convolution and image global pooling operations on the features obtained in step S41; S43, connect the four convolutional layers in step S42 and the global features of the image, and obtain the output of the encoder through a 1×1 ordinary convolution; S44, upsample the output of the encoder in S43 by 4 times and concatenate it with the low-level features of the same size as step S41.

2. The integrated scrap steel grading and foreign body early warning method based on multi-scale image analysis according to claim 1 is characterized by: The step S1 specifically includes: S11. According to the distribution of scrap steel purchases and enterprise recycling standards, establish the major classification and sub-classification of scrap steel identification; S12. Set up a camera monitoring environment at the unloading site; S13, complete the construction of positive and negative samples required by prior knowledge; S14, performing feature encoding on each category image of each pair of positive sample data through a feature extractor constructed by ResNet; S15. Define similarity measurement function ; S16, retaining the image feature encoder trained in steps S13-S15 as prior knowledge in a prior knowledge base; S17. Weigh and measure the volume of the unloading vehicles entering the site, and calculate the density information of the loaded scrap steel.

3. The integrated scrap steel grading and foreign body early warning method based on multi-scale image analysis according to claim 2 is characterized by: The step S2 specifically includes: S21, based on the labeled signal image dataset obtained in step S12; S22, monitor the unloading information in real time, and send a card swiping signal after the unloading vehicle enters the site and triggers the grade judgment service; S23, triggering an extraction action for the unloading action signal identified by the detection frame; S24, using the manually classified large-category image data obtained in step S12, training a single-image classification model based on the ResNet101 network; S25. Input each layer unloading image obtained in S23 into the category classifier constructed in step S24 to obtain the major category information contained in the scrap steel image.

4. The integrated scrap steel grading and foreign body early warning method based on multi-scale image analysis according to claim 3 is characterized by: The step S3 specifically includes: S31. Annotate the major classification images obtained in S12, including scrap steel type annotations for segmentation and quality inspection and detection annotations for foreign body warning and impurity identification; S32, the model guide integrates the material type classification information of each layer of the image in steps S24-S25; S33, for each layer of image captured in step S23, a foreign body recognition model and an impurity detection model are called to perform target detection.

5. The integrated scrap steel grading and foreign body early warning method based on multi-scale image analysis according to claim 1 is characterized by: The step S4 further comprises: S45, determining the proportion of different types of scrap steel according to the result of image semantic segmentation, and finally determining the category of the scrap steel through dynamic threshold classification; S46, in the foreign body recognition and impurity detection part, firstly pad the data image to fix its aspect ratio, and finally adjust the image size through the multi-resolution image input device to perform multi-scale training; S47, use FPN network to extract features from the image to obtain features at different levels; S48, using the output of step S47 as the input of the ROI pooling layer to obtain a feature map of the target level; S49, performing area percentage statistics on the pixel-level scrap steel type determination output obtained in step S45 to obtain the number of each small category and the proportion of each large category.

6. The integrated scrap steel grading and foreign body early warning method based on multi-scale image analysis according to claim 5 is characterized by: The step S5 specifically includes: S51, summarizing the scrap steel type output, foreign matter information and doping status obtained in step S49, completing the presentation of visualization results, including providing eye-catching reminders for dangerous foreign matter, building a scrap steel category information list for the entire vehicle, etc., and completing the push of the grading results to the remote central console; S52, completing the settlement of the vehicle grading process corresponding to the unloading vehicle.

7. The integrated scrap steel grading and foreign body early warning method based on multi-scale image analysis according to claim 6 is characterized by: The step S6 specifically includes: S61. In the scrap steel quality inspection process, the communication between the unloading vehicle information and the prior knowledge base and the real-time monitoring of the scrap steel unloading scene are realized.

Images