A real-time monitoring method for oversize materials based on visual tracking

Abstract

The application discloses a kind of real-time monitoring method of screen above impurities based on visual tracking, camera 1 obtains the training sample set of screen above impurities by establishing screen front side image;According to the different illumination condition, the training sample set is expanded using generative adversarial network, and the semantic segmentation detection model is obtained based on the training of depth semantic segmentation network;According to the result of semantic segmentation detection, draw the positive circumscribed rectangle in original drawing and intercept the detected impurity target;Through camera 2, the impurity target feature detected by the semantic segmentation detection model is extracted for feature matching, and the impurity position in camera 2 is detected;With the impurity target detected by feature matching detection as initial frame, target tracking is carried out;Fusion semantic segmentation and target tracking information, complete impurity real-time positioning tracking.The degree of automation of the scheme is higher, and the accurate monitoring of various types of impurities can be completed. Through double industrial cameras, the impurities in the screen above coal are monitored in an open scene, which is more efficient than a single camera.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mineral processing, in particular to a real-time monitoring method for on-screen sundries based on visual tracking. BACKGROUND

[0002] In the process of coal mining and coal preparation, a large amount of sundries such as anchor rods, support nets, iron wires, blasting devices, household garbage and equipment parts will be produced. The sundries mixed in the coal flow not only affect the normal coal preparation process, but also reduce the quality of commercial coal if they enter the product. Therefore, effective means need to be taken to remove sundries in the coal flow.

[0003] At present, the methods for removing sundries in coal mainly include hand sorting belt, iron remover, sundry blocking hook and sundry removing grate, etc. These methods have low automation degree and cannot meet the requirement of accurately removing sundries. With the development of intelligentization of coal preparation plants, some coal preparation plants begin to use mechanical hands to replace the sundry picking personnel in the hand sorting belt link. However, in the background of the elimination of hand sorting belt, the sundry removing mechanical hand increasingly loses installation space, and the use of computer vision and mechanical hand for on-screen sundry removal becomes a feasible scheme.

[0004] However, in the field of computer vision, due to the different shapes of sundries, the rectangular frame provided by target detection cannot accurately guide the operation of the subsequent grabbing mechanism, and the pixel-by-pixel classification semantic segmentation is more suitable for on-screen sundry monitoring tasks. Therefore, in some schemes, the recognition rate and speed of sundries are low. SUMMARY

[0005] The purpose of the present application is to provide a real-time monitoring method for on-screen sundries based on visual tracking, which can monitor the sundries on the screen in an open scene by using double industrial cameras, effectively improve the recognition accuracy and speed, and provide a technical basis for subsequent automatic removal of sundries.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] The present application provides a real-time monitoring method for on-screen sundries based on visual tracking, which comprises:

[0008] Step 1: acquiring the image in front of the screen by camera 1, and establishing the training sample set of on-screen sundries;

[0009] Step 2: performing filtering and noise reduction processing, uneven illumination processing and motion blur elimination processing on the training sample set;

[0010] Step 3: according to the different illumination conditions, expanding the training sample set by using a generative adversarial network to obtain the processed training sample set;

[0011] Step 4, training based on a deep semantic segmentation network for the processed training sample set in step 3 to obtain a semantic segmentation detection model, wherein the current processed training sample set is input into the semantic segmentation detection model to obtain a semantic segmentation detection result;

[0012] Step 5, drawing a positive outer rectangle on the original image according to the semantic segmentation detection result, and cutting the detected foreign matter target;

[0013] Step 6, acquiring a screen rear-end image by a camera 2 and performing perspective conversion;

[0014] Step 7, extracting a foreign matter target feature detected by the semantic segmentation detection model, and performing feature matching with the image in step 6 to detect a foreign matter position in the camera 2;

[0015] Step 8, taking the foreign matter target detected by the feature matching as an initial frame to perform target tracking;

[0016] Step 9, fusing semantic segmentation and target tracking information to complete foreign matter detection.

[0017] Optionally, in step 1, the process of acquiring a screen front-end image by the camera 1 is specifically:

[0018] An open image acquisition device is built on the screen front end, an industrial camera is vertically arranged with the screen surface, and an image acquisition area is between the first beam and the excitation beam to acquire position and shape information of the foreign matter on the screen front side.

[0019] Optionally, in step 2, the process of filtering and denoising, uneven illumination processing, and motion blur elimination for the training sample set is specifically:

[0020] A spatial domain filter is used to reduce image noise, gamma correction and histogram equalization are used for adaptive correction of uneven illumination images, and a fuzzy motion kernel is used to reduce image motion blur.

[0021] Optionally, in step 3, according to different illumination conditions, the process of expanding the training sample set by using a generative adversarial network is specifically:

[0022] According to the illumination condition, the image data is divided into multiple categories, and a CycleGAN network is used to perform style transfer from each category to other categories to expand the training sample set.

[0023] Optionally, in step 4, the processed training sample set is trained based on a deep semantic segmentation network, and a post-processing operation is performed, and the process is specifically:

[0024] The processed training sample set is input, and the deep semantic segmentation network includes but is not limited to UNet, Segmenter, and TransUNet semantic segmentation network. An encoder in the deep semantic segmentation network extracts features of an image, and a decoder restores the features to obtain a sundry segmentation region.

[0025] The post-processing operation is to use a skeleton extraction algorithm to refine a connected region of a target mask in the semantic segmentation region into a skeleton with a pixel width, and calculate a skeleton centroid position as a grabbing point position.

[0026] Optionally, in step 5, the process of intercepting the detected sundry target is specifically as follows:

[0027] A connected region is found in the segmentation result, a circumscribed rectangle of the connected region is drawn, the sundry target is intercepted according to the position of the circumscribed rectangle, and a ROI region of the target sundry is obtained as a feature matching template provided by the camera 2.

[0028] Optionally, in step 6, the process of acquiring a screen machine rear-end image by the camera 2 and performing perspective conversion is specifically as follows:

[0029] The camera 2 is installed at the rear side of the vibration beam, the shooting perspective is oblique top view, and the image acquisition region is the region below and behind the vibration beam. A trapezoidal region is selected in the image acquired by the camera 2 as an original image for perspective transformation, and the oblique top view is transformed into a top view by using a perspective transformation formula.

[0030] Optionally, in step 7, the process of feature matching is specifically as follows:

[0031] The ROI region of the sundry target obtained in step 5 is extracted, and a feature matching algorithm is used to calculate the similarity between the sundry ROI region obtained by semantic segmentation and post-processing in the camera 1 and the image of the camera 2 in a specific time period. The region with the highest similarity is taken as the starting position of the target tracking of the camera 2.

[0032] Optionally, in the implementation process of steps 7 and 8:

[0033] The target tracking algorithm includes: taking the ROI region of the sundry target obtained in step 5 as a feature matching template, cyclically shifting in the horizontal and vertical directions in the image of the camera 2 to construct a cyclic matrix sample set, training a ridge regression function by using the sample set, finding the region with the maximum regression function response value around the preselected region of the sundry target in the next frame image of the camera 2, constantly updating the image, calculating the cyclic regression function response position, and outputting the current position of the sundry target in real time.

[0034] Optionally, step 9 of fusing semantic segmentation and target tracking information is specifically as follows:

[0035] Obtain the positive rectangular frame of the sundry target by the semantic segmentation detection model in step 5, and perform feature matching on the image in the camera 2 by taking the positive rectangular frame as a feature matching module to obtain the real-time position of the target sundry rectangular frame in the camera 2;

[0036] Obtain the skeleton centroid position of the sundry target by the post-processing operation in step 5, and map the relative position in the positive rectangular frame to the sundry target rectangular frame obtained by feature matching on the image of the camera 2 to obtain the real-time position of the grabbing midpoint in the camera 2, which is used for controlling the subsequent sorting equipment to grab.

[0037] The technical scheme has the following advantages:

[0038] 1. The present application monitors sundries on the vibrating screen through computer vision, and then uses a mechanical hand to complete the picking of the sundries. Compared with other sundry removal methods, this method does not need to occupy human resources, has higher automation, and can complete accurate monitoring of various types of sundries.

[0039] 2. The present application monitors the sundries in the coal on the screen in an open scene by using two industrial cameras. The front-end camera obtains sundry target shape information by using a semantic segmentation algorithm, and the rear-end camera obtains real-time position information of the sundries by using a target tracking algorithm. Compared with a single camera, this method is more efficient. DETAILED DESCRIPTION

[0040] In order to more clearly illustrate the technical scheme of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0041] Figure 1 The present application provides a sundry monitoring method flowchart on the screen;

[0042] Figure 2 The present application provides an image data enhancement diagram;

[0043] Figure 3 The present application provides a camera installation diagram;

[0044] Figure 4 The present application provides a general flowchart. DETAILED DESCRIPTION

[0045] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0046] The embodiments of the present application will be further described in detail below with reference to the accompanying drawings, as shown in the drawings Figure 1 Fig. 1 is a flowchart of a coal gangue identification method provided by an embodiment of the present application, and the method comprises the following steps:

[0047] Step 1: An image between a screen surface feeding end and an exciting beam is collected by a No. 1 camera (i.e., camera 1) installed on the front side of a screening machine, and a training sample set of screen sundries is established.

[0048] Specifically, an open image collection device is built at the front end of the screening machine, an industrial camera is perpendicular to the screen surface in terms of shooting angle, and an image collection area is between the first beam and the exciting beam, so as to obtain position and shape information of sundries at the front side of the screening machine.

[0049] Step 2: The training sample set in Step 1 is subjected to filtering and noise reduction processing, illumination uniformization processing, and screen motion blur elimination processing.

[0050] Since the image collection environment is natural light, the screen surface image obtained will have an uneven illumination phenomenon, in addition, the vibration of the screen surface will cause a certain degree of motion blur, which will affect the identification effect of the subsequent semantic segmentation network, therefore, illumination uniformization processing and screen motion blur elimination are required.

[0051] The process of filtering and noise reduction processing is specifically as follows:

[0052] A spatial domain filter is used to reduce image noise.

[0053] Here, the process of illumination uniformization processing is specifically as follows:

[0054] Based on a high-scale Gaussian function, an illumination component in the image is extracted, a correction function is constructed according to the distribution characteristics of the illumination, the brightness of an area with too low illumination is enhanced, and the brightness of an area with too strong illumination is reduced. Gamma correction and histogram equalization are used for adaptive correction of the uneven illumination image.

[0055] The process of screen motion blur elimination is specifically as follows:

[0056] A DeblurGAN generative adversarial network is trained to process the motion blurred image.

[0057] Step 3: According to different illumination conditions, a generative adversarial network is used to expand the training sample set, and a processed training sample set is obtained.

[0058] Here, as Figure 2 The process schematic diagram for expanding the training sample set by using the generative adversarial network provided by the embodiment of the present application is shown. Since the training of the semantic segmentation network requires a large amount of data set, and the acquisition of the screen debris image requires a large amount of human cost, it is necessary to expand the training sample set.

[0059] For example, referring to Figure 2 The images are collected under natural light conditions. Due to the difference in light conditions at different time periods, the collected images are different in brightness and contrast. Assuming that there are three kinds of image data of A, B and C light conditions, in order to achieve the purpose of image expansion, the A class images are respectively style transferred to the B and C classes to obtain the expanded A B and Ac images, the expansion strategy of the B and C class images is similar. Specifically, the CycleGAN network can be used to perform style transfer from each class to other classes to expand the training sample set.

[0060] Step 4, training the training sample set processed in steps 2 and 3 based on the deep semantic segmentation network to obtain a semantic segmentation detection model, wherein the current processed training sample set is input into the semantic segmentation detection model to obtain a semantic segmentation detection result.

[0061] In this step, the processed training sample set is specifically input, and the deep semantic segmentation network includes but is not limited to UNet, Segmenter, TransUNet semantic segmentation network. The encoder in the deep semantic segmentation network extracts features from the image, and the decoder restores the features to obtain a debris segmentation region; the post-processing operation is to use a skeleton extraction algorithm to refine the connected region of the target mask in the semantic segmentation region into a skeleton of one pixel width, and calculate the centroid position of the skeleton, which is the grabbing point position.

[0062] In particular, the deep semantic segmentation network uses processed training samples as input, including different network structures such as UNet, Segmenter and TransUNet. In the deep semantic segmentation network, the encoder is used to extract the features of the image, and the decoder is used to restore these features to obtain the clutter segmentation region. The processed training sample set is input, which may include images and corresponding segmentation labels. The encoder part of the network performs feature extraction on the input image. The encoder is usually composed of multiple convolutional layers and pooling layers to gradually reduce the spatial size of the feature map while increasing the abstraction level of the features. After the extracted feature map passes through the encoder, it enters the decoder part. The decoder is usually composed of convolutional layers and up-sampling layers to restore the spatial size of the feature map to the original image size. The output of the decoder is the restored feature map, which contains the semantic information of each pixel in the original image. According to the semantic information of the feature map, pixel-level classification is performed to assign each pixel in the image to a specific clutter category, thereby obtaining the clutter segmentation region. These steps combine the functions of the encoder and the decoder to achieve semantic segmentation of the image, i.e., to segment different objects or regions in the image. Different network structures and algorithms can be improved and optimized on this basic process to improve the accuracy and effectiveness of segmentation.

[0063] In particular, in the semantic segmentation region, the connected region of the target mask is made more refined using a skeleton extraction algorithm, so that it becomes a skeleton of one pixel width. Then the centroid position of the skeleton is calculated, which can be used as the position of the grabbing point. (It can also be understood that in the semantic segmentation region, the connected region of the target mask is made more refined by the skeleton extraction algorithm, and only a skeleton of one pixel width is retained. Then the centroid position of the skeleton is calculated, which is the position of the grabbing point.) The input is an image and its corresponding semantic segmentation result, in which the target region is marked as a mask. The target region in the mask is operated by the skeleton extraction algorithm to make it more refined, becoming a skeleton of one pixel width. The centroid position of the skeleton is calculated, which is the average position of all pixels on the skeleton. The grabbing point position is the centroid position of the skeleton, which can be used to control the position of the robotic arm or other grabbing equipment to achieve the grabbing operation of the target.

[0064] Step 5, draw the outer rectangle of the clutter in the original image according to the semantic segmentation detection result, and cut off the detected clutter target.

[0065] Find the connected region in the segmentation result, draw the outer rectangle of the connected region, cut off the clutter target according to the position of the outer rectangle, and obtain the ROI region of the target clutter as the feature matching template provided by the camera 2.

[0066] Specifically, according to the result of semantic segmentation detection, the bounding rectangle of the foreign matter is drawn on the original image, and then the image is cropped from the detected foreign matter target. Next, the connected regions are found in the semantic segmentation result, and the bounding rectangle is drawn on these regions. According to the position of the bounding rectangle, the ROI (region of interest) of the target foreign matter is cropped from the original image again. This ROI will serve as a template for feature matching provided by camera 2.

[0067] Specifically, semantic segmentation detection is performed to obtain the class information of each pixel. According to the detection result, the bounding rectangle of the foreign matter is drawn on the original image. The image is cropped from the detected foreign matter target to obtain the initial foreign matter ROI region. The connected regions are found in the semantic segmentation result. For each connected region, a bounding rectangle is drawn. According to the position of the bounding rectangle, the ROI region of the foreign matter is cropped from the original image. This ROI region is used as a template for feature matching provided by camera 2 for subsequent feature matching tasks.

[0068] Step 6, acquire the screen rear-end image through camera 2, and perform perspective transformation to obtain a top-down view image.

[0069] Here, the installation positions of camera 1 (camera No. 1) and camera 2 (camera No. 2) are as follows Figure 3 Since the coal flow on the screen surface of the vibrating screen is in irregular motion, the position and shape information of the foreign matter acquired at the front end of the screen cannot accurately guide the operation of the actuator at the rear end of the screen, and the semantic segmentation detection model has poor real-time performance, so the rear-end foreign matter image can be acquired by camera 2 to realize real-time tracking of the foreign matter in combination with the foreign matter information obtained by semantic segmentation. At the same time, the camera installation space at the rear end of the vibrating screen is limited, and only a diagonal top-down view can be acquired, so the view angle needs to be converted for matching between the two cameras.

[0070] In the field of computer vision, both target detection and semantic segmentation can complete the identification of foreign matter in coal. The vibration of the vibrating screen causes the foreign matter to be in irregular motion, and semantic segmentation takes a long time, so it cannot acquire real-time position and shape information of the foreign matter, and therefore a second industrial camera is needed to use a real-time target tracking algorithm to complete real-time monitoring of the foreign matter on the screen.

[0071] In this step, the perspective transformation process is as follows:

[0072] A trapezoidal region of the image captured by camera 2 is selected, and the perspective transformation formula is used to convert it to a top-down view.

[0073] Step 7, extract the features of the foreign matter target detected by the semantic segmentation detection model, and perform feature matching with the image obtained in step 6 to detect the position of the foreign matter in camera 2.

[0074] Specifically, the image is processed using a semantic segmentation detection model to identify the location of the litter target. Features are extracted from the identified litter target, for example, a convolutional neural network can be used to extract features. Features are also extracted from the image obtained in step 6 using the same method. The features extracted in step 3 are matched with the features extracted in step 2, which can use feature descriptors and matching algorithms such as SIFT or ORB. The litter target corresponding to the feature pair with the highest matching score is the litter location in camera 2.

[0075] The ROI region of the litter target obtained in step 5 is extracted, and the similarity between the litter ROI region obtained by semantic segmentation and post-processing in camera 1 and the image of camera 2 is calculated using a feature matching algorithm within a certain time period. The region with the highest similarity is selected as the starting position of the target tracking in camera 2.

[0076] Specifically, the ROI region of the target is extracted from the litter image, which may involve image segmentation algorithms and post-processing methods. Within a certain time period (e.g. 1 hour or 5 hours), the image of camera 2 is obtained. The similarity between the litter ROI region obtained in camera 1 and each region in the image of camera 2 is calculated using a feature matching algorithm (such as SIFT, SURF, etc.). For each region in the image of camera 2, the similarity with the ROI region of camera 1 is calculated. The region with the highest similarity is selected as the starting position of the target tracking in camera 2.

[0077] Step 8, the litter target detected by feature matching is used as the initial frame for litter target tracking.

[0078] The target tracking algorithm includes: taking the ROI region of the litter target obtained in step 5 as the feature matching template, and shifting it in the horizontal and vertical directions to construct a cyclic matrix sample set. The sample set is used to train a ridge regression function to find the region with the maximum regression function response value in the preselected area around the litter target in the next frame of the image of camera 2. The image is constantly updated, and the cyclic regression function response position is calculated to output the current position of the litter target in real time.

[0079] In this step, the litter target tracking process is as follows:

[0080] The target region in the first frame image is extracted and shifted in the x and y axes to construct a cyclic matrix sample set. The sample set is used to train a ridge regression function to find the region with the maximum regression function response value in the preselected area around the litter target in the next frame of the image, which is the real-time position of the litter. Finally, the shape information of the litter obtained by semantic segmentation is fused with the position information of the target tracking to complete the real-time positioning and tracking of the on-screen litter.

[0081] Step 9, the process of fusing semantic segmentation and target tracking information in step 9 is as follows:

[0082] The positive rectangular frame of the sundry target is obtained by the semantic segmentation detection model in step 5, and the positive rectangular frame is used as a feature matching module to match the image in camera 2 to obtain the real-time position of the target sundry rectangular frame in camera 2;

[0083] Specifically, the semantic segmentation detection model is used to process the image and find the positive rectangular frame of the sundry target. These positive rectangular frames are used as the input of the feature matching module. The image in camera 2 is obtained and the feature matching module is used to match the image in camera 2. Through feature matching, the target sundry rectangular frame corresponding to the positive rectangular frame in step 1 is found. The real-time position information of the target sundry rectangular frame in camera 2 is obtained. This can be achieved through the results of feature matching and the coordinate system and positioning algorithm of camera 2.

[0084] The skeleton centroid position of the sundry target is obtained by the post-processing operation in step 5, and the relative position of the skeleton centroid position in a positive rectangular frame is mapped to the sundry target rectangular frame obtained by feature matching in the image of camera 2 to obtain the real-time position of the grabbing midpoint in camera 2 for controlling the subsequent sorting equipment to grab.

[0085] Specifically, through the post-processing operation in step 5, the skeleton centroid position of the sundry target can be obtained. Then, the relative position of the skeleton centroid position in a positive rectangular frame is mapped to the image of camera 2, and the position of the sundry target rectangular frame can be obtained through feature matching. Finally, the position information of the center of the sundry target in camera 2 can be determined in real time, and the position information is used to control the subsequent sorting equipment to grab.

[0086] Specifically, the post-processing operation in step 5 is performed to obtain the skeleton centroid position of the sundry target. The relative position of the skeleton centroid position in a positive rectangular frame is determined. The relative position is mapped to the image of camera 2. The position of the sundry target rectangular frame in the image of camera 2 is determined using the feature matching method. According to the position of the sundry target rectangular frame, the real-time position of the grabbing midpoint in camera 2 is calculated. This real-time position information is used to control the subsequent sorting equipment to accurately grab the sundry target.

[0087] The skeleton centroid position refers to the position of the centroid of an object or system within its skeleton or structure. The skeleton refers to the frame or support structure of the object, and the centroid is the center of mass of the object or system, i.e. the position of the center of gravity when the overall mass of the object or system is evenly distributed. The determination of the skeleton centroid position is of great significance in studying the balance, motion and mechanical properties of the object or system. In practical applications, the calculation of the skeleton centroid position can be achieved by measuring the geometric shape and mass distribution of the object or system.

[0088] Specifically, asFigure 4 As shown in the figure, the overall flow of the scheme of the present application is as follows:

[0089] The front-end image of the screening machine is acquired by the camera 1, and a training sample set of the sundries on the screen is established;

[0090] The training sample set is subjected to filtering and denoising processing, uneven illumination processing, and motion blur elimination processing;

[0091] The image data is divided into multiple categories according to the illumination conditions, and CycleGAN network is used to perform style transfer from each category to other categories to expand the training sample set;

[0092] The processed training sample set is trained based on a deep semantic segmentation network to obtain a semantic segmentation detection model;

[0093] The real-time image of the camera 1 is transmitted into the semantic segmentation detection model, and the output of the semantic segmentation detection model is the sundry shape information (i.e. the detected sundry target);

[0094] The rear-end image of the screening machine is acquired by the camera 2, and the perspective conversion is performed;

[0095] According to the obtained sundry target features, the similarity of each position in the specific detection area of the obtained image in a specific time period is calculated, and the area with the highest similarity is found as the starting position of target tracking;

[0096] The target tracking algorithm is used to output the current position of the sundry target in real time;

[0097] The sundry mask shape information obtained by semantic segmentation and the sundry position information obtained by target tracking are fused to complete real-time positioning and tracking of the sundries on the screen, and guide the operation of the subsequent grabbing device.

[0098] The embodiment schemes and network structure parameters listed in the specification are only to help the reader understand the principles of the present application and illustrate the advantages of the present application, and do not represent the most optimized case. Those skilled in the art can obtain better results by optimizing the parameters of the present application. Those skilled in the art should understand that various possible equivalent replacements and changes made on the basis of the present application all belong to the scope of protection of the present application.

Claims

1. A method for real-time monitoring of oversize materials based on visual tracking, characterized in that, The method comprises: Step 1, acquiring a front-end image of a screening machine through a camera 1 to establish a training sample set of the screening machine; Step 2, performing filtering and noise reduction processing, uneven illumination processing, and motion blur elimination processing on the training sample set; Step 3, according to different illumination conditions, expanding the training sample set by using a generative adversarial network to obtain a processed training sample set; Step 4, training based on a deep semantic segmentation network for the processed training sample set in step 3 to obtain a semantic segmentation detection model, wherein the current processed training sample set is input into the semantic segmentation detection model to obtain a semantic segmentation detection result; Step 5, drawing a minimum bounding rectangle on an original image according to the semantic segmentation detection result to cut off the detected debris target; Step 6, acquiring a rear-end image of the screening machine through a camera 2 and performing perspective conversion; Step 7, extracting the debris target features detected by the semantic segmentation detection model and performing feature matching with the image in step 6 to detect the debris position in the camera 2; Step 8, taking the debris target detected by feature matching as an initial frame to perform target tracking; Step 9, fusing semantic segmentation and target tracking information to complete debris detection.

2. The on-screen debris real-time monitoring method of claim 1, wherein, In step 1, the process of acquiring a front-end image of a screening machine through a camera 1 is specifically as follows: An open image acquisition device is built at the front end of the screening machine, an industrial camera is vertically arranged with the screening surface, and an image acquisition area is between a first crossbeam and a vibration beam to obtain position and shape information of the debris at the front side of the screening machine.

3. The on-screen debris real-time monitoring method of claim 1, wherein, In step 2, the process of performing filtering and noise reduction processing, uneven illumination processing, and motion blur elimination processing on the training sample set is specifically as follows: A spatial domain filter is used to reduce image noise, gamma correction and histogram equalization are used for adaptive correction of uneven illumination images, and a blurred motion kernel is used to reduce image motion blur.

4. The on-screen debris real-time monitoring method of claim 1, wherein, In step 3, the process of expanding the training sample set by using a generative adversarial network according to different illumination conditions is specifically as follows: Image data is classified into multiple categories according to illumination conditions, CycleGAN network is used to perform style transfer from each category to other categories to expand the training sample set.

5. The on-screen debris real-time monitoring method of claim 1, wherein, In step 4, the process of training based on a deep semantic segmentation network using the processed training sample set and performing post-processing operations is specifically as follows: The processed training sample set is input, the deep semantic segmentation network includes but is not limited to UNet, Segmenter, and TransUNet semantic segmentation network, an encoder in the deep semantic segmentation network extracts features of an image, a decoder recovers the features to obtain a debris segmentation area; The post-processing operation is to use a skeleton extraction algorithm to refine a connected region of a target mask in the semantic segmentation area into a skeleton with a pixel width, and calculate a skeleton centroid position as a grabbing point position.

6. The on-screen debris real-time monitoring method of claim 1, wherein, In step 5, the process of cutting off the detected debris target is specifically as follows: A connected region is found in a segmentation result, a minimum bounding rectangle of the connected region is drawn, and a debris target is cut off according to the position of the minimum bounding rectangle to obtain an ROI area of the target debris as a feature matching template for the camera 2.

7. The on-screen debris real-time monitoring method of claim 1, wherein, In step 6, the camera 2 acquires the screen machine rear-end image, and the perspective conversion process is specifically as follows: The camera 2 is installed at the rear side of the vibration beam, and the shooting perspective is oblique top view, and the image acquisition area is the area below and behind the vibration beam; a trapezoidal area is selected in the image acquired by the camera 2 as the original image of perspective transformation, and the oblique top view is transformed into a top view by using the perspective transformation formula.

8. The on-screen debris real-time monitoring method of claim 1, wherein, In step 7, the feature matching process is specifically as follows: The ROI region of the sundry target obtained in step 5 is extracted, and the similarity between the sundry ROI region obtained by semantic segmentation and post-processing in the camera 1 and the image of the camera 2 is calculated by using a feature matching algorithm in a specific time period, and the region with the highest similarity is taken as the starting position of the target tracking of the camera 2.

9. The on-screen debris real-time monitoring method of claim 1, wherein, In the implementation process of steps 7 and 8: The target tracking algorithm includes: taking the ROI region of the sundry target obtained in step 5 as a feature matching template, and shifting in the horizontal and vertical directions in the image of the camera 2 to construct a cyclic matrix sample set, and training a ridge regression function by using the sample set to find the region with the maximum regression function response value around the sundry target in the next frame image of the camera 2, and constantly updating the image to calculate the cyclic regression function response position and output the current position of the sundry target in real time.

10. The on-screen debris real-time monitoring method of claim 1, wherein, Step 9 fusion semantic segmentation and target tracking information process is specifically as follows: The rectangular frame of the sundry target obtained by the semantic segmentation detection model in step 5 is taken as a feature matching module to perform feature matching with the image in the camera 2 to obtain the real-time position of the sundry target rectangular frame in the camera 2; The skeleton centroid position of the sundry target obtained by the post-processing operation in step 5 is mapped to the sundry target rectangular frame obtained by feature matching in the image of the camera 2 in the relative position in the rectangular frame to obtain the real-time position of the grabbing midpoint in the camera 2, which is used to control the subsequent sorting equipment to grab.

Images