Mine belt conveyor image structured modeling method and device

Abstract

The application discloses a mine belt conveyor image structured modeling method and device, the method comprises the following steps: acquiring a mine belt conveyor image, and segmenting the image into multiple regions based on semantic segmentation; detecting and positioning the rollers and roller supports of the belt edge in each region through a target bounding box detection according to a target detection method; detecting whether there is a foreign matter in each region according to the positioned rollers and roller supports and obtaining a detection result; converting the detection result into a vector, and forming a belt conveyor image structured modeling. The method solves the problem that the prior art cannot realize more accurate and more detailed image content monitoring and analysis in a fixed scene such as a mine conveying belt.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a mine belt conveyor image structured modeling method and device. BACKGROUND

[0002] Deep neural network models are widely used in image and video segmentation, target detection and recognition tasks, and are widely used in various video monitoring systems. Compared with the previous simple target detection and recognition tasks, with the development and improvement of algorithm technology and deployment of computing resources, current image analysis and video monitoring technology is developing towards more detailed, more in-depth, and human visual characteristics, achieving more intelligent analysis and understanding.

[0003] For the monitoring of mine transmission belts and other fixed scenes, the scene content is relatively stable, and users often require more accurate and detailed image content analysis to form semantic understanding. For example, when facing a scene where the transmission belt system is running without abnormal conditions or with few abnormal conditions, it is necessary to automatically detect the belt edge and the roller, and verify whether the two spatial positions meet the requirements, so as to detect and monitor the equipment state of the belt system. Through belt edge detection, the edge position of the belt is quickly and accurately located, so as to judge the running state of the belt and whether there is abnormal vibration or foreign matter blocking in the conveyor belt. Through roller detection, the position and state of the roller in the belt system can be monitored to further judge whether the equipment is running normally. Then, by verifying whether the two spatial positions meet the requirements, problems caused by abnormal belt edge or roller position can be excluded, and abnormal conditions can be further analyzed, such as judging whether there is foreign matter blocking in the belt track or the equipment is running abnormally. However, traditional neural network models cannot achieve spatial position relationship modeling closer to human visual understanding, so they cannot achieve the above-mentioned fine monitoring. SUMMARY

[0004] The present application provides a mine belt conveyor image structured modeling method and device to solve the problem that in the prior art, in the fixed scene of mine transmission belt, it is impossible to achieve more accurate and detailed image content monitoring and analysis.

[0005] A mine belt conveyor image structured modeling method according to the present application comprises:

[0006] Obtaining a mine belt conveyor image, and segmenting the image into multiple regions based on semantic segmentation;

[0007] According to a target detection method, detecting and positioning the roller and roller support of the belt edge in each region through a target bounding box;

[0008] According to the positioned roller and roller support, detecting whether there is foreign matter in each region and obtaining a detection result;

[0009] The detection result is converted into a vector, and a belt conveyor image structure is formed.

[0010] Optionally, the image is segmented into a plurality of regions based on semantic segmentation, including:

[0011] The semantic segmentation line network unit obtains semantic segmentation lines from the image according to a deep Hough transform;

[0012] The vanishing point of the image is determined according to intersection points of two or more parallel semantic segmentation lines;

[0013] The belt edge in the image is determined according to the vanishing point;

[0014] The image is segmented according to the belt edge.

[0015] Optionally, the method further includes:

[0016] The semantic segmentation line network unit outputs a belt confidence score and a distance of the belt from the vanishing point;

[0017] An overall score is obtained according to the belt confidence score and the distance of the belt from the vanishing point;

[0018] The belt edge with the maximum overall score is taken as a semantic segmentation line.

[0019] Optionally, the method further includes:

[0020] The semantic segmentation line is corrected by a distortion correction sub-network.

[0021] Optionally, the method includes:

[0022] The distortion correction sub-network learns a parameter matrix ρ = (k1, k2, k3, p1, p2, f x ,f y ,c x ,c y ); wherein k1, k2, k3, are radial distortion correction parameters; p1 and p2 are tangential distortion parameters; f x ,f y ,c x ,c y are intrinsic matrix parameters;

[0023] A corrected image is calculated according to the parameter matrix to correct the semantic segmentation line.

[0024] Optionally, before detecting and locating the rollers and roller supports of the belt edge in each of the regions by a target bounding box, the method includes:

[0025] obtaining a plurality of initial bounding boxes;

[0026] removing interference bounding boxes from the initial bounding boxes according to a set confidence threshold to obtain target bounding boxes.

[0027] Optionally, determining a position of the foreign matter based on the detection result;

[0028] judging whether the position meets a preset range from a semantic segmentation line;

[0029] If yes, it is determined that the foreign matter exists on the belt.

[0030] An image structured modeling device for a mine belt conveyor, the device comprising:

[0031] a semantic segmentation line network unit, configured to obtain a mine belt conveyor image and segment the image into a plurality of regions based on semantic segmentation;

[0032] a target detection unit, configured to detect and locate a roller and a roller support of a belt edge in each region through a target bounding box according to a target detection method;

[0033] a foreign matter detection unit, configured to detect whether a foreign matter exists in each region according to the located roller and the roller support and obtain a detection result;

[0034] a modeling unit, configured to convert the detection result into a vector and form a belt conveyor image structured modeling.

[0035] An apparatus comprising a processor and a memory, wherein the processor implements the method of any one of the above when executing a computer program stored in the memory.

[0036] A computer readable storage medium in the present application, the computer readable storage medium stores one or more programs, the one or more programs can be executed by one or more processors to implement the steps of the method of any one of the above.

[0037] The present application performs efficient semantic segmentation line detection on images with relatively obvious longitudinal spatial rules (such as belt edges), and divides obvious different regions. The edge information is not concerned, but the segmentation line with clear semantics is concerned. For example, the segmentation line between the belt and the roller can be located, and the influence of the edge between the belt and the roller being a certain rule non-straight line is not affected, and whether the spatial position between them meets the expected rule is verified, so that the spatial position relationship modeling closer to human visual understanding is realized. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1is a flowchart of a mine belt conveyor image structured modeling method in an embodiment of the present application;

[0039] Figure 2 is a representation method of a target bounding box in an embodiment of the present application;

[0040] Figure 3 is a structural diagram of a mine belt conveyor image structured modeling device in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The present application will be further described below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the present application, but not to limit the present application. In addition, it should be noted that only the parts related to the present application are shown in the drawings for the convenience of description, rather than all the structures.

[0042] It should be understood that the size of the sequence number of each process described above in various embodiments herein does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments herein.

[0043] The specific embodiments of the present application provide a mine belt conveyor image structured modeling method, as shown in Figure 1 The method comprises the following steps:

[0044] In step 100, a mine belt conveyor image is obtained, and the image is segmented into multiple regions based on semantic segmentation. Specifically, the mine belt conveyor image is subjected to semantic segmentation, and the image is segmented into different regions to accurately identify different objects in the image.

[0045] In step 200, the rollers and roller supports of the belt edge in each region are detected and positioned by a target detection method. In step 200, the target detection technology is used to analyze each region in the image, and the rollers and roller supports of the belt edge are identified first, which facilitates the subsequent accurate detection of foreign matter on the belt.

[0046] In step 300, whether there is foreign matter in each region is detected and the detection result is obtained according to the positioned rollers and roller supports. Specifically, the YOLOv5 model is selected for target detection, which has high real-time efficiency and high accuracy, and can effectively identify foreign matter in the image.

[0047] In step 400, the detection result is converted into a vector, and a belt conveyor image structured modeling is formed.

[0048] The method provided by the embodiment of the application is suitable for a scenario where the belt conveying system is in a normal state or a less abnormal state, and can automatically detect the belt edge and the roller in the belt conveying system, and verify whether the two spatial positions meet the requirements, so as to detect and monitor the device state of the belt conveying system. Through the belt edge detection, the edge position of the belt can be quickly and accurately located, so as to judge the running state of the belt, and whether the conveying belt is in an abnormal vibration state or an abnormal foreign matter blocking state. Through the roller detection, the position and state of the roller can be monitored, and whether the device is normally running can be further judged. Then, by verifying whether the two spatial positions meet the requirements, problems caused by abnormal belt edges or rollers can be excluded, and the abnormal conditions can be further analyzed, for example, whether the belt track is blocked by foreign matters or the device is in an abnormal running state, and finally, a spatial position relationship modeling closer to human visual understanding is formed, an efficient and convenient structured scheme is formed, and quick and accurate content understanding and monitoring of the belt conveying system are realized.

[0049] Preferably, when the foreign matter is detected, the system triggers a corresponding alarm mechanism or records abnormal information, so that the abnormal condition can be handled in time, and the safe operation of the belt conveying system is ensured.

[0050] The image structured modeling method of the mine belt conveyor provided by the embodiment of the application preferably comprises the following steps of:

[0051] The semantic segmentation line network unit acquires a semantic segmentation line from the image according to the deep Hough transform.

[0052] Specifically, the semantic segmentation line network unit first performs a deep Hough transform on the image, converts from a Cartesian coordinate system to a polar coordinate system, and in the polar coordinate conversion process, takes the image center as the origin, the polar angle range is 0-2pi, and a fixed sampling rate is adopted The maximum polar radius is different for different feature layers of different scales in the image feature pyramid, and therefore a group of different polar radii is designed Different feature layers are mapped to polar coordinate space feature layers with the same size, so as to be quickly combined together for the next convolution calculation.

[0053] Before the deep Hough transform is performed, an edge activation function network EATF is performed. Because traditional line detection based on Hough transform uses a binary edge image as input, it depends on the result of edge detection, and it is difficult to obtain an ideal result for a complex real scene image. However, the main position of the semantic segmentation line is distributed on the boundary of different regions, and the edge information of the image plays an important role in semantic line segmentation. In order to utilize the role of the edge information in semantic segmentation line positioning, the edge activation function network introduces a channel self-attention mechanism to weight the convolution, implicitly guides the learning of the convolution that is conducive to semantic segmentation line, and is implemented by referring to the self-attention mechanism network mode SENet; in addition, a Tanh activation function is added, and the purpose is to realize the role of weakening sharp edges and reduce the influence of sharp short lines, so that the network pays more attention to large-scale semantic segmentation lines with clear semantics. Generally, SENet uses a Sigmoid function as the last activation function, and therefore, in the channel attention mechanism, a Relu function is used, so as to avoid the problem of small gradient in the network training process and accelerate the network convergence.

[0054] For the semantic segmentation line with clear semantics, the target loss function is designed as L = L line + L cls In the polar coordinate space, the line projection is a point, and in the training, the line projection is marked as a point. A labeled point is smoothed by a Gaussian kernel to obtain a distribution in a neighborhood range, and L line is the cross-entropy loss function of the prediction result and the smoothed labeled graph. L cls is the class loss corresponding to the line, and a multi-class cross-entropy loss (softmax loss) can be used. After the target loss function, the inverse transformation of the deep Hough transform is performed to convert from the polar coordinate system to the Cartesian coordinate system of the image, and finally the actual semantic segmentation line is output.

[0055] The vanishing point of the image is determined according to the intersection points of two or more parallel semantic segmentation lines. Specifically, all semantic segmentation lines in the same direction are truncated at the vanishing point, forming a segmentation of the main region in the image.

[0056] The belt edge in the image is determined according to the vanishing point; after the vanishing point is truncated, a detection box is formed from a certain point of the belt in the distance, and the box is helpful for distinguishing the belt edge. In a specific embodiment, the intersection point of the belt edge is not unique, because various errors can cause the intersection to occur in a region. In order to obtain the estimated vanishing point, a threshold range is set, and multiple intersection points are averaged. At the same time, the belt that may fail to be detected can also be excluded.

[0057] The image is segmented according to the belt edge. Specifically, in the present application, the belt edge is directly used as the semantic segmentation line of the image.

[0058] The mine belt conveyor image structure modeling method according to the embodiment of the present application, preferably, the method further comprises:

[0059] The semantic segmentation line network unit outputs a belt credibility score and a distance of the belt from the vanishing point;

[0060] An overall score is obtained according to the belt credibility score and the distance of the belt from the vanishing point;

[0061] The belt edge with the maximum overall score is taken as the semantic segmentation line.

[0062] In the actual main area boundary semantic segmentation line detection, the result may not be perfect. For a certain semantic segmentation, two coal belts with threshold meeting the requirements may be detected. In order to solve this problem, an evaluation method is designed, which combines the belt credibility score (denoted as S c ) and the distance of the belt from the vanishing point (denoted as S d ) output by the semantic segmentation line network unit. The belt credibility score is the result of the normalized processing of the semantic segmentation line network output, and the value range is between 0 and 1. The distance of the belt from the vanishing point is calculated by the distance from the point to the line of the vector diagram, and the unit is pixel. The overall score of a belt is S=S c +1 / S d , and the greater the S value, the more credible the corresponding belt. Therefore, the belt is selected as the final segmentation line. Finally, a result image of the vanishing point and the fixed strip segmentation line is obtained.

[0063] In the preferred embodiment, for the fusion analysis of the semantic segmentation line detection result and the fixed target detection result, there may be two cases:

[0064] When the distribution is reasonable, for example, the belt (line object) overlaps with the roller (surface object), the intersection point is recorded, and the intersection range is marked out, so as to facilitate subsequent analysis and modeling. In order to judge whether the distribution is reasonable, the following steps can be taken:

[0065] The center position of the roller is determined. The geometric center of the contour is calculated by detecting the contour of the roller in the image.

[0066] The least squares method is commonly used to fit multiple roller centers to obtain the straight line they lie on. This method treats roller center coordinates as data points, calculates the most suitable straight line model, and minimizes the sum of squared distances from data points to the fitted straight line. This straight line represents the overall position and direction of the roller, thereby evaluating the rationality of the belt and the roller. The edge of the belt is detected to extract its position information. By calculating the angle between the belt edge and the fitted straight line, it is determined whether they are essentially parallel. If the angle is small, it can be considered that the distribution is reasonable.

[0067] To more accurately evaluate the rationality, the rollers on both sides can be fitted separately and their parallelism with the belt edge can be checked. If the angles between the fitted straight lines of the rollers on both sides and the belt edge are similar, it can be confirmed that the distribution is reasonable.

[0068] The position and intersection range of the intersection point are recorded and marked, which facilitates subsequent analysis and modeling work. This information is very important for understanding the internal structure and characteristics of the coal belt.

[0069] When the distribution is not reasonable, there can be two cases:

[0070] It may be that the semantic segmentation line or target detection is wrong. In this case, the semantic segmentation line is backtracked. If the currently selected semantic segmentation line intersects with the roller, the alternative semantic segmentation line is re-evaluated. If there is another alternative semantic segmentation line, it will be replaced and the next step will be re-entered; if there is no other alternative semantic segmentation line, the currently selected semantic segmentation line will be fine-tuned to pass through the vanishing point, and the distribution will be re-evaluated.

[0071] If there is no unreasonable intersection, it is considered that the modeling is complete, and the next operation is entered. If there is an unreasonable intersection (for example, the semantic segmentation line intersects with the roller), it may indicate that the belt has deviated, or the content captured in the modeling video has changed from the set situation and is different from the content expected to be processed, both of which need to be alarmed to prompt manual inspection.

[0072] The mine belt conveyor image structure modeling method described in the specific embodiments of the present application, preferably, the method further comprises:

[0073] The semantic segmentation line is corrected by a distortion correction sub-network.

[0074] Specifically, since the camera generally exists distortion, and each camera deformation is different, the monitoring camera with low distortion degree does not need to be corrected. However, slight distortion may cause the bending change of the important semantic segmentation line in the image, resulting in that the main semantic segmentation line with long length cannot be detected. The front lens of the camera can make the camera obtain better imaging effect, however, the addition of the lens is the main reason causing the image distortion. One is the influence of the lens shape on the light propagation. Two is that the lens and the imaging plane are not completely parallel in the mechanical assembly process, causing the position change of the light projection to the imaging plane. The distortion caused by the lens shape is called radial distortion. The closer to the image edge, the more obvious this phenomenon is. Since the lens is generally center-symmetric, the distortion is radially symmetric, mainly including barrel distortion and pincushion distortion. The distortion caused by the non-parallelism of the lens and the imaging plane is called tangential distortion. Therefore, various distortions need to be corrected.

[0075] The mine belt conveyor image structured modeling method provided in the embodiment of the application, preferably, the method comprises:

[0076] The distortion correction sub-network learns the parameter matrix ρ=(k1, k2, k3, p1, p2, f x ,f y ,c x ,c y ); wherein k1, k2, and k3 are radial distortion correction parameters; p1 and p2 are tangential distortion parameters; f x ,f y ,c x ,c y is a parameter of an intrinsic matrix.

[0077] The corrected image is calculated according to the parameter matrix to correct the semantic segmentation line.

[0078] Specifically, the radial distortion can be described by a polynomial for the coordinate change before and after the distortion, and can be corrected by a quadratic and high-order polynomial function related to the distance from the center. On the normalized imaging plane, [x, y] is the coordinate of the uncorrected point, written in polar coordinate form as [r, θ], r represents the distance of the point from the coordinate origin, θ represents the angle with the horizontal axis, and [x dis ,y dis ] represents the coordinate of the distorted point, and has:

[0079]

[0080] For a general camera, two radial distortion correction parameters k1 and k2 can be used to correct the radial distortion. For a camera with large distortion, such as a fisheye lens, k1, k2 and k3 can be used to correct the distortion.

[0081] Tangential distortion, generally uses the following two formulas to correct, p1 and p2 are tangential distortion parameters:

[0082]

[0083] Synthesis formula 1 and formula 2, there is

[0084]

[0085] The point on the normalized imaging plane is projected onto the pixel plane by the intrinsic matrix of the camera, which is the actual pixel coordinate in the image, and the relationship is as follows:

[0086]

[0087] Because the intrinsic matrix of different cameras is also different, f x , f y , c x and c y are intrinsic matrix parameters.

[0088] In order to reduce the influence of distortion on the determination of semantic segmentation line (straight line), it is proposed to introduce a distortion correction subnetwork (DTN) in the depth semantic segmentation line network. At this time, the image (pixel coordinates represented as (u, v)) with distortion is known, and the undistorted image (pixel coordinates represented as (u', v')) is obtained only by calculating the corrected image according to formula 3 and formula 4. The task of the distortion correction subnetwork (DTN) is to learn the parameter matrix ρ = (k1, k2, k3, p1, p2, f x , f y , c x , c y ). The distortion correction subnetwork (DTN) is implemented by several convolutional layers and fully connected layers. ρ (U) represents the distortion correction transformation calculation, according to formula 3 and formula 4, the value of each pixel position (u', v') of the correction result (V) corresponds to the pixel position (u", v") of the undistorted image (U), the calculation result is not an integer, and needs to be interpolated according to the pixel value of U. Sampling correction to obtain the output V.

[0089] The distortion correction subnetwork DTN can dynamically perform spatial transformation according to different transformed images. The distortion correction subnetwork is a module independent of the semantic segmentation line network unit, and the DTN can be inserted at any position of the semantic segmentation line network unit. Before performing the depth Hough change straight line detection, the DTN network is accessed, that is, it can be connected after the image, or it can be connected after the convolution feature map. At the same time, the DTN module can also be used in other classification, detection and segmentation tasks to realize image distortion correction.

[0090] The training of the DTN depends on the training of the whole network, and the optimization target is to make the detected semantic segmentation line result better, without additional distortion image labeling, as a part of the whole network, without affecting the end-to-end training of the task.

[0091] The mine belt conveyor image structured modeling method according to the embodiment of the present application, preferably, before detecting and positioning the rollers and roller supports of the belt edge in each of the regions through the target bounding box, the method comprises:

[0092] Obtaining a plurality of initial bounding boxes;

[0093] Removing interference bounding boxes from the initial bounding boxes according to a set confidence threshold to obtain target bounding boxes.

[0094] Specifically, for the rollers and roller supports of the belt edge and other fixed facilities, a target detection method is used for detection and positioning. Since the shooting is not directly opposite the target, the target appears as a quadrilateral that is not a rectangle in the image. In order to perform high-precision structured modeling on the belt, the target detection method of the rotatable adjustment target bounding box is used to detect these fixed targets.

[0095] An initial bounding box is represented as (u, v, w, h, alpha1=s1 / w, alpha2=s2 / h, alpha3=s3 / w, alpha4=s4 / h). Where (u, v) is the center coordinate of the target bounding rectangle, (w, h) is the width and height of the target bounding rectangle. s1, s2, s3, s4 are intermediate variables representing the length of the image pixels, see Figure 2 For example, in the YOLOv5 implementation, only the four parameters (u, v, w, h) representing the target box are changed to (u, v, w, h, alpha1, alpha2, alpha3, alpha4). The original bounding box output of YOLOv5 detection has five parameters (u, v, w, h, condfidencescore), which is changed to (u, v, w, h, alpha1, alpha2, alpha3, alpha4, condfidencescore). When the whole network is finally a grid of SxS, each grid predicts B bounding boxes, and detects C categories of targets. The output of the whole network is SxSx(9xB+C). In the training, the loss function term about (alpha1, alpha2, alpha3, alpha4) is added to the optimization target Where alpha i is the predicted value, is the true value. is defined as:

[0096]

[0097] After detection, useless interference bounding boxes are removed according to a set confidence threshold, and then non-maximum suppression (NMS) is performed to obtain a target bounding box. The NMS can be based on a circumscribed rectangle or a quadrangle.

[0098] The mine belt conveyor image structured modeling method according to the embodiment of the present application, preferably, the method further comprises:

[0099] determining the position of the foreign matter based on the detection result;

[0100] judging whether the distance between the position and the semantic segmentation line meets a preset range;

[0101] if yes, determining that the foreign matter exists on the belt.

[0102] Specifically, for each detected foreign matter, its position is compared with the two segmentation lines using a mathematical method. Specifically, the distance between the foreign matter and the two segmentation lines is calculated using a point-to-line distance formula in geometry. If the distance meets a preset range, the foreign matter is determined to exist on the belt. This method can effectively eliminate errors caused by image viewing angles and other factors, and improve the accuracy of foreign matter detection.

[0103] The embodiment of the present application also provides a mine belt conveyor image structured modeling device, as shown in Figure 3 The device comprises:

[0104] a semantic segmentation line network unit 301 configured to acquire a mine belt conveyor image and segment the image into multiple regions based on semantic segmentation;

[0105] a target detection unit 302 configured to detect and locate a roller and a roller support of a belt edge in each region by target bounding box detection according to a target detection method;

[0106] a foreign matter detection unit 303 configured to detect whether a foreign matter exists in each region and acquire a detection result according to the located roller and the roller support;

[0107] a modeling unit 304 configured to convert the detection result into a vector and form a mine belt conveyor image structured modeling.

[0108] The embodiment of the present application also provides a device comprising a processor and a memory, wherein the processor implements the method according to any one of the above when executing a computer program stored in the memory.

[0109] A computer readable storage medium storing one or more programs, the one or more programs configured to be executed by one or more processors to implement the steps of any of the methods in the above specific embodiments.

[0110] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can be embodied in the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage media, etc.) having computer usable program code embodied thereon.

[0111] The present application is described in reference to the flowchart illustrations and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0112] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0114] The foregoing description of specific exemplary embodiments of the application has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the application to the precise forms disclosed, and obviously many modifications and variations are possible in light of the above teaching. It is intended that the scope of the application be limited not with this detailed description, but rather by the claims appended hereto.

Claims

1. A method for image-structured modeling of a mine belt conveyor, characterized in that, The method includes: Acquire images of a mine belt conveyor and segment the images into multiple regions based on semantic segmentation; Semantic segmentation lines are obtained from the image based on the depth Hough transform; The vanishing point of the image is determined based on the intersection of two or more parallel semantic segmentation lines; The belt edge in the image is determined based on the vanishing point; The image is segmented based on the edge of the belt; According to the target detection method, the rollers and roller brackets at the edge of the belt in each region are detected and located by the target bounding box; Based on the positioned rollers and roller brackets, the presence of foreign objects in each area is detected and the detection results are obtained; in particular, by monitoring the position and status of the rollers in the belt system, it is further determined whether the equipment is operating normally; by verifying whether the spatial position of the belt and the rollers is consistent, it is determined whether there are foreign objects blocking the belt track or abnormal equipment operation. The detection results are converted into vectors, and a structured model of the belt conveyor image is formed to represent the spatial positional relationships that are understood by human vision. The location of the foreign object is determined based on the detection results; Determine whether the distance between the location and the semantic segmentation line conforms to a preset range; If so, then it is determined that the foreign object exists on the belt.

2. The method for image-structured modeling of a mine belt conveyor according to claim 1, characterized in that, The method further includes: Output the belt reliability score and the distance of the belt from the vanishing point; The overall score is obtained based on the belt reliability score and the distance of the belt from the vanishing point; The edge of the belt with the highest overall score is used as the semantic segmentation line.

3. The image structured modeling method for a mine belt conveyor according to claim 2, characterized in that, The method further includes: The semantic segmentation lines are corrected using a distortion correction subnetwork.

4. The image structured modeling method for a mine belt conveyor according to claim 3, characterized in that, The method includes: The distortion correction subnetwork learns the parameter matrix. ;in, All are radial distortion correction parameters; p1 and p2 are tangential distortion parameters. These are the parameters of the intrinsic parameter matrix; The corrected image is calculated based on the parameter matrix to correct the semantic segmentation lines.

5. The image structured modeling method for a mine belt conveyor according to claim 1, characterized in that, Before detecting and locating the rollers and roller supports at the edges of the belt in each region using a target bounding box, the method includes: Obtain multiple initial bounding boxes; The target bounding box is obtained by removing interfering bounding boxes from the initial bounding box according to the set confidence threshold.

6. A structured image modeling device for mine belt conveyors, characterized in that, The device includes: A semantic segmentation line network unit is used to acquire images of a mine belt conveyor and segment the images into multiple regions based on semantic segmentation; wherein, the semantic segmentation line network unit obtains semantic segmentation lines from the image according to the depth Hough transform; The vanishing point of the image is determined based on the intersection of two or more parallel semantic segmentation lines; The belt edge in the image is determined based on the vanishing point; The image is segmented based on the edge of the belt; The target detection unit is used to detect and locate the rollers and roller brackets at the edge of the belt in each region by means of a target bounding box, according to the target detection method. The foreign object detection unit is used to detect whether there are foreign objects in each area based on the positioned rollers and roller brackets and to obtain the detection results; wherein, by monitoring the position and status of the rollers in the belt system, it is further determined whether the equipment is operating normally; by verifying whether the spatial position of the belt and the rollers is consistent, it is determined whether there are foreign objects blocking the belt track or abnormal equipment operation. The modeling unit is used to convert the detection results into vectors and form a structured model of the belt conveyor image for spatial positional relationships that are understood by human vision. The device is also used to determine the location of the foreign object based on the detection results; Determine whether the distance between the location and the semantic segmentation line conforms to a preset range; If so, then it is determined that the foreign object exists on the belt.

7. A device, characterized in that, It includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the method according to any one of claims 1 to 5.

Images