Machine vision-based automatic detection method for tiny foreign matters on long and straight guide rail

Abstract

The invention discloses a machine vision-based automatic detection method for tiny foreign matters on a long and straight guide rail, which comprises the following steps of: horizontally arranging a converging light source along the guide rail direction to enhance the visual difference between the foreign matters and the background; obtaining an imaging result of the whole guide rail range by using an automatic scanning camera; performing image preprocessing on an imaging result to obtain a potential foreign matter region thermodynamic diagram; performing region dynamic amplification on the potential foreign matter region thermodynamic diagram to obtain an amplified region bounding box; and carrying out feature modeling and coding on the amplified region bounding box through a deep attention convolutional neural network, and judging whether foreign matters exist or not. The algorithm is high in foreign matter detection and positioning precision; the non-contact measurement method does not influence the normal operation state of the equipment; the real-time performance is high, the sensitivity is high, and tiny foreign matters on the surface of the guide rail can be rapidly detected; the method is high in robustness and adaptability, and can meet task requirements of guide rail foreign matter intrusion detection in factories, railways and other scenes; compared with other types of sensors, the visual sensor is lower in cost and higher in titer ratio.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition and equipment maintenance, and more specifically, relates to a method for automatically detecting tiny foreign objects on a long straight guide rail based on machine vision. Background Art

[0002] Linear guide rails are common key components in mechanical equipment. Objects that may affect the normal and safe operation of the equipment may invade the operating area, such as fallen parts, broken ropes, lost tool parts and other solid foreign objects. Invading objects may get stuck in the gaps of the machinery, causing structural deformation or damage, leading to abnormal wear and shortening the service life, and creating huge safety hazards. Therefore, in order to ensure the safety and stability of the operation of linear guide rails, timely detection and handling of invading foreign objects on the guide rails is an important guarantee and prerequisite for the normal operation of linear guide rails.

[0003] At present, foreign body intrusion detection of linear guide rails mainly relies on manual inspections, which has great limitations and defects: inspections require shutdown, and manual inspections are usually regular inspections, which make it difficult to detect faults and issue alarms in a timely manner; manual inspections rely on the professional quality and physical and mental state of the staff, the working environment is relatively poor, which increases the labor intensity of the staff and poses many safety hazards. Summary of the invention

[0004] In order to overcome the defects of the above-mentioned background technology, the present invention provides a method for automatically detecting tiny foreign objects on a long straight guide rail based on machine vision.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A method for automatically detecting tiny foreign objects on a long straight guide rail based on machine vision, comprising:

[0007] Step 1: Arrange a convergent light source horizontally along the guide rail to enhance the visual difference between foreign objects and the background; use an automatic focus camera to obtain imaging results of the entire guide rail range;

[0008] Step 2, performing image preprocessing on the imaging results to obtain a thermal map of the potential foreign body area;

[0009] Step 3, dynamically zoom in on the potential foreign body area heat map to obtain the enlarged area bounding box;

[0010] Step 4: The enlarged area bounding box is feature modeled and encoded through a deep attention convolutional neural network to determine whether there is a foreign object.

[0011] Preferably, in step 2, the specific method of performing image preprocessing on the imaging result and obtaining the thermal map of the potential foreign body area includes:

[0012] Step 211, performing SLIC superpixel segmentation on the color image of the area where the foreign matter may occur;

[0013] For each pixel of the image, the following 5 features are extracted: the norm of the first-order gradient in the ax and y directions and the norm of the second-order gradient As well as the grayscale of the image I(x, y), each pixel can be represented by a 5-dimensional feature vector:

[0014]

[0015] According to the above eigenvectors, the covariance matrix of region R is calculated as shown in the following formula:

[0016]

[0017] Among them, μ is the mean of the regional feature vector, n is the total number of pixels in the region, and C is the d×d dimensional covariance matrix;

[0018] Step 212, extracting the enhanced Sigma features of each region after segmentation,

[0019] The Sigma feature is used as shown in the following formula:

[0020]

[0021] Where L i is the i-th column of matrix L, C = LL T , α is a coefficient, and the mean of the d-dimensional vector is introduced. Then the enhanced Sigma eigenvector of region C is as shown in the formula:

[0022] ψ C =(s1+μ, s2+μ, ..., s 2d +μ) T

[0023] Region R i The spatial distance weighted average of the enhanced Sigma features of the region R and its neighboring regions is defined as i The local foreign body significance of is:

[0024]

[0025] Where R j Belongs to region R i neighborhood, m represents the number of neighborhood areas, K represents the normalization factor to ensure that the sum of the spatial distance weighted coefficients is 1, ||R i -R j || represents the Euclidean distance between the centers of two regions, and σ1 represents the influence of the distance between regions on local significance. ψ Ri Indicates region Ri The enhanced Sigma feature of is Ri and ψ Rj The Euclidean distance of

[0026] Step 213, improve the region R using the following formula i The local foreign body significance S l (R i ):

[0027]

[0028] Among them, N(R j ) represents region R j The number of pixels in the image, N(I) represents the number of pixels in the image, R j The neighborhood belonging to the current region and the boundary region of the image;

[0029] Step 214, perform global kernel density estimation on each region one by one, construct a global heat map, and use the kernel density estimation of the features of the entire image region to calculate the global foreign body saliency S g (R i ), the specific formula is as follows

[0030]

[0031] where κ(x) is the Gaussian kernel density function, I Ri Represents the average grayscale of the region, and M is the number of image regions;

[0032] Step 215, using an exponential weighting method to fuse the local and global heat maps to obtain a fused heat map;

[0033] The combination of local and global foreign body saliency maps obtains the fusion heat map F pre , the formula is:

[0034] F pre =S g (R i )×exp(σ3×S l (R i ))

[0035] Among them, σ3 is the importance control parameter of the local saliency map. There are seven main parameters of the fusion heat map algorithm: the total number of superpixels ns, density cp, σ1, σ2, σ3, C and ra (Ratio); among them, ns and cp are parameters of the SLIC method, ns is the number of superpixels, the smaller the value, the larger the superpixel block; cp represents the shape of the superpixel, the smaller the value, the higher the degree of fit between the superpixel block and the regional block boundary; ns and cp need to be adjusted according to different images; among them, σ1=3, σ2=10, σ3=6, C=2000, ra>0.5.

[0036] Preferably, in step 3, the specific method of dynamically zooming in on the potential foreign body area heat map and obtaining the enlarged area bounding box includes:

[0037] rtg in the rectangular area of ​​the heat map after grid division i The total pixel value s ump x i ,

[0038]

[0039] Among them, px j Represents rtg i The pixel value of the jth pixel in the region, s ump x i Value and rectangular area rtg i is proportional to the amplified benefits;

[0040] The pixel value threshold is adaptively selected by the second-order difference method to complete the initial screening of the central block of the region; the heat map is divided into 8×8 to obtain 64 candidate regions, and each candidate region can obtain an overall pixel value s ump x i Indicates the magnification of benefits, and obtains a 64×1 array. If the element is less than 0.1, it is determined that there is no target and is discarded, thus obtaining an n×1 dimensional array C. Set the function f(g) to estimate s ump x i The trend of change from large to small is shown in the formula

[0041]

[0042] Then f(C k ) takes the maximum value of C k As s of this heat map ump x i Threshold value;

[0043] The 3×3 rectangles centered on the central block of each region constitute the enlarged screening region. If there are multiple rectangular regions that meet the pixel value threshold condition for the same enlarged screening region class, the one with the largest pixel value is taken as the region center;

[0044] Taking the center point of the enlarged screening area as the center position, 6 fixed-size prediction bounding boxes are predicted according to different aspect ratios. The area of ​​the enlarged screening area is S Z The area of ​​each predicted bounding box is as follows:

[0045]

[0046] Among them, s min =0.1×S Z ,s max =0.7×S Z , m = 5, giving different aspect ratios to different predicted bounding boxes

[0047]

[0048] Among them, W and H represent the width and length of the bounding box respectively, and the corresponding width and length of the bounding box are when a r = 1, there is also a predicted bounding box with a size of There are 6 predicted bounding boxes in total;

[0049] For any bounding box b l , calculate the total pixel value s in the box ump x i

[0050]

[0051] Area S

[0052] S(b l )=W×L

[0053] The proportion of high magnification gain pixels in the area P

[0054]

[0055] Pn1 means b l The total number of pixels with magnification benefits in the region, Pn1 represents b l The total number of pixels in the region, each predicted bounding box b l The eigenvector of is (x, y, sumpx, W, L, P), where x and y represent b l The horizontal and vertical coordinates of the center point;

[0056] For the input predicted bounding box, b l (x, y, sumpx, W, L, P), Logistic classifier, introduce weight parameter θ = (θ1, θ2, K, θ6), for b l We weight the attributes in to get θT b l , introduce the logistic function (sigmoid function) to get the function h θ (b l )

[0057]

[0058] The probability estimation function P(y|b l ;θ)

[0059]

[0060] It means that given a test sample b l With parameter θ, the probability of label y;

[0061] From the test sample set and the training sample set, obtain the joint probability density likelihood function:

[0062]

[0063] Maximize the likelihood function and find the appropriate parameter θ. The formula is transformed into

[0064]

[0065] Obtain the parameter θ by the gradient descent method; derive the parameter θ

[0066]

[0067] The update rule formula is as follows:

[0068] θ j : =θ j +α((y (i) -h θ (b l (i) ))b lj (i) )

[0069] After evaluating the selection effect of each predicted bounding box through the Logistic classifier, a corresponding selection evaluation score is assigned to each predicted bounding box, and then a non-maximum suppression is performed to obtain the final prediction as the final enlarged bounding box; the Logistic classifier is a classifier trained using manually calibrated training samples to evaluate the selection effect of each predicted bounding box;

[0070] After completing the selection of the enlarged bounding box, all pixel values ​​in the enlarged screening area are set to 0, and the corresponding area of ​​the heat map is updated at the same time, and it is detected whether all high enlargement yield areas on the heat map have been detected. If so, the detection is completed, if not, the detection process continues to be iterated;

[0071] The original image of the obtained enlarged detection candidate area is first enlarged by bilinear interpolation to the minimum size of the detector detection candidate area, and then input into the subsequent foreign body detector for fine detection.

[0072] Preferably, the method for determining whether all high magnification yield areas on the heat map have been detected is: whether the total value of the heat map pixels is 0, if so, it means that all high magnification yield areas have been detected, if not, it means that not all high magnification yield areas have been detected.

[0073] Preferably, the minimum size of the candidate region to be detected by the detector is enlarged to 10×10.

[0074] Preferably, in step 4, feature modeling and encoding are performed on the enlarged area bounding box through a deep attention convolutional neural network, and the method for determining whether there is a foreign object includes:

[0075] The screened foreign body potential area image is subjected to feature modeling and encoding through a deep attention convolutional neural network, including 3 attention layers, 13 convolutional layers and 3 fully connected layers. The feature modeling and encoding is realized through a deep attention convolutional neural network composed of 3 attention layers, 13 convolutional layers and 3 fully connected layers. The deep features obtained by the deep attention convolutional neural network encoding are input into the SoftMax function to determine whether there is a foreign body in the area. The formula is as follows: soft max(x) i 1 indicates that there is foreign matter, and 0 indicates that there is no foreign matter:

[0076]

[0077] The beneficial effects of the present invention are: the foreign body detection and positioning accuracy of the algorithm of the present invention is high; the non-contact measurement method does not affect the normal operating status of the equipment; it has strong real-time performance and high sensitivity, and can quickly detect tiny foreign bodies on the surface of the guide rail; the method has strong robustness and adaptability, and can meet the task requirements of foreign body intrusion detection on guide rails in scenarios such as factories and railways, and has good application prospects; compared with other types of sensors, visual sensors have lower costs and higher cost-effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 Implementation principle of the embodiment of the present invention Figure 1 ,

[0079] Figure 2Implementation principle of the embodiment of the present invention Figure 2 ,

[0080] Figure 3 This is a diagram showing the effect of implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0081] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. Figure 1 and Figure 2 This is a schematic diagram of an implementation principle of an embodiment of the present invention. Figure 2 Since the fixed-focus camera has a limited field of view, it is not possible to clearly image the entire area of ​​the long straight rail. Therefore, the designed zoom camera can achieve clear imaging of the entire area of ​​the long straight rail. The drive motor drives the camera to move horizontally, and the zoom distance is achieved by changing the distance between the camera and the lens.

[0082] A method for automatically detecting tiny foreign objects on a long straight guide rail based on machine vision, comprising:

[0083] Step 1: Arrange a convergent light source horizontally along the guide rail to enhance the visual difference between foreign objects and the background; use an automatic focus camera to obtain imaging results of the entire guide rail range;

[0084] Step 2: For the imaging result obtained in step 1, the following operations are performed to detect and locate tiny foreign objects, image preprocessing, (in order to avoid local noise in the image causing highlight areas similar to foreign objects) the median filtering algorithm is used to remove local highlight noise points, according to the preset actual position of the guide rail, the area where the foreign object may occur is selected, and the guide rail area in the image is manually selected as the input area of ​​the subsequent foreign object detection algorithm; the heat map of the potential foreign object area is obtained (to reduce the amount of calculation, improve the operating efficiency, and enhance the sensitivity of the model to faint small targets); the specific method includes the following steps:

[0085] Step 21, performing SLIC (Simple Linear Iterative Cluster) simple linear iterative clustering superpixel segmentation on the color image of the selected area where the foreign matter may occur;

[0086] Step 22, extracting the enhanced Sigma features of each region after segmentation;

[0087] Step 23, comprehensively considering the influence of region size, neighborhood contrast, spatial distance, and background contrast, to generate a local heat map;

[0088] Step 24, performing global kernel density estimation on each region one by one to construct a global heat map;

[0089] Step 25, using exponential weighting to fuse the local and global heat maps to obtain a fused heat map.

[0090] The specific process corresponding to the above steps 21 to 25 includes:

[0091] This embodiment extracts the following five features for each pixel of the image: the norm of the first-order gradient in the x and y directions and the norm of the second-order gradient As well as the grayscale of the image I(x, y), each pixel can be represented by a 5-dimensional feature vector:

[0092]

[0093] According to the above eigenvectors, the covariance matrix of region R is calculated as shown in the formula:

[0094]

[0095] Wherein, μ is the mean of the regional feature vector, and n is the total number of pixels in the region. In order to make the regional covariance better reflect the characteristics of the image region and facilitate the calculation of similarity, C is a d×d dimensional covariance matrix. The present invention uses the Sigma feature as shown in the formula:

[0096]

[0097] Where L i is the i-th column of matrix L, C = LL T , α is a coefficient, and the mean of the d-dimensional vector is introduced. Then the enhanced Sigma eigenvector of region C is as shown in the formula:

[0098] ψ C =(s1+μ, s2+μ, ..., s 2d +μ) T

[0099] Region R i The spatial distance weighted average of the enhanced Sigma features of the region R and its neighboring regions is defined as i The local foreign body significance of is:

[0100]

[0101] Where R j Belongs to region R i neighborhood, m represents the number of neighborhood areas, K represents the normalization factor to ensure that the sum of the spatial distance weighted coefficients is 1, ||R i -R j || represents the Euclidean distance between the centers of two regions, and σ1 represents the influence of the distance between regions on local significance. ψ Ri Indicates region R i The enhanced Sigma feature of isRi and ψ Rj After comprehensively considering the influence of neighborhood contrast, background contrast, spatial distance and region size, the improved region R i The local foreign body significance is as follows:

[0102]

[0103] Where N(R j ) represents region R j The number of pixels in the image, N(I) represents the number of pixels in the image, R j The neighborhood of the current region and the boundary area of ​​the image. The probability of the gray value of each region appearing can represent the global foreign body saliency of the region. The target region with a low probability of appearing means that there is a higher probability of foreign body targets, and vice versa, it may be the background region. Therefore, the kernel density estimation of the features of the entire image region is used to calculate the global foreign body saliency, as shown in the formula

[0104]

[0105] where κ(x) is the Gaussian kernel density function, I Ri represents the average grayscale of the region, and M is the number of image regions. Considering that local foreign body saliency is usually better than global foreign body saliency, and the exponential function can increase the importance of local foreign body saliency, the combination of local and global foreign body saliency maps to obtain the fusion heat map formula is:

[0106] F pre =S g (R i )×exp(σ3×S l (R i ))

[0107] Among them, σ3 controls the importance of the local saliency map. There are seven main parameters of the fusion heat map algorithm: ns (numSuperpixels), cp (Compactness), σ1, σ2, σ3, C and ra (Ratio). Among them, ns and cp are parameters of the SLIC method, ns is the number of superpixels, the smaller the value, the larger the superpixel block; cp represents the shape of the superpixel, the smaller the value, the higher the degree of fit between the superpixel block and the regional block boundary. ns and cp need to be adjusted according to different images. The present invention selects σ1=3, σ2=10, σ3=6, C=2000, ra>0.5.

[0108] Step 22: Dynamically zoom in on the potential foreign body area heat map obtained in step 21; the strength of the heat map response reflects the possibility of the presence of foreign bodies in the area. For areas where there is a response but the degree is weak, the present invention uses a dynamic selection strategy for zooming in on the area. According to the acquired foreign body area heat map, the area where low-resolution weak abnormal state targets may exist is sequentially selected for zooming in to achieve data enhancement, thereby avoiding missed detection and false detection. The specific method of step 22 for dynamically zooming in on the potential foreign body area heat map obtained in step 21 includes: dividing the heat map into several rectangular areas according to an 8×8 grid, and counting the sum of the pixel values ​​in each rectangular area. The present invention adaptively selects a pixel value threshold through a subsequent second-order difference method, sets a threshold, and selects a regional center block. The 3×3 rectangles centered on each regional center block constitute an zoomed-in screening area. It is determined whether there are multiple rectangular areas that meet the pixel value threshold conditions in each zoomed-in screening area. If so, the rectangular area with the largest pixel value is used as the regional center block.

[0109] Whether the center block of the region is on the edge of the large square is determined by the following method: first, the 8×8 rectangular blocks are numbered from left to right and then from top to bottom, and whether the rectangular block belongs to an edge rectangular block is determined based on the numbering; if the center block of the region is on the edge of the large square after the above determination, a 3×3 enlarged screening area is formed by adding small blank squares of the same size; within the enlarged screening area, 6 prediction bounding boxes with different aspect ratios are constructed based on artificial experience and with the center point of the enlarged screening area as the center; the best enlarged region bounding box is selected by comparing the construction indicators (pixel value, ratio, area) of the areas contained in each prediction bounding box; the comparison and selection method is as follows: the number of pixels of the target area in each bounding box and the ratio of the pixels of the target area to all the pixels of the bounding box are counted.

[0110] The specific mathematical solution of step 22 is as follows:

[0111] rtg in the rectangular area of ​​the heat map after grid division i The total pixel value s ump x i , as shown in the formula

[0112]

[0113] where px j Represents rtg i The pixel value of the jth pixel in the region, s ump x i The larger the value, the more it represents a rectangular area rtg iThe greater the magnification benefit. The present invention uses the second-order difference method to adaptively select the pixel value threshold to complete the initial screening of the regional center block. For a heat map, 64 candidate regions are obtained by 8×8 division, and each candidate region can obtain an overall pixel value s ump x i It is used to represent the magnification benefit, so a total of 64×1 arrays can be obtained. When the element is less than 0.1, it is judged as no target and discarded, thus obtaining an n×1 dimensional array C. Assume that the function f(g) is used to estimate s ump x i The trend of change from large to small is shown in the formula

[0114]

[0115] Then f(C k ) takes the maximum value of C k As s of this heat map ump x i Threshold.

[0116] In order to reduce the amount of calculation for regional magnification and precise detection, effectively improve the efficiency and real-time performance of the algorithm, and at the same time ensure that the selected area has good inclusiveness, the present invention uses a 3×3 rectangle centered on the central block of each region to form an enlarged screening area. If there are multiple rectangular areas that meet the pixel value threshold conditions for the same enlarged screening area class, the one with the largest pixel value is taken as the area center.

[0117] The present invention takes the center point of the enlarged screening area as the center position, predicts 6 prediction bounding boxes of fixed sizes according to different aspect ratios, and the area of ​​the enlarged screening area is S Z The area formula of each predicted bounding box is as follows:

[0118]

[0119] where s min =0.1×S Z ,s max =0.7×S Z , m = 5, we assign different aspect ratios to different predicted bounding boxes

[0120]

[0121] W and H represent the width and length of the bounding box respectively. Then the width and length of the predicted bounding box are when a r = 1, there is also a predicted bounding box with a size of That is, there are 6 predicted bounding boxes in total. For any bounding box b l , calculate the total pixel value s in the box ump xi

[0122]

[0123] Area S

[0124] S(b l )=W×L

[0125] The proportion of high magnification gain pixels in the area P

[0126]

[0127] Pn1 means b l The total number of pixels with magnification benefits (i.e., pixels with pixel values ​​greater than 0.1) in the region, Pn1 represents b l The total number of pixels in the region. That is, each predicted bounding box, b l There is a eigenvector (x,y,sumpx,W,L,P), where x and y represent b respectively. l The horizontal and vertical coordinates of the center point.

[0128] The present invention uses manually calibrated training samples to train a Logistic classifier to evaluate the selection effect of each predicted bounding box, and divides the evaluation results into two categories, namely, the predicted bounding boxes that can meet the magnification requirements and the predicted bounding boxes that cannot meet the magnification requirements.

[0129] For the input predicted bounding box, b l (x,y,sumpx,W,L,P), Logistic classifier, introduce weight parameter θ=(θ1,θ2,K,θ6), for b l We weight the attributes in to get θ T b l , introduce the logistic function (sigmoid function) to get the function h θ (b l )

[0130]

[0131] The probability estimation function P(y|b l ;θ)

[0132]

[0133] It means that given a test sample b l The probability that the label is y when the parameter θ is used.

[0134] From the test sample set and the training sample set, we can get their joint probability density, that is, the likelihood function:

[0135]

[0136] Maximize the likelihood function and find the appropriate parameter θ. The formula is transformed into

[0137]

[0138] The parameter θ is obtained by gradient descent. First, the parameter θ is differentiated

[0139]

[0140] The update rule is shown in the formula

[0141] θ j : =θ j +α((y (i) -h θ (b l (i) ))b lj (i) )

[0142] After evaluating the selection effect of each predicted bounding box through the Logistic classifier, we can obtain a corresponding selection evaluation score for each predicted bounding box. Then, a non-maximum suppression is performed to obtain the final prediction as the final enlarged bounding box.

[0143] After completing the selection of the magnified bounding box, we set all the pixel values ​​in the magnified screening area to 0 to avoid the inefficiency caused by repeated selection. At the same time, we update the corresponding area of ​​the heat map and check whether all high magnification yield areas have been detected on the heat map (whether the total value of the heat map pixels is 0). If so, the detection is completed, otherwise the detection process continues to iterate.

[0144] The original image portion of the obtained enlarged detection candidate area is first enlarged by bilinear interpolation to the minimum size of the detector detection candidate area (the minimum candidate area set by the present invention is 10×10), and then sent to the subsequent foreign body detector for fine detection.

[0145] The screened foreign body potential area images are feature modeled and encoded through a deep attention convolutional neural network, including 3 attention layers, 13 convolution layers and 3 fully connected layers. Is the relationship between "feature modeling and encoding" and the following "3 attention layers, 13 convolution layers and 3 fully connected layers" as follows: feature modeling includes 3 attention layers, 13 convolution layers and 3 fully connected layers; the obtained deep features are input into the SoftMax function to determine whether there is a foreign body in the area. The formula is as follows:

[0146]

[0147] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A method for automatic detection of tiny foreign objects on long straight guide rails based on machine vision. It is characterized in that include: Step 1: Arrange the convergent light source horizontally along the guide rail to enhance the visual difference between the foreign object and the background; Use an auto-scanning camera to obtain imaging results of the entire rail range; Step 2, performing image preprocessing on the imaging result to obtain a thermal map of the potential foreign body area; Step 3, dynamically zooming in on the potential foreign body area heat map to obtain a bounding box of the zoomed area; Step 4: The enlarged area bounding box is feature modeled and encoded through a deep attention convolutional neural network to determine whether there is a foreign object.

2. According to the method of automatic detection of tiny foreign bodies on a long straight guide rail based on machine vision according to claim 1, It is characterized in that The specific method of performing image preprocessing on the imaging result in step 2 to obtain a thermal map of the potential foreign body area includes: Step 211, performing SLIC superpixel segmentation on the color image of the selected area where the foreign matter may occur; For each pixel of the image, the following 5 features are extracted: the norm of the first-order gradient in the ax and y directions and the norm of the second-order gradient As well as the grayscale of the image I(x, y), each pixel can be represented by a 5-dimensional feature vector: According to the above eigenvectors, the covariance matrix of region R is calculated as shown in the following formula: Among them, μ is the mean of the regional feature vector, n is the total number of pixels in the region, and C is the d×d dimensional covariance matrix; Step 212, extracting the enhanced Sigma features of each region after segmentation, The Sigma feature is used as shown in the following formula: Where Li is the i-th column of matrix L, C = LLT, α is the coefficient, and the mean of the d-dimensional vector is introduced. The enhanced Sigma eigenvector of region C is shown in the formula: ψC=(s1+μ, s2+μ,.., s2d+μ)T Region R i The spatial distance weighted average of the enhanced Sigma features of the region R and its neighboring regions is defined as i The local foreign body significance of is: Where R j Belongs to region R i neighborhood, m represents the number of neighborhood areas, K represents the normalization factor to ensure that the sum of the spatial distance weighted coefficients is 1, ||R i -R j || represents the Euclidean distance between the centers of two regions, σ 1 Represents the effect of the distance between regions on local significance. Ri Indicates region R i The enhanced Sigma feature of is Ri and ψ Rj The Euclidean distance of Step 213, improve the region R using the following formula i The local foreign body significance S l (R i ): Among them, N(R j ) represents region R j The number of pixels in the image, N(I) represents the number of pixels in the image, R j The neighborhood belonging to the current region and the boundary region of the image; Step 214, perform global kernel density estimation on each region one by one, construct a global heat map, and use the kernel density estimation of the features of the entire image region to calculate the global foreign body saliency S g (R i ), the specific formula is as follows where κ(x) is the Gaussian kernel density function, I Ri Represents the average grayscale of the region, and M is the number of image regions; Step 215, using an exponential weighting method to fuse the local and global heat maps to obtain a fused heat map; The combination of local and global foreign body saliency maps obtains the fusion heat map F pre , the formula is: F pre =S g (R i )×exp(σ 3 ×S l (R i )) Among them, σ 3 The importance control parameters of the local saliency map, the parameters of the fusion heat map algorithm mainly include seven: total number of superpixels ns, density cp, σ 1 , σ 2 , σ 3 , C and ra(Ratio); where ns and cp are the parameters of the SLIC method, ns is the number of superpixels, the smaller the value, the larger the superpixel block; cp represents the shape of the superpixel, the smaller the value, the higher the fit between the superpixel block and the regional block boundary; ns and cp need to be adjusted according to different images; where σ 1 =3,σ 2 =10,σ 3 =6, C=2000, ra>0.

5.

3. According to the method of automatic detection of tiny foreign bodies on a long straight guide rail based on machine vision in claim 1, It is characterized in that The specific method of dynamically amplifying the potential foreign body area heat map and obtaining the enlarged area bounding box in step 3 includes: rtg in the rectangular area of ​​the heat map after grid division i The total pixel value s ump x i , Among them, px j Represents rtg i The pixel value of the jth pixel in the region, s ump x i Value and rectangular area rtg i is proportional to the amplified benefits; The pixel value threshold is adaptively selected by the second-order difference method to complete the initial screening of the central block of the region; the heat map is divided into 8×8 to obtain 64 candidate regions, and each candidate region can obtain an overall pixel value s ump x i Indicates the magnification of benefits, and obtains a 64×1 array. If the element is less than 0.1, it is determined that there is no target and is discarded, thus obtaining an n×1 dimensional array C. Set the function f(g) to estimate s ump x i The trend of change from large to small is shown in the formula Then f(C k ) takes the maximum value of C k As s of this heat map ump x i Threshold value; The 3×3 rectangles centered on the central block of each region constitute the enlarged screening region. If there are multiple rectangular regions that meet the pixel value threshold condition for the same enlarged screening region class, the one with the largest pixel value is taken as the region center; Taking the center point of the enlarged screening area as the center position, 6 prediction bounding boxes of fixed sizes are predicted according to different aspect ratios. The area of ​​the enlarged screening area is S Z The area of ​​each predicted bounding box is as follows: Among them, s min =0.1×S Z ,s max =0.7×S Z ,m=5, different aspect ratios are assigned to different prediction bounding boxes Among them, W and H represent the width and length of the bounding box respectively, and the corresponding width and length of the bounding box are when a r = 1, there is also a predicted bounding box with a size of There are 6 predicted bounding boxes in total; For any bounding box b l , calculate the total pixel value s in the box ump x i Area S S(b l )=W×L The proportion of high magnification gain pixels in the area P Pn 1 Indicates b l The total number of pixels with magnification gain in the region, Pn 1 Indicates b l The total number of pixels in the region, each predicted bounding box b l The eigenvector of is (x, y, sumpx, W, L, P) x, y represent b l The horizontal and vertical coordinates of the center point; For the input predicted bounding box, b l (x,y,sumpx,W,L,P)Logistic classifier, introduce weight parameter θ=(θ 1 ,θ 2 ,K,θ 6 ) for b 1 We weight the attributes in to get θ T b 1 , introduce the logistic function (sigmoid function) to get the function h θ (b l ) The probability estimation function P(y|b l ;θ) It means that given a test sample b l With parameter θ, the probability of label y; From the test sample set and the training sample set, obtain the joint probability density likelihood function: Maximize the likelihood function and find the appropriate parameter θ. The formula is transformed into Obtain the parameter θ by the gradient descent method; derive the parameter θ The update rule formula is as follows: i j =θ j +α((y (i) -h θ (b) l (i) ))b lj (i) ) After evaluating the selection effect of each predicted bounding box by a Logistic classifier, a corresponding selection evaluation score is assigned to each predicted bounding box, and then a non-maximum suppression is performed to obtain a final prediction as the final enlarged bounding box; the Logistic classifier is a classifier trained using manually calibrated training samples to evaluate the selection effect of each predicted bounding box; After completing the selection of the enlarged bounding box, all pixel values ​​in the enlarged screening area are set to 0, and the corresponding area of ​​the heat map is updated at the same time, and it is detected whether all high enlargement yield areas on the heat map have been detected. If so, the detection is completed, if not, the detection process continues to be iterated; The original image of the enlarged detection candidate area is first enlarged by bilinear interpolation to the minimum size of the detector detection candidate area, and then input into the subsequent foreign body detector for fine detection.

4. According to the method of automatic detection of tiny foreign bodies on long straight rails based on machine vision according to claim 3, Features : The method for judging whether all high magnification gain areas have been detected on the heat map is: whether the total value of the heat map pixels is 0, if so, it means that all high magnification gain areas have been detected, if not, it means that all high magnification gain areas have not been detected.

5. According to the method of automatic detection of tiny foreign bodies on long straight rails based on machine vision as described in claim 3, Features : Enlarge the minimum size of the candidate region detected by the detector to 10×10.

6. According to the method of automatic detection of tiny foreign bodies on long straight rails based on machine vision according to claim 1, It is characterized in that In step 4, the enlarged area bounding box is subjected to feature modeling and encoding through a deep attention convolutional neural network, and the method for determining whether there is a foreign object includes: The screened foreign body potential area image is feature modeled and encoded through a deep attention convolutional neural network, including 3 attention layers, 13 convolutional layers and 3 fully connected layers. The deep attention convolutional neural network composed of 3 attention layers, 13 convolutional layers and 3 fully connected layers realizes feature modeling and encoding. The deep features obtained by the deep attention convolutional neural network encoding are input into the SoftMax function to determine whether there is a foreign body in the area. The formula is as follows: soft max(x) i 1 indicates that there is foreign matter, and 0 indicates that there is no foreign matter:

Images