An industrial feeding abnormity identification method based on visual flow detection

Abstract

This invention relates to the field of image processing, and more particularly to an industrial loading anomaly identification method based on visual flow detection. The method includes: acquiring and preprocessing an image sequence of an industrial loading port to obtain an image sequence for optical flow calculation; obtaining a specular suppression factor by nonlinearly fusing pixel brightness values ​​and temporal brightness change rates; obtaining a gradient structure confidence factor by performing distribution analysis on the local gradient structure tensor eigenvalues; obtaining a dynamic spatiotemporal confidence weight by jointly weighting the specular suppression factor and the gradient structure confidence factor; and obtaining the loading anomaly identification result by performing weighted optical flow calculation and variance threshold discrimination on the dynamic spatiotemporal confidence weight. This method solves the problems of optical flow calculation distortion and false alarms caused by specular highlights during the loading of SMT micro-components.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for identifying industrial material feeding anomalies based on visual flow detection. Background Technology

[0002] In modern electronics manufacturing, surface mount technology (SMT) has become the mainstream process for assembling microelectronic components. Real-time monitoring of the feeding process at the feeder outlet of micro-components on SMT production lines is crucial for ensuring production efficiency and product quality. The stability and continuity of the feeding process directly determine whether the pick-and-place machine can maintain high-speed operation. Issues such as material blockage, empty feed, or unstable flow not only lead to decreased production efficiency but may also increase defective products. Therefore, the industry widely adopts non-contact monitoring solutions based on machine vision, using image analysis technology to determine the feeding status, which has become an important means of achieving intelligent production line management. In existing technologies, a common method is to use an industrial camera fixed above the feeder outlet to continuously acquire image sequences from the outlet and use optical flow algorithms to model and analyze material movement. Among these, the Lucas-Kanade (LK) optical flow algorithm is widely used due to its high computational efficiency and relatively simple implementation. This method is based on the assumption of constant brightness, meaning that the brightness of the same physical point remains unchanged between adjacent image frames. By solving for the motion vectors of pixels in the local neighborhood using the least squares method, a dense optical flow field can be obtained. Based on this, the state of the loading process can be judged by statistically analyzing indicators such as the magnitude, direction consistency, and variance of the optical flow vectors. However, in actual high-speed SMT loading scenarios, the optical properties of the component electrode surfaces can significantly interfere with the reliability of this method. In particular, when metal electrodes form a specific reflection angle with industrial lighting sources, specular highlights are easily generated in the image. The brightness of these highlights changes drastically within a very short time, resulting in a temporal gradient much larger than the brightness change caused by normal motion, thus violating the constant brightness assumption of the optical flow algorithm.

[0003] For the reasons mentioned above, traditional LK-based optical flow monitoring methods generate numerous pseudo-motion vectors in the presence of specular highlights, contaminating the entire optical flow field. Subsequent anomaly detection modules based on optical flow statistics incorrectly classify this optically-induced distortion as "unstable material feeding," leading to frequent false alarms and unnecessary shutdowns, severely impacting the automation level and overall equipment efficiency of SMT production lines. Therefore, effectively suppressing the interference of optical artifacts such as specular highlights on optical flow calculations while ensuring real-time performance, and improving the accuracy of material feeding anomaly identification, has become a pressing technical problem to be solved. Summary of the Invention

[0004] In view of this, the present invention aims to propose an industrial material loading anomaly identification method based on visual flow detection, so as to solve the problems of optical flow calculation distortion and false alarms caused by specular highlights during the loading of SMT micro components.

[0005] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0006] A method for identifying industrial material feeding anomalies based on visual flow detection, the method comprising:

[0007] Step S1: Acquire and preprocess the image sequence of the industrial feed port to obtain an image sequence for optical flow calculation;

[0008] Step S2: Obtain the specular suppression factor by nonlinearly fusing the pixel brightness value with the time brightness change rate;

[0009] Step S3: Obtain the gradient structure confidence factor by performing distribution analysis on the eigenvalues ​​of the local gradient structure tensor;

[0010] Step S4: Obtain the dynamic spatiotemporal confidence weight by jointly weighting the specular suppression factor and the gradient structure confidence factor;

[0011] Step S5: Obtain the results of abnormal feeding status identification by performing weighted optical flow calculation and variance threshold discrimination on the dynamic spatiotemporal confidence weights.

[0012] Furthermore, the step of acquiring and preprocessing the image sequence of the industrial feed port to obtain the image sequence for optical flow calculation includes:

[0013] An industrial area array camera is fixedly installed directly above the feeder outlet, and a coaxial LED light source is used to ensure clear and uniform illumination for image acquisition. The camera continuously acquires image sequences from the feeder outlet at a preset frame rate, with a resolution of 640×480 pixels and a frame rate of 100 frames per second. The acquired image sequences are 8-bit grayscale images. The acquired image sequences are processed by a 3×3 pixel Gaussian filter to obtain an image sequence for optical flow calculation.

[0014] Furthermore, the step of obtaining the specular suppression factor by nonlinearly fusing pixel brightness values ​​with the rate of change of brightness over time includes:

[0015] By performing feature extraction processing on the brightness values ​​and time-varying brightness change rate data of pixels in the image sequence, brightness features and change features for confidence assessment are obtained.

[0016] By performing nonlinear fusion processing on brightness features and variation features, the specular suppression factor corresponding to each pixel is obtained.

[0017] Furthermore, the step of extracting features from the brightness values ​​and time-varying brightness change rate data of pixels in the image sequence to obtain brightness features and change features for confidence assessment includes:

[0018] For any target pixel at any target time in the image sequence of all time times, the gray value of the target pixel in the image at the target time is divided by the maximum gray value of the image sensor as the brightness feature of the target pixel in the image at the target time.

[0019] The gradient of the image at the target time is obtained by performing gradient calculation on the image at the target time using the Sobel operator. For the target pixel in the gradient image at the target time, the square of the horizontal gradient of the target pixel is added to the square of the vertical gradient and the square root is calculated. The square root calculation result is used as the gradient evaluation of the target pixel. The gradient evaluation of the target pixel is normalized and added to the constant 1. The result is used as the texture richness of the target pixel.

[0020] The absolute value of the difference between the gradient evaluation of the target pixel in the image at the target time and the gradient evaluation of the target pixel at the previous time is used as the time partial derivative evaluation at the target time. The texture richness of each pixel in the image at the target time is used as the weight to perform a weighted calculation of the time partial derivative evaluation of each pixel, and the median of the weighted calculation results of all pixels in the image at the target time is used as the adaptive time change scale factor at the target time.

[0021] The change feature of the target pixel in the image at the target time is obtained by dividing the time partial derivative of the target pixel in the image at the target time by the calculated result of the adaptive time-varying scaling factor at the target time.

[0022] Furthermore, the step of obtaining the specular suppression factor corresponding to the pixel by performing nonlinear fusion processing on the brightness features and variation features includes:

[0023] For any target pixel at any target time in the image sequence across all time points, the negative of the result of multiplying the brightness feature and the change feature of the target pixel is subjected to an exponential mapping with the natural constant as the base, and the corresponding mapping result is used as the highlight suppression factor of the target pixel.

[0024] Furthermore, the step of obtaining the gradient structure confidence factor by performing distribution analysis on the eigenvalues ​​of the local gradient structure tensor includes:

[0025] By performing local neighborhood tensor quantization on the spatial gradient data of pixels in an image sequence, the feature values ​​of the local gradient structure tensor are obtained.

[0026] By performing morphological and intensity differentiation processing on the eigenvalues ​​of the local gradient structure tensor, gradient anisotropy and significance gating indices are obtained.

[0027] The gradient structure confidence factor is obtained by jointly weighting the gradient anisotropy degree and the significance gating index.

[0028] Furthermore, the step of obtaining local gradient structure tensor feature values ​​by performing local neighborhood tensor quantization on the spatial gradient data of pixel points in the image sequence includes:

[0029] Define the neighborhood range of a pixel; for any target pixel in the image at any target time, take the square of the horizontal gradient of the target pixel as the top-left element of the structure matrix, take the result of multiplying the horizontal gradient and the vertical gradient of the target pixel as the top-right and bottom-left elements of the structure matrix, and take the square of the vertical gradient of the target pixel as the bottom-right element of the structure matrix; obtain the structure tensor of the target pixel by summing the structure matrices of all pixels in the neighborhood range of the target pixel; perform eigenvalue analysis on the structure tensor of the target pixel to obtain two eigenvalues ​​of the structure tensor of the target pixel, take the larger eigenvalue as the first eigenvalue of the target pixel, and take the smaller eigenvalue as the second eigenvalue of the target pixel.

[0030] Furthermore, the step of obtaining gradient anisotropy and significance gating indices by performing morphological and intensity differentiation processing on the eigenvalues ​​of the local gradient structure tensor includes:

[0031] For any target pixel in the image at any target time, the result of subtracting the first feature value and the second feature value of the target pixel is used as the numerator, and the result of adding the first feature value and the second feature value of the target pixel is used as the denominator. The corresponding fraction is used as the anisotropy of the target pixel.

[0032] The median of the first eigenvalues ​​of all pixels in the target time image is used as the adaptive gradient intensity scaling factor of the target time image; the result of dividing the first eigenvalue of the target pixel by the adaptive gradient intensity scaling factor of the target time image is mapped by the hyperbolic tangent function, and the corresponding mapping result is used as the saliency gating index of the target pixel.

[0033] Furthermore, the step of obtaining the gradient structure confidence factor by jointly weighting the gradient anisotropy degree and the significance gating index includes:

[0034] For any target pixel in the image at any target time, the negative of the result of multiplying the anisotropy of the target pixel by the significance gating index is used for exponential mapping with the natural constant as the base. The corresponding mapping result is used as the gradient structure confidence factor of the target pixel at the target time.

[0035] Furthermore, the method of obtaining dynamic spatiotemporal confidence weights by jointly weighting the specular suppression factor and the gradient structure confidence factor includes:

[0036] For any target pixel in the image at any target time, the result of multiplying the specular suppression factor of the target pixel by the gradient structure confidence factor is used as the dynamic spatiotemporal confidence weight of the target pixel.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] This invention discloses an industrial material feeding anomaly identification method based on visual flow detection. By introducing two types of weighting factors—a highlight suppression factor and a gradient structure confidence factor—the reliability of data at different pixels is dynamically adjusted during the optical flow field calculation, significantly reducing the interference of optical artifacts. Because sudden brightness changes and abnormal gradients in the specular highlight region are promptly identified and assigned low-confidence weights, the optical flow calculation process avoids deviations caused by these pseudo-signals. The resulting optical flow field more closely resembles the actual movement of the material, enabling the monitoring system to stably capture the physical flow characteristics during the feeding process. Even in complex scenarios with strong light irradiation and high-speed component rotation, the accuracy and stability of the optical flow vector are guaranteed.

[0039] At the application level, this invention significantly reduces false alarms caused by optical artifacts in the material feeding monitoring system, avoiding frequent and unnecessary downtime and manual intervention on the production line, thereby improving the automation level of the production process. Through linkage with the production line control system, the system can accurately respond to real anomalies such as material blockage, empty material, and unstable flow, ensuring continuous equipment operation while triggering safety protection and process adjustments when necessary. This optimization method, which fundamentally improves robustness, not only enhances the overall efficiency and product quality of SMT production lines but also provides a reliable solution for high-precision intelligent inspection in the electronics manufacturing industry under complex lighting conditions. Attached Figure Description

[0040] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0041] Figure 1This is a flowchart of an industrial material feeding anomaly identification method based on visual flow detection, as described in an embodiment of the present invention. Detailed Implementation

[0042] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0043] See Figure 1 This is a flowchart of an industrial material feeding anomaly identification method based on visual flow detection provided in Embodiment 1 of the present invention. Figure 1 As shown, a method for identifying industrial material feeding anomalies based on visual flow detection may include:

[0044] Step S1: Acquire and preprocess the image sequence of the industrial feed port to obtain an image sequence for optical flow calculation.

[0045] The implementation of this invention first requires acquiring continuous image data of the feeder outlet of microelectronic components on the SMT production line. Specifically, an industrial area array camera is fixedly installed directly above the feeder outlet, and a coaxial LED light source is used to ensure clear and uniform illumination for image acquisition. The camera continuously acquires image sequences of the feeder outlet at a preset frame rate, with a resolution of 640×480 pixels and a frame rate of 100 frames per second. The acquired image sequence is an 8-bit grayscale image. By performing Gaussian filtering on the acquired image sequence with a 3×3 kernel, an image sequence for optical flow calculation is obtained.

[0046] This completes the acquisition and preprocessing of image sequences from industrial feed inlets to obtain image sequences for optical flow calculation.

[0047] Step S2: Obtain the specular suppression factor by nonlinearly fusing the pixel brightness value with the time brightness change rate.

[0048] The fundamental problem this invention addresses stems from the failure of the constant brightness assumption relied upon by the existing LK optical flow method when dealing with specular highlights in SMT micro-component loading scenarios. The direct cause of this failure is that when the metal electrode surface of a component forms a specific reflection angle with the light source, the brightness value of the corresponding pixel in the image instantly saturates from a normal gray level to near pure white, resulting in an abnormally large temporal gradient. This huge temporal gradient does not reflect the actual motion of the object but is merely a product of the lighting effect. However, the least squares solution mechanism of the LK algorithm treats this temporal gradient caused by optical phenomena as a brightness change caused by extremely high-speed motion and attempts to compensate by calculating an equally abnormally large motion vector. This directly leads to the generation of a large number of erroneous pseudo-vectors in the highlight region and its neighborhood, which are the root cause of subsequent false alarms. Therefore, the first optimization step of this invention is to accurately identify these unreliable pixels caused by specular highlights before the LK algorithm performs its least squares solution, thus reducing their contribution weight in the solution process. Based on the analysis of scene data, these untrusted pixels exhibit two key characteristics: first, their brightness values ​​are extremely high, approaching saturation; second, their brightness changes at an extremely high rate over time, far exceeding the gradient caused by normal physical motion. This step constructs a specular suppression factor, non-linearly fusing these two characteristics to generate a weighted value that dynamically reflects the confidence level of each pixel's data. This factor directly impacts the least-squares objective function of the LK algorithm. For pixels identified as specular core regions, their specular suppression factor value approaches zero, rendering it almost unaffected by the final optical flow vector calculation, thus suppressing the generation of pseudo-vectors.

[0049] In summary, firstly, feature extraction processing is performed on the brightness values ​​and temporal brightness change rate data of pixels in the image sequence to obtain brightness and change features for confidence assessment. Specifically, for any target pixel at any target time in the image sequence across all time points, the gray value of the target pixel in the image at the target time is divided by the maximum gray value of the image sensor to obtain the brightness feature of the target pixel in the image at the target time. Next, the gradient of the image at the target time is calculated using the Sobel operator to obtain the gradient image at the target time. For target pixels in the gradient image at the target time, the squares of the horizontal and vertical gradients of the target pixel are added together and the square root is calculated. The result of the square root calculation is used as the gradient assessment of the target pixel. The gradient evaluation of the target pixel is normalized and added to a constant 1 to obtain the texture richness of the target pixel. The absolute value of the difference between the gradient evaluation of the target pixel in the image at the target time and the gradient evaluation of the target pixel at the previous time is used as the temporal partial derivative evaluation at the target time. The texture richness of each pixel in the image at the target time is used as a weight to calculate the temporal partial derivative evaluation of each pixel, and the median of the weighted calculation results of all pixels in the image at the target time is used as the adaptive time-varying scale factor at the target time. The result of dividing the temporal partial derivative evaluation of the target pixel in the image at the target time by the adaptive time-varying scale factor at the target time is used as the change feature of the target pixel in the image at the target time.

[0050] After obtaining the brightness features and variation features for confidence assessment, the brightness features and variation features are further fused nonlinearly to obtain the highlight suppression factor corresponding to the pixel. Specifically, for any target pixel at any target time in the image sequence of all time times, the negative of the result of multiplying the brightness feature and variation feature of the target pixel is subjected to exponential mapping with the natural constant as the base, and the corresponding mapping result is used as the highlight suppression factor of the target pixel.

[0051] In one implementation, assume the first The image at time t. The brightness value of each pixel The maximum brightness value of the image sensor is ;No. The image at time t. The time partial derivative of each pixel is ;No. The adaptive time-varying scaling factor in the image at time is: Then the first The image at time t. The expression for calculating the specular suppression factor of each pixel is:

[0052]

[0053] in, Indicates the first The image at time t. Highlight suppression factor per pixel; Indicates the first The image at time t. The brightness value of a pixel is, in this embodiment of the invention, the brightness value of a pixel is the gray value corresponding to the image after it has been converted into a grayscale image. This represents the maximum brightness value of the image sensor. In this embodiment of the invention, the maximum brightness value of the image sensor is 255. Indicates the first The image at time t. The time partial derivative of each pixel; Indicates the first Adaptive time-varying scaling factor in images at each time point; Denotes the natural constant e; It represents the absolute value.

[0054] It should be noted that the specular highlight suppression factor constructed in this invention couples the static brightness and dynamic changes of pixels through an exponential decay function, thereby achieving accurate identification and suppression of specular highlight regions. In the least squares solution process of the LK algorithm, pixels in highlight regions are outliers that cause the calculation results to deviate significantly from the actual motion. The specular highlight suppression factor of this invention dynamically assigns a confidence weight to each pixel before solving the problem, thus minimizing the influence of these outliers.

[0055] pass This structure ensures that the specular suppression factor only strongly suppresses pixels that simultaneously meet both conditions, which is precisely the unique data fingerprint of specular highlights. First, the normalized luminance term... The brightness of a pixel is quantified; the closer a pixel is to saturated white, the closer its value is to 1. Secondly, the normalized time variation term... This quantifies the severity of the brightness change at a given point. When the brightness change of a pixel far exceeds the overall average change level of the current frame, the value of this term will be significantly greater than 1. Only when a pixel is simultaneously very bright (brightness term close to 1) and its brightness change is extremely drastic (change term much greater than 1), the product of the two will be a large value, causing the output value of the function to quickly approach 0, achieving strong suppression of that point. This design avoids misjudging two common situations: first, for a bright but stable object (such as a white component), its brightness value will be very small, causing the specular suppression factor to remain close to 1, and its effective motion information is preserved; second, for a dark object that suddenly enters the field of view, its temporal partial derivative may be large, but because its brightness value is small, the decay of the specular suppression factor is also very limited. In addition, the adaptive temporal scaling factor of this invention needs an accurate benchmark to characterize the normal brightness change level in the current scene when determining whether the brightness change of a pixel is drastic. Directly performing undifferentiated statistical calculations on the brightness temporal changes of all pixels in the entire image would have limitations in this specific application scenario. This is because the physical sources of brightness changes in images from SMT loading scenarios are heterogeneous: brightness changes occurring at the edges and surface textures of micro-components are mainly caused by physical motion and contain signals with valid information; while brightness changes occurring in flat areas such as the background conveyor belt or smooth surface of components are more attributable to sensor noise or the initial disturbance of specular highlights, which are invalid interference signals. Indiscriminate statistics will mix these two types of signals, causing the calculated adaptive time-varying scale factor to be contaminated by interference signals and thus inaccurate. To solve this problem, this invention proposes an adaptive time-varying scale factor calculation method based on spatial gradient gating. This method is based on the fact that the effective time changes generated by object motion have a spatial correlation with regions in the image with significant spatial gradients (i.e., textures or edges). Therefore, this invention uses the magnitude of the spatial gradient as a weight to... A weighted median is calculated. This weight acts as a content filter: pixels located at the edge of elements and with rich textures have higher weight values, and their corresponding median values ​​are lower. It also contributes significantly to the calculation of adaptive time-varying scale factors; conversely, for pixels in flat regions, its weight value is lower, and its corresponding... The impact on computation is effectively mitigated. The adaptive time-varying scaling factor obtained through this configuration is no longer a simple statistical value of the inter-frame differences of the entire image, but a physical quantity that can more stably characterize the typical brightness change rate generated by moving objects in the scene. This provides a more accurate benchmark for judging whether brightness changes are abnormal, enabling the specular suppression factor to more reliably identify and suppress optical artifacts.

[0056] Thus, the specular suppression factor was obtained by nonlinearly fusing pixel brightness values ​​with the rate of change of brightness over time.

[0057] Step S3: Obtain the gradient structure confidence factor by performing distribution analysis on the eigenvalues ​​of the local gradient structure tensor.

[0058] In step S2, this invention addresses the problem of pseudo-vectors directly generated by the specular highlight core region (i.e., the brightness saturation region) through a highlight suppression factor. However, a problem remains after applying the highlight suppression factor: gradient distortion caused by the edge region where the highlight meets the component body. In this edge transition region, the brightness value of the pixel may not yet have reached saturation, thus the suppression effect of the highlight suppression factor will be correspondingly weakened. However, this region contains a spatial brightness gradient with high intensity and simple structure formed by the highlight phenomenon. The LK optical flow algorithm relies on accurate spatial gradient information to construct the equation system. The pseudo-gradient generated by the lighting effect usually points unidirectionally to the highlight center, and its intensity is far greater than the gradient generated by the actual surface texture of the object. When the LK algorithm performs calculations in these highlight edge regions, it will be misled by this high-intensity pseudo-gradient. Even if the value of the time partial derivative is the least squares solution of the normal algorithm, it will be forcibly pulled in this wrong direction, thus calculating a motion vector that does not match the actual motion of the object and points towards or away from the highlight center. This phenomenon can be understood as the algorithm tracking the movement of the bright spot itself, rather than the movement of the physical entity carrying the bright spot.

[0059] Therefore, after suppressing the specular core region, this invention further distinguishes between two completely different types of high gradients: one is a reliable gradient generated by the real texture of the object's surface (such as silkscreen characters or material boundaries); the other is an unreliable gradient with a single structure generated by abrupt changes in illumination (spectral edges). This step constructs a gradient structure confidence factor to quantify the gradient distribution structure in the neighborhood of each pixel, thereby assigning a confidence score to the gradient information of each pixel. This factor will work synergistically with the specular suppression factor to jointly adjust the weights in the LK algorithm solution process, thereby achieving precise suppression of pseudo-gradients in the specular edge region.

[0060] In summary, firstly, local neighborhood tensor quantization is performed on the spatial gradient data of pixels in the image sequence to obtain the local gradient structure tensor feature values. Specifically, the neighborhood range of each pixel is set. In this embodiment of the invention, the neighborhood range of each pixel is set as follows: For any target pixel in the image at any target time, the square of the horizontal gradient of the target pixel is taken as the top-left element of the structure matrix, the result of multiplying the horizontal gradient and the vertical gradient of the target pixel is taken as the top-right and bottom-left elements of the structure matrix, and the square of the vertical gradient of the target pixel is taken as the bottom-right element of the structure matrix. The structure tensor of the target pixel is obtained by summing the structure matrices of all pixels in the neighborhood of the target pixel. Eigenvalue analysis is performed on the structure tensor of the target pixel to obtain two eigenvalues ​​of the structure tensor of the target pixel. The eigenvalue with the larger value is taken as the first eigenvalue of the target pixel, and the eigenvalue with the smaller value is taken as the second eigenvalue of the target pixel.

[0061] After obtaining the local gradient structure tensor feature values ​​of the pixels, the morphology and intensity of the local gradient structure tensor feature values ​​are further differentiated to obtain the gradient anisotropy and significance gating index. Specifically, for any target pixel in the image at any target time, the result of subtracting the first feature value and the second feature value of the target pixel is used as the numerator, and the result of adding the first feature value and the second feature value of the target pixel is used as the denominator. The corresponding fraction is used as the anisotropy of the target pixel. The median of the first feature values ​​of all pixels in the image at the target time is used as the adaptive gradient intensity scaling factor of the image at the target time. The result of dividing the first feature value of the target pixel by the adaptive gradient intensity scaling factor of the image at the target time is mapped using the hyperbolic tangent function, and the corresponding mapping result is used as the significance gating index of the target pixel.

[0062] After obtaining the anisotropy and saliency gating index of the pixel, the gradient structure confidence factor is obtained by jointly weighting the gradient anisotropy and saliency gating index. Specifically, for any target pixel in the image at any target time, the negative of the product of the anisotropy and saliency gating index of the target pixel is subjected to an exponential mapping with the natural constant as the base, and the corresponding mapping result is used as the gradient structure confidence factor of the target pixel at the target time.

[0063] In one implementation, assume the first The image at time t. The first feature value of each pixel is ;No. The image at time t. The second feature value of each pixel is ;No. The adaptive gradient intensity scaling factor of the image at time is Then the first The image at time t. The expression for calculating the gradient structure confidence factor of each pixel is:

[0064]

[0065] in, Indicates the first The image at time t. Gradient structure confidence factor for each pixel; Indicates the first The image at time t. The first feature value of each pixel; Indicates the first The image at time t. The second feature value of each pixel; Indicates the first The adaptive gradient intensity scaling factor of the image at each time step.

[0066] It should be noted that the gradient structure confidence factor constructed in the above formula analyzes the structure tensor of the LK algorithm to determine the reliability of the gradient information source. The structure tensor has two eigenvalues. and It describes the gradient distribution pattern within the local neighborhood. The strength of the strongest gradient The gradient intensity corresponding to the direction orthogonal to it.

[0067] The exponential part of the gradient structure confidence factor consists of two multiplied sub-terms, which evaluate the gradient reliability from the dimensions of gradient shape and gradient strength, respectively. The first sub-term... Anisotropy is used to quantify the shape of the gradient distribution. For a sharp edge formed by illumination in an ideal state, the gradient direction is highly unidirectional. Much larger ( (Close to 0), the value of this term approaches 1. However, for a region formed by real corner points or complex textures on an object's surface, the gradient is distributed in multiple directions. and The values ​​are all relatively large and close in magnitude, and the value of this term approaches 0.

[0068] Second sub-item This is a significance-gated method used to assess the degree of anomaly in gradient strength. It identifies locally strongest gradients. Compared with the baseline gradient strength calculated using the global median Comparison is needed only when the gradient strength of a region significantly exceeds the general level of the scene (i.e., Much larger The output of the tanh function will saturate and approach 1 only after these two terms are multiplied. For gradients within the normal range of intensity, this term will be very small. The gradient structure confidence factor will only sharply approach 0 when a pixel simultaneously satisfies both the conditions of highly anisotropic gradient shape (uniform shape) and extremely significant gradient intensity (abnormal intensity), due to the large negative exponent, thus strongly suppressing the gradient at that point. This precisely describes the data characteristics of highlight edge regions. For normal texture edges on an object, although their anisotropy may also be high, their gradient intensity... The gradients are not significantly significant, and the significance gate term is very small, keeping the overall gradient structure confidence factor close to 1, thus protecting this useful gradient information from being erroneously suppressed. With this configuration, the present invention can effectively distinguish between reliable gradients generated by real physical structures and unreliable gradients generated by illumination effects, solving the gradient distortion problem in specular edge regions and further improving the accuracy of optical flow field calculations.

[0069] Thus, the distribution analysis of the eigenvalues ​​of the local gradient structure tensor was completed, and the confidence factor of the gradient structure was obtained.

[0070] Step S4: The dynamic spatiotemporal confidence weight is obtained by jointly weighting the specular suppression factor and the gradient structure confidence factor.

[0071] After obtaining the specular suppression factor and gradient structure confidence factor, for any target pixel in the image at any target time, the result of multiplying the specular suppression factor and gradient structure confidence factor of the target pixel is used as the dynamic spatiotemporal confidence weight of the target pixel.

[0072] Step S5: By performing weighted optical flow calculation and variance threshold discrimination on the dynamic spatiotemporal confidence weights, the abnormal feeding status identification result is obtained.

[0073] After obtaining the dynamic spatiotemporal confidence weights of the pixels, these optimized weights are multiplied and weighted by the least squares objective function of the standard LK optical flow algorithm. For each pixel in the image, its optical flow vector is obtained by solving the weighted least squares problem. By calculating the optical flow vectors of all pixels in the image, a dense optical flow field optimized by the method of this invention is obtained.

[0074] Based on this optimized dense optical flow field, abnormal feeding conditions can be identified. In this embodiment, the identification threshold is determined by offline calibration of data from the production line under normal operating conditions. The specific calibration process is as follows:

[0075] During production line debugging or initial operation, an image sequence covering at least 10,000 frames of a material loading process that needs to be confirmed as completely normal is acquired. The optimized optical flow calculation method proposed in this invention is applied to this sequence to obtain a time series of optical flow variance.

[0076] Subsequently, statistical analysis was performed on the variance sequence under normal conditions. The 5th and 95th quantiles of the sequence were calculated. The 5th quantile was set as the lower threshold, and the 95th quantile as the upper threshold. During actual operation, the real-time calculated optical flow variance was compared with these calibrated thresholds. If the variance value was lower than the lower threshold, it was judged as a material blockage or empty material anomaly; if the variance value was higher than the upper threshold, it was judged as an unstable material feeding anomaly.

[0077] After the judgment process is completed, when an abnormal signal is output, it is transmitted to the central control system of the production line (such as a PLC or industrial PC). The control system executes corresponding automated processing logic based on the specific type of abnormality received. Specifically, if the judgment result is a blockage or empty material, the control system immediately issues a command to suspend the downstream pick-and-place machine's material handling action and simultaneously illuminates the alarm light at the corresponding feeder location, prompting on-site operators to check and replenish materials. If the judgment result is unstable feeding, the control system first attempts to fine-tune the feeder's vibration frequency or drive voltage through program instructions to try to restore stable material flow. If the abnormal signal of unstable feeding is eliminated within a preset time, the production line continues to operate normally; if the abnormal signal persists, the system escalates the event to a shutdown alarm, notifying personnel to intervene for maintenance. The method proposed in this invention effectively filters out the pseudo-instability caused by specular highlights, and the system only reacts to genuine physical flow instability. This greatly reduces misjudgments and unnecessary production line stoppages caused by optical artifacts, significantly improving the automation efficiency and stability of the production line.

[0078] Thus, the identification results of abnormal feeding status are obtained by weighted optical flow calculation and variance threshold discrimination based on the dynamic spatiotemporal confidence weights.

[0079] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An industrial feeding abnormality recognition method based on visual flow detection, characterized in that, The method includes: Step S1: Obtain an image sequence for optical flow calculation by acquiring and preprocessing the image sequence of the industrial feed port; Step S2: Obtain the specular suppression factor by nonlinearly fusing the pixel brightness value with the time brightness change rate; Step S3: Obtain the gradient structure confidence factor by performing distribution analysis on the eigenvalues ​​of the local gradient structure tensor; Step S4: Obtain the dynamic spatiotemporal confidence weight by jointly weighting the specular suppression factor and the gradient structure confidence factor; Step S5: Obtain the results of abnormal feeding status identification by performing weighted optical flow calculation and variance threshold discrimination on the dynamic spatiotemporal confidence weights; The step of obtaining a highlight suppression factor by nonlinearly fusing pixel brightness values ​​and time-varying brightness change rates includes: performing feature extraction processing on the brightness values ​​and time-varying brightness change rate data of pixels in the image sequence to obtain brightness features and change features for confidence assessment; and performing nonlinear fusing processing on the brightness features and change features to obtain the highlight suppression factor corresponding to the pixel. The step of obtaining a gradient structure confidence factor by performing distribution analysis on the feature values ​​of the local gradient structure tensor includes: obtaining the feature values ​​of the local gradient structure tensor by performing local neighborhood tensor quantization on the spatial gradient data of pixels in the image sequence; obtaining the gradient anisotropy degree and significance gating index by performing morphological and intensity differentiation processing on the feature values ​​of the local gradient structure tensor; and obtaining the gradient structure confidence factor by performing joint weight processing on the gradient anisotropy degree and significance gating index. The method of obtaining dynamic spatiotemporal confidence weight by jointly weighting the specular suppression factor and the gradient structure confidence factor includes: for any target pixel in the image at any target time, the result of multiplying the specular suppression factor and the gradient structure confidence factor of the target pixel is used as the dynamic spatiotemporal confidence weight of the target pixel.

2. The industrial feeding abnormality recognition method based on visual flow detection according to claim 1, characterized in that, The process of acquiring and preprocessing image sequences from industrial feed inlets to obtain image sequences for optical flow calculation includes: An industrial area array camera is fixedly installed directly above the feeder outlet, and a coaxial LED light source is used to ensure clear and uniform illumination for image acquisition. The camera continuously acquires image sequences from the feeder outlet at a preset frame rate, with a resolution of 640×480 pixels and a frame rate of 100 frames per second. The acquired image sequences are 8-bit grayscale images. The acquired image sequences are processed by a 3×3 pixel Gaussian filter to obtain an image sequence for optical flow calculation.

3. The industrial feeding abnormality recognition method based on visual flow detection according to claim 1, characterized in that, The process involves extracting features from the brightness values ​​and time-varying brightness rates of pixels in an image sequence to obtain brightness features and change features for confidence assessment, including: For any target pixel at any target time in the image sequence of all time times, the gray value of the target pixel in the image at the target time is divided by the maximum gray value of the image sensor as the brightness feature of the target pixel in the image at the target time. The gradient image of the target moment is obtained by performing gradient calculation on the image of the target moment through a Sobel operator; for a target pixel point in the gradient image of the target moment, the square of the horizontal gradient of the target pixel point is added to the square of the vertical gradient, and square root calculation is performed to obtain a gradient evaluation of the target pixel point; the gradient evaluation of the target pixel point is normalized and added to a constant 1 to obtain a texture richness of the target pixel point; The absolute value of the difference between the gradient evaluation of the target pixel point in the image of the target moment and the gradient evaluation of the target pixel point in the previous moment is taken as a time partial derivative evaluation of the target moment; the texture richness of each pixel point in the image of the target moment is taken as a weight to perform weighted calculation on the time partial derivative evaluation of each pixel point, and the median of the weighted calculation results of all the pixel points in the image of the target moment is taken as an adaptive time change scale factor of the target moment; The calculation result of the division of the time partial derivative evaluation of the target pixel point in the image of the target moment by the adaptive time change scale factor of the target moment is taken as a change feature of the target pixel point in the image of the target moment.

4. The industrial feeding abnormality recognition method based on visual flow detection according to claim 1, characterized in that, The high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing on the brightness feature and the change feature, and the high light suppression factor corresponding to the pixel point is obtained by performing nonlinear fusion processing ​ 5. The industrial feeding abnormality recognition method based on visual flow detection according to claim 1, characterized in that, ​ ​ 6. The industrial feeding abnormality recognition method based on visual flow detection according to claim 1, characterized in that, ​ ​ The median of the first eigenvalues of all pixel points in the target time instant image is taken as an adaptive gradient intensity scale factor of the target time instant image; the first eigenvalue of the target pixel point is divided by the calculation result of the adaptive gradient intensity scale factor of the target time instant image, and the mapping result obtained by hyperbolic tangent function mapping is taken as the saliency gating index of the target pixel point.

7. The industrial feeding abnormality recognition method based on visual flow detection according to claim 1, characterized in that, The gradient structure confidence factor is obtained by jointly weighting the gradient anisotropy degree and the saliency gating index, including: For any target pixel point in any target time instant image, the opposite number of the calculation result of the anisotropy degree of the target pixel point multiplied by the saliency gating index is exponentially mapped with a natural constant as the base number, and the mapping result obtained is taken as the gradient structure confidence factor of the target pixel point of the target time instant.

Images