Image processing-based material loading mistake proofing detection method

Abstract

The application discloses a material loading mistake-proofing detection method based on image processing and relates to the technical field of industrial automation vision detection, which comprises the following steps: acquiring a multi-view image sequence of a material loading area and extracting a key frame set; decomposing visual features of the key frame image to obtain surface texture features, geometric contour features and spatial pose features; comparing the surface texture features and the geometric contour features with a standard template in multiple levels to generate a preliminary difference feature mapping; based on the mapping and the spatial pose feature fusion analysis, identifying candidate areas of material type abnormalities and pose deviation abnormalities; calling a pre-trained abnormality correlation network to cross-verify the two types of candidate areas to generate a comprehensive abnormality confidence score; and according to the score and a preset threshold, confirming the specific type and position of the material loading error. Through hierarchical feature fusion and abnormality correlation verification, the application improves the accuracy and reliability of material mistake-proofing detection in complex industrial scenes.

Description

Technical Field

[0001] This invention belongs to the field of industrial automation visual inspection technology, specifically a material feeding error prevention and detection method based on image processing. Background Technology

[0002] In industrial automated production lines, the accuracy and consistency of material feeding are crucial for ensuring product quality and production efficiency. Currently, machine vision-based error-proofing detection technology is widely used. This technology primarily uses cameras to capture images of materials, extracting single or a few features such as color, texture, or contour, and directly comparing them with a preset standard template to determine if there are errors or misalignments in the material. This type of method achieves a certain degree of automated inspection, replacing traditional manual visual inspection.

[0003] Existing conventional technical solutions have shortcomings. In complex and realistic industrial scenarios, materials may simultaneously exhibit multiple complex issues such as mixed types, batch texture differences, and misaligned placement postures. Simple feature template direct comparison methods extract features with limited dimensions or coarse fusion methods, making it difficult to effectively distinguish feature fluctuations caused by lighting changes and background interference from genuine material defects or type errors, resulting in high false alarm and false negative rates. Furthermore, for material type errors and installation posture errors, which are often closely related anomalies, conventional technologies often use independent detection models or judgment processes, lacking analysis of their inherent correlation. When a material's geometry resembles a standard part due to a type error, or when a correct material's local texture mismatches with the standard template due to a significant posture shift, the independently judged system is prone to making incorrect decisions and cannot cope with complex scenarios involving coexisting anomalies. Summary of the Invention

[0004] This invention aims to solve at least one of the technical problems existing in the prior art;

[0005] Therefore, this invention proposes a material feeding error prevention detection method based on image processing, including:

[0006] Acquire a multi-view image sequence of the material feeding area within a preset time interval, and extract a set of keyframe images from it;

[0007] Visual feature decomposition processing is performed on the keyframe image set to obtain surface texture feature data, geometric contour feature data and spatial pose feature data of the material;

[0008] The surface texture feature data and geometric contour feature data are compared with the pre-stored standard material feature templates at multiple levels to generate a preliminary difference feature mapping map.

[0009] Based on the preliminary difference feature mapping map and the spatial pose feature data, a fusion analysis is performed to identify candidate regions for abnormal material types and candidate regions for abnormal pose deviations.

[0010] The pre-trained anomaly association network is invoked to perform cross-validation between the material type anomaly candidate region and the pose offset anomaly candidate region to generate a comprehensive anomaly confidence score.

[0011] Based on the comprehensive anomaly confidence score and the preset judgment threshold, the specific type and location information of the material feeding error are confirmed.

[0012] Further, the step of acquiring a multi-view image sequence of the material feeding area within a preset time interval and extracting a set of keyframe images from it includes:

[0013] Multiple synchronized industrial cameras, arranged around the material loading area and spatially calibrated, are used to simultaneously trigger image acquisition at preset time intervals, thereby obtaining a sequence of synchronized images covering the entire material loading area.

[0014] The image sequences from each camera are preprocessed, including brightness equalization, distortion correction, and timestamp alignment, to generate standardized multi-view image sequences;

[0015] The inter-frame difference method or optical flow method is used to calculate the pixel-level changes or motion vectors between consecutive frames in each viewpoint image sequence, and the frame index where the image content has changed significantly is detected based on the preset change threshold;

[0016] Based on the detected frame index, initial keyframes are extracted from each viewpoint image sequence, and the structural similarity index between the initial keyframes is calculated.

[0017] The initial keyframes are deredundant by clustering algorithm. Frames with structural similarity index higher than the merging threshold are grouped together, and the frame with the highest clarity in each group is selected as the representative frame to form an optimized keyframe subset for each viewpoint.

[0018] Synchronize and align the optimized keyframe subsets from all perspectives based on timestamps, and organize them according to perspective number and time order to construct a multi-view keyframe image set.

[0019] Further, the step of performing visual feature decomposition processing on the keyframe image set to obtain surface texture feature data, geometric contour feature data, and spatial pose feature data of the material includes:

[0020] The set of keyframe images is input into a pre-trained convolutional neural network for multi-scale feature extraction;

[0021] Local texture descriptors that can characterize the microstructure of the material surface are extracted from the mid-layer feature map of the convolutional neural network to form the surface texture feature data;

[0022] Global contour descriptors that can characterize the overall shape of the material are extracted from the high-level feature map of the convolutional neural network to form the geometric contour feature data;

[0023] Using a binocular vision computing model, stereo matching and 3D reconstruction are performed on multi-view images in the keyframe image set to calculate the coordinates and orientation of the material in 3D space and generate the spatial pose feature data.

[0024] Further, the step of performing multi-level feature comparison between the surface texture feature data, geometric contour feature data, and pre-stored standard material feature templates to generate a preliminary difference feature mapping map includes:

[0025] The surface texture feature data is compared with the standard texture features in the standard material feature template on a pixel-by-pixel basis.

[0026] The geometric contour feature data is compared with the standard contour features in the standard material feature template to calculate the shape distance measurement.

[0027] The similarity calculation result and the shape distance metric calculation result are both normalized to the same dimension;

[0028] By weighted fusion of the normalized similarity calculation results and the shape distance metric calculation results, a two-dimensional difference intensity matrix is ​​generated;

[0029] The difference intensity matrix is ​​subjected to region clustering and boundary extraction to form the preliminary difference feature map labeled with the difference intensity level.

[0030] Furthermore, the fusion analysis based on the preliminary difference feature mapping map and the spatial pose feature data to identify candidate regions for abnormal material types and candidate regions for abnormal pose shifts includes:

[0031] Continuous regions with difference intensity exceeding the type anomaly threshold are selected from the preliminary difference feature map and used as initial material type anomaly regions;

[0032] The initial material type anomaly region is mapped to a three-dimensional spatial model composed of the spatial pose feature data;

[0033] In the three-dimensional spatial model, the Euclidean distance between the three-dimensional attitude angle of the material corresponding to the initial material type abnormal area and the preset standard attitude angle is calculated;

[0034] If the Euclidean angular distance exceeds the pose offset threshold, the region is determined to be a composite anomaly candidate region involving both type and pose.

[0035] If the Euclidean angular distance does not exceed the pose offset threshold, but the difference intensity exceeds another higher independent anomaly threshold, then the region is determined to be a simple candidate region for the material type anomaly.

[0036] For regions in the preliminary difference feature map where the difference intensity does not exceed the type anomaly threshold, but the Euclidean angular distance exceeds the pose offset threshold, they are determined to be simple pose offset anomaly candidate regions.

[0037] Furthermore, the pre-trained anomaly association network is invoked to cross-validate the candidate regions for material type anomalies and the candidate regions for pose offset anomalies, generating a comprehensive anomaly confidence score, including:

[0038] The difference intensity feature vector of the material type anomaly candidate region is input into the first sub-network of the anomaly association network, and the type anomaly confidence is output.

[0039] The pose angle distance feature vector of the candidate region of the pose offset anomaly is input into the second sub-network of the anomaly association network, and the pose anomaly confidence is output.

[0040] The confidence scores for the type anomalies and pose anomalies, along with the mapping vector between the initial material type anomaly region and the three-dimensional spatial model, are input into the third sub-network of the anomaly association network.

[0041] The third sub-network recalibrates and fuses the two confidence levels by analyzing the symbiotic relationship and causal logic between type anomalies and pose anomalies in spatial location.

[0042] Output a scalar value between zero and one as the comprehensive anomaly confidence score.

[0043] Furthermore, before acquiring the multi-view image sequence of the material feeding area within a preset time interval, the method further includes:

[0044] The steps for building a standard material feature template library include:

[0045] Collect multiple sets of high-precision image samples of standard materials under standard feeding posture;

[0046] For each set of high-precision image samples, perform visual feature decomposition processing to extract standard surface texture feature data and standard geometric contour feature data;

[0047] Statistical averaging and noise reduction are performed on multiple sets of standard surface texture feature data and standard geometric contour feature data to generate robust standard texture features and standard contour features.

[0048] The standard texture features, the standard contour features, and the three-dimensional coordinates and orientation information under the standard loading posture are packaged and stored as the standard material feature template.

[0049] Furthermore, the step of confirming the specific type and location information of the material feeding error based on the comprehensive anomaly confidence score and a preset judgment threshold includes:

[0050] The comprehensive anomaly confidence score is compared with a preset global judgment threshold;

[0051] If the overall anomaly confidence score is lower than the global judgment threshold, then the current material feeding is determined to be correct and no error has occurred.

[0052] If the comprehensive anomaly confidence score is higher than or equal to the global judgment threshold, then it is determined that there is a material feeding error;

[0053] Further compare the relationship between the confidence scores of the aforementioned types of anomalies and the confidence scores of pose anomalies and their corresponding local judgment thresholds;

[0054] If the confidence level of the type of anomaly exceeds its local judgment threshold, the error type is confirmed as a material type error, and the image coordinates of the candidate region of the material type anomaly are recorded as location information.

[0055] If the confidence level of the pose anomaly exceeds its local judgment threshold, the error type is confirmed as material pose error, and the coordinates of the candidate region of the pose offset anomaly in the three-dimensional space model are recorded as position information.

[0056] If both exceed their local judgment thresholds, the error type is confirmed as a compound error, and both types of location information are recorded simultaneously.

[0057] Furthermore, after confirming that the material loading error has occurred, the process also includes:

[0058] The steps for triggering real-time alarms and encapsulating error messages specifically include:

[0059] Based on the confirmed error type and location information, generate a structured error description text;

[0060] Bind the error description text, the timestamp that triggered the alarm, and the corresponding set of keyframe images or the abnormal area image extracted from them;

[0061] The bound data packet is sent to the production execution system or operator terminal via industrial communication protocol;

[0062] At the same time, an emergency stop or pause command is sent to the programmable logic controller that controls the loading robotic arm or conveyor line.

[0063] Further, the step of calculating the Euclidean distance between the three-dimensional attitude angle of the material corresponding to the initial material type abnormal region and the preset standard attitude angle in the three-dimensional spatial model includes:

[0064] The preset standard attitude angle is extracted from the standard material feature template. The preset standard attitude angle is defined in the world coordinate system of the three-dimensional space model in the form of a triplet of yaw angle, pitch angle and roll angle.

[0065] In the three-dimensional spatial model, for the material point cloud data corresponding to the initial material type anomaly area, the principal component analysis method is used to calculate its three principal directions, and the three principal directions are aligned with the world coordinate axis of the three-dimensional spatial model to solve the current three-dimensional attitude angle of the material corresponding to the initial material type anomaly area. The current three-dimensional attitude angle is also represented in the form of a triplet of yaw angle, pitch angle and roll angle.

[0066] The triplet of the current three-dimensional attitude angle and the triplet of the preset standard attitude angle are used as two three-dimensional vectors;

[0067] Calculate the Euclidean distance between the two three-dimensional vectors, and use the calculation result as the Euclidean angular distance, wherein the Euclidean distance is calculated by the square root of the sum of the squares of the differences between the corresponding components of the two vectors.

[0068] Compared with the prior art, the beneficial effects of the present invention are:

[0069] By performing visual feature decomposition on material images and strategically integrating surface texture, geometric contour features, and spatial pose features in a phased and guided manner, this approach departs from the conventional parallel or simple stitching processing mode of multi-feature methods. First, texture and contour features strongly correlated with the material's intrinsic properties are compared to generate preliminary difference regions. Then, spatial pose features are introduced to specifically verify and interpret these regions. This progressive fusion mechanism effectively isolates the influence of environmental interference and pose changes on the recognition of intrinsic features, resulting in a purer and more accurate judgment of anomalies in the material type itself. This improves the detection accuracy of intrinsic defects such as foreign objects and incorrect materials in complex backgrounds.

[0070] Instead of processing them independently, a pre-trained anomaly association network is invoked to perform cross-validation and association analysis on the separately identified candidate regions for material type anomalies and pose offset anomalies. By learning the symbiotic and causal patterns between the two types of anomalies from a large number of historical anomaly samples, this network can determine whether pose anomalies are caused by incorrect material types, or identify false positives where extreme pose offsets are misclassified as type anomalies. This cross-validation mechanism enables intelligent reasoning and integration of the two types of anomaly clues, transforming multiple local uncertainties into a global comprehensive confidence score, thereby reducing the system's misjudgment rate and enhancing the reliability and robustness of decision-making under complex anomaly conditions. Attached Figure Description

[0071] Figure 1 This is a flowchart illustrating the steps of the image processing-based material feeding error prevention and detection method described in this invention.

[0072] Figure 2 A flowchart for generating a preliminary differential feature map;

[0073] Figure 3 A pie chart showing the results of error-proofing inspection for automotive nuts;

[0074] Figure 4 A bar chart comparing the error-proofing detection thresholds for material feeding;

[0075] Figure 5 This is a distribution chart of alarm frequency and command type for each workstation. Detailed Implementation

[0076] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0077] See Figure 1The system acquires multi-view image sequences of the material feeding area within a preset time interval, extracts keyframe image sets from these sequences, performs visual feature decomposition on the keyframe image sets to obtain surface texture feature data, geometric contour feature data, and spatial pose feature data of the material. It then performs multi-level feature comparison between the surface texture feature data and geometric contour feature data and pre-stored standard material feature templates to generate a preliminary difference feature map. Based on the preliminary difference feature map and spatial pose feature data, it performs fusion analysis to identify candidate regions for material type anomalies and candidate regions for pose deviation anomalies. A pre-trained anomaly association network is then invoked to cross-validate these candidate regions, generating a comprehensive anomaly confidence score. Based on the comprehensive anomaly confidence score and a preset judgment threshold, the specific type and location information of the material feeding error are confirmed.

[0078] In one embodiment of the present invention, a multi-view image sequence of the material loading area within a preset time interval is acquired, and a set of key frame images is extracted from it. Multiple synchronous industrial cameras arranged around the material loading area and spatially calibrated are used, for example, four synchronous industrial cameras are deployed around the loading station on an automotive parts assembly line. Each camera is installed at a 30-degree angle interval and covers the entire view of the material. Image acquisition is triggered simultaneously at a preset time interval of 100 milliseconds to acquire a synchronous image sequence covering the entire view of the material loading area. The synchronous image sequence contains image frames acquired by each camera at the same timestamp to ensure the temporal consistency of the multi-view data. The image sequence of each camera is preprocessed, including brightness equalization, distortion correction, and timestamp alignment. Brightness equalization uses a histogram matching method to adjust the image brightness distribution. Distortion correction uses camera calibration parameters to mathematically correct lens distortion. Timestamp alignment is based on a hardware trigger signal to align the image frames of different cameras according to the acquisition time, generating a standardized multi-view image sequence.

[0079] In some embodiments, the inter-frame difference method or optical flow method is used to calculate the pixel-level changes or motion vectors between consecutive frames in each viewpoint image sequence. Based on a preset change threshold, the frame index where the image content changes significantly is detected. The preset change threshold is set according to the motion amplitude during the material feeding process. For example, when the image pixel change exceeds the threshold of 50 when the material enters the feeding area, the frame is marked as a candidate keyframe. The inter-frame difference method obtains the pixel-level changes by calculating the absolute difference of the pixel gray values ​​between adjacent frames. The optical flow method describes the motion pattern by calculating the motion vectors of pixels in the image sequence. Based on the detected frame index, the initial keyframe is extracted from each viewpoint image sequence, and the structural similarity index between the initial keyframes is calculated. The structural similarity index is used to quantify the degree of structural similarity between two frames.

[0080] It is understandable that the initial keyframes are deredundant through clustering algorithms, and frames with structural similarity indices higher than the merging threshold are grouped together. The merging threshold is set to 0.85, and the frame with the highest sharpness in each group is selected as the representative frame. Sharpness is evaluated by calculating the sum of image gradient magnitudes, forming an optimized keyframe subset for each viewpoint. The optimized keyframe subset reduces data redundancy and retains key visual information. The optimized keyframe subsets of all viewpoints are synchronized and aligned according to timestamps, and organized according to viewpoint number and time order to construct a multi-view keyframe image set. The multi-view keyframe image set serves as the input data for subsequent feature decomposition processing.

[0081] Optionally, when calculating the structural similarity index between the initial keyframes, a custom similarity metric formula can be used, expressed as follows:

[0082]

[0083] in: The structural similarity index represents the relationship between image A and image B. and Let represent the grayscale values ​​of the i-th pixel in image A and image B, respectively. and Let A and B represent the mean gray levels of image A and image B, respectively. This represents the total number of pixels in an image. The formula is based on normalized covariance and is used to evaluate the structural similarity between images.

[0084] In some embodiments, the pixel-level change calculation of the inter-frame difference method is achieved by comparing the grayscale difference of corresponding pixels in consecutive frames. The motion vector calculation of the optical flow method is based on the assumption of constant brightness and spatial consistency constraints. The preset change threshold is dynamically adjusted according to the specific application scenario. For example, in a high-speed material loading scenario, the threshold is increased to 70 to filter out minute movements. After detecting the frame index where the image content has changed significantly, the process of extracting the initial keyframe involves extracting the corresponding frame image from the original sequence. When calculating the structural similarity index between the initial keyframes, the above-mentioned custom similarity measurement formula is applied to each group of initial keyframes to obtain a similarity score.

[0085] Optionally, the clustering algorithm uses K-means clustering or hierarchical clustering to group initial keyframes with structural similarity indices higher than the merging threshold. Within each group, the frame with the highest sharpness is selected as the representative frame. Sharpness is evaluated by calculating the image gradient using the Sobel operator. The frame with the largest sum of gradient magnitudes is considered to have the highest sharpness. The keyframe subset is optimized to ensure that only the most representative image frame is retained for each viewpoint. When aligning according to timestamps, a linear interpolation method is used to handle small timestamp deviations, aligning representative frames from different viewpoints at the same time point. The final multi-view keyframe image set is stored in matrix form, with rows corresponding to viewpoint numbers and columns corresponding to time order.

[0086] In one embodiment of the present invention, see [reference] Figure 2 The keyframe image set is subjected to visual feature decomposition to obtain surface texture feature data, geometric contour feature data, and spatial pose feature data of the material. The surface texture feature data and geometric contour feature data are then compared with pre-stored standard material feature templates at multiple levels to generate a preliminary difference feature map. The keyframe image set is then input into a pre-trained convolutional neural network for multi-scale feature extraction. The pre-trained convolutional neural network uses a ResNet-50 model pre-trained on the ImageNet dataset, with its input size adjusted to match the image size in the keyframe image set. Multi-scale feature extraction is achieved by acquiring convolutional... The feature maps at different levels in the neural network are implemented by extracting local texture descriptors that can characterize the microstructure of the material surface from the mid-level feature maps of the convolutional neural network. The mid-level feature maps correspond to the third-stage output of ResNet-50. The local texture descriptors use scale-invariant feature transform descriptors or directional gradient histogram descriptors to form surface texture feature data. The global contour descriptors that can characterize the overall shape of the material are extracted from the high-level feature maps of the convolutional neural network. The high-level feature maps correspond to the fifth-stage output of ResNet-50. The global contour descriptors are represented by feature vectors after global average pooling to form geometric contour feature data.

[0087] In some embodiments, a binocular vision computing model is used to perform stereo matching and 3D reconstruction on multi-view images in a keyframe image set, calculate the coordinates and orientation of the material in 3D space, and generate spatial pose feature data. The binocular vision computing model uses a calibrated binocular camera pair, and the stereo matching uses a semi-global matching algorithm to calculate the disparity map. The 3D reconstruction calculates the 3D coordinates of the material surface points based on the disparity map and camera intrinsic and extrinsic parameters using the triangulation principle. The orientation of the material in 3D space is calculated by fitting the principal direction vector to the reconstructed material point cloud data. For example, in the scenario of loading a car engine cylinder block, the binocular vision computing model reconstructs the cylinder block point cloud and calculates its yaw angle, pitch angle, and roll angle relative to the standard attitude. These angle values, together with the 3D centroid coordinates, constitute the spatial pose feature data. Data comparison shows that the geometric contour feature data vector extracted based on ResNet-50 has a dimension of 2048, and the surface texture feature data consists of a 128-dimensional scale-invariant feature transformation descriptor.

[0088] It can be understood that the surface texture feature data is compared with the standard texture features in the standard material feature template by performing pixel-by-pixel similarity calculation. The standard texture features are pre-generated reference texture descriptors corresponding to the material to be inspected. The pixel-by-pixel similarity calculation is achieved by calculating the cosine similarity between the two descriptor vectors. The geometric contour feature data is compared with the standard contour features in the standard material feature template by performing shape distance measurement calculation. The standard contour features are pre-generated reference contour feature vectors corresponding to the material to be inspected. The shape distance measurement calculation is achieved by calculating the Euclidean distance between the two feature vectors. The similarity calculation results and the shape distance measurement calculation results are normalized to the same dimension. The normalization adopts the min-max normalization method to map the cosine similarity value and the Euclidean distance value to the interval [0, 1].

[0089] Optionally, a two-dimensional difference intensity matrix is ​​generated by combining the weighted fusion and normalized similarity calculation results with the shape distance metric calculation results. The weight coefficients of the weighted fusion are pre-set based on the contribution of surface texture and geometric contour to material recognition. For example, in the detection of metal parts, the geometric contour weight is set to 0.6, and the surface texture weight is set to 0.4. Each element value of the difference intensity matrix represents the comprehensive difference degree of the corresponding pixel in the image. Its calculation uses a custom difference metric formula, expressed as follows:

[0090]

[0091] in: The difference intensity matrix is ​​represented by the coordinates ( , The element value at position ) This represents the normalized surface texture similarity value. This represents the distance value of the normalized geometric contour shape. and Preset weighting coefficients representing surface texture and geometric contour respectively, and satisfying The difference intensity matrix is ​​subjected to region clustering and boundary extraction. The region clustering uses a density-based clustering algorithm to aggregate adjacent pixels with difference intensity values ​​higher than a threshold into connected regions. The boundary extraction uses an edge detection algorithm to outline the contour of each connected region, forming a preliminary difference feature map labeled with the difference intensity level. The preliminary difference feature map is visualized in the form of a heat map, where red areas represent high intensity differences and blue areas represent low intensity differences.

[0092] In some embodiments, the multi-level feature comparison process is embodied in the detection of automobile gear loading in a specific example. The standard material feature template stores the scale-invariant feature transformation descriptor of the standard gear and the 2048-dimensional contour feature vector. The cosine similarity between the surface texture feature data extracted from the current loading gear image and the standard texture feature is 0.92. The Euclidean distance between the geometric contour feature data and the standard contour feature is 45.7. After normalization, the values ​​are 0.92 and 0.32, respectively. After fusion according to the preset weight, the difference intensity matrix is ​​obtained. The value of the gear tooth shape missing region in the matrix is ​​significantly higher than that of other regions. After region clustering and boundary extraction, the abnormal region and its difference intensity level are clearly marked in the preliminary difference feature mapping map.

[0093] In one embodiment of the present invention, a specific method is described, which involves fusing and analyzing a preliminary difference feature map and spatial pose feature data to identify candidate regions for material type anomalies and candidate regions for pose shift anomalies; calling a pre-trained anomaly association network to cross-validate the candidate regions for material type anomalies and candidate regions for pose shift anomalies to generate a comprehensive anomaly confidence score; and calculating the Euclidean distance. From the preliminary difference feature map, continuous regions with difference intensity exceeding a type anomaly threshold are selected as initial material type anomaly regions. The type anomaly threshold is set to 0.65. A continuous region is defined as a set of pixels in the difference intensity matrix whose values ​​are greater than the threshold and whose eight neighbors are connected. In a specific example of nut loading on an automotive assembly line, the preliminary difference feature map shows that the average difference intensity of a certain region is 0.72, exceeding the type anomaly threshold. This region is marked as the initial material type anomaly region. The initial material type anomaly region is mapped to a three-dimensional spatial model composed of spatial pose feature data. The mapping process is achieved by establishing a one-to-one correspondence between the two-dimensional image coordinates of the initial material type anomaly region and the corresponding three-dimensional point cloud coordinates reconstructed through stereo vision in the three-dimensional spatial model.

[0094] In some embodiments, the Euclidean distance between the three-dimensional attitude angle of the material corresponding to the initial material type anomaly region and the preset standard attitude angle is calculated in the three-dimensional spatial model. The preset standard attitude angle is extracted from the standard material feature template. The preset standard attitude angle is defined in the form of a triplet of yaw angle, pitch angle, and roll angle in the world coordinate system of the three-dimensional spatial model. For example, the preset standard attitude angle is (0°, 0°, 0°). In the three-dimensional spatial model, the principal component analysis method is used to calculate the three principal directions of the material point cloud data corresponding to the initial material type anomaly region. The three principal directions are aligned with the world coordinate axis of the three-dimensional spatial model to solve for the current three-dimensional attitude angle of the material corresponding to the initial material type anomaly region. The current three-dimensional attitude angle is also represented in the form of a triplet of yaw angle, pitch angle, and roll angle. The Euclidean distance between the triplet of the current three-dimensional attitude angle and the triplet of the preset standard attitude angle is calculated as two three-dimensional vectors. The calculation result is used as the Euclidean distance. The formula for calculating the Euclidean distance is expressed as:

[0095]

[0096] in: This represents the scalar value of Euclidean angular distance. , , These represent the values ​​of the current three-dimensional attitude angles: yaw angle, pitch angle, and roll angle. , , These represent the yaw angle, pitch angle, and roll angle values ​​of the preset standard attitude angles, respectively.

[0097] It is understandable that if the Euclidean angular distance exceeds the pose offset threshold, the region is identified as a composite anomaly candidate region involving both type and pose. The pose offset threshold is set to 15 degrees. If the Euclidean angular distance does not exceed the pose offset threshold but the difference intensity exceeds another higher independent anomaly threshold, the region is identified as a simple material type anomaly candidate region. The independent anomaly threshold is set to 0.80. For regions in the preliminary difference feature mapping map where the difference intensity does not exceed the type anomaly threshold but the Euclidean angular distance exceeds the pose offset threshold, they are identified as simple pose offset anomaly candidate regions. (This applies to the same automotive nut loading process.) The data comparison in the example shows that when a region with a difference intensity of 0.75 and an Euclidean angular distance of 20 degrees is detected, this region is identified as a composite anomaly candidate region because the Euclidean angular distance exceeds the pose offset threshold of 15 degrees. Another region with a difference intensity of 0.70 does not exceed the independent anomaly threshold of 0.80, but the Euclidean angular distance is 18 degrees, which exceeds the pose offset threshold. This region is identified as a simple pose offset anomaly candidate region. A third region with a difference intensity of 0.82 exceeds the independent anomaly threshold, but the Euclidean angular distance is 10 degrees, which does not exceed the pose offset threshold. This region is identified as a simple material type anomaly candidate region.

[0098] Optionally, a pre-trained anomaly association network is invoked to perform cross-validation on the candidate regions for material type anomalies and pose offset anomalies to generate a comprehensive anomaly confidence score. The difference intensity feature vector of the candidate regions for material type anomalies is input into the first sub-network of the anomaly association network to output the type anomaly confidence score. The difference intensity feature vector is a fixed-length vector formed by mean pooling the difference intensity values ​​of all pixels in the candidate regions for material type anomalies. The first sub-network is a three-layer fully connected neural network. The pose angular distance feature vector of the candidate regions for pose offset anomalies is input into the second sub-network of the anomaly association network to output the pose anomaly confidence score. The pose angular distance feature vector is composed of the Euclidean angular distance value and the difference of the three angular components constituting the Euclidean angular distance. The second sub-network is a fully connected neural network with a structure symmetrical to the first sub-network. The type anomaly confidence score, the pose anomaly confidence score, and the mapping relationship vector between the initial material type anomaly region and the three-dimensional spatial model are input into the third sub-network of the anomaly association network. The mapping relationship vector describes the coordinate transformation parameters between the two-dimensional anomaly region and the three-dimensional point cloud.

[0099] In some embodiments, the third sub-network of the anomaly association network recalibrates and fuses the two confidence scores by analyzing the symbiotic relationship and causal logic between type anomalies and pose anomalies in spatial location. The third sub-network contains an attention mechanism module to evaluate the association weight between the type anomaly confidence score and the pose anomaly confidence score, and outputs a scalar value between zero and one as the comprehensive anomaly confidence score. During the training phase, the anomaly association network is trained in a supervised manner using a dataset containing known type errors, pose errors, and composite error samples. The loss function is cross-entropy loss. In a specific implementation, for the previously determined composite anomaly candidate region, its type anomaly confidence score is 0.88, and its pose anomaly confidence score is 0.91. The mapping relationship vector describes that the region occupies 150x150 pixels in the image and corresponds to a point cloud block with a length, width, and height of 30mm x 30mm x 15mm in three-dimensional space. The comprehensive anomaly confidence score output by the third sub-network after processing is 0.93.

[0100] It is understandable that the principal component analysis method in Euclidean angular distance calculation specifically involves calculating the covariance matrix of the material point cloud data corresponding to the initial material type anomaly region, solving for the eigenvalues ​​and eigenvectors of the covariance matrix, and the eigenvectors corresponding to the three largest eigenvalues ​​being the three principal directions of the point cloud. These three principal direction vectors are then used to calculate the rotation matrix with the world coordinate system coordinate axis vectors, thereby solving for the yaw angle, pitch angle, and roll angle. The triplet of the preset standard attitude angle and the current three-dimensional attitude angle are both expressed in degrees. When calculating the Euclidean angular distance, the angle values ​​are directly used for vector operations. The specific values ​​of the pose offset threshold and the type anomaly threshold can be configured and adjusted according to the material type and the material feeding accuracy requirements.

[0101] See Figure 3 This is a pie chart used to display the results of error-proofing detection for automotive nuts, visually presenting the distribution of different anomaly types. This chart represents the final statistical step in the error-proofing detection process, helping production managers quickly identify high-incidence anomaly types and optimize detection strategies. It clearly shows the percentages of "no anomalies," "compound anomalies," "simple type anomalies," and "simple positional anomalies," allowing managers to quickly assess the overall stability of the current feeding process. The high percentage of "no anomalies" directly verifies the reliability of the error-proofing detection scheme, providing data support for subsequent process upgrades or the introduction of new materials. Fluctuations in the percentage of anomaly types can serve as early warning signals for production risks. By combining historical data on percentage changes, the root cause of anomalies can be traced.

[0102] In one embodiment of the present invention, the steps of constructing a standard material feature template library and confirming the specific type and location information of material loading errors based on a comprehensive anomaly confidence score and a preset judgment threshold include: collecting multiple sets of high-precision image samples of standard materials in standard loading postures; for example, collecting 100 sets of high-precision image samples of the front and side of resistors in an electronic component mounting scenario; the high-precision image samples are captured by an industrial camera with a resolution of 2048x2048 under uniform lighting conditions; and performing visual feature decomposition processing on each set of high-precision image samples to extract standard surface texture feature data and standard geometric contour feature data. The feature decomposition process uses the same pre-trained convolutional neural network and binocular vision computing model as the feature decomposition process. It performs statistical averaging and noise reduction on multiple sets of standard surface texture feature data and standard geometric contour feature data to generate robust standard texture features and standard contour features. The statistical averaging calculates the arithmetic mean of all sample feature vectors. The noise reduction process uses median filtering to remove outliers from the feature vectors. The standard texture features, standard contour features, and three-dimensional coordinates and orientation information under the standard loading posture are packaged and stored as standard material feature templates. The standard material feature templates are stored in the form of structured database records, and each record contains material number, feature vector, and pose parameters.

[0103] In some embodiments, the process of confirming the specific type and location information of a material loading error based on a comprehensive anomaly confidence score and a preset judgment threshold begins by comparing the comprehensive anomaly confidence score with a preset global judgment threshold, which is set to 0.75. If the comprehensive anomaly confidence score is lower than the global judgment threshold, the current material loading is determined to be correct and no error has occurred. If the comprehensive anomaly confidence score is higher than or equal to the global judgment threshold, a material loading error is determined to exist. Further comparison is made between the type anomaly confidence score and the pose anomaly confidence score and their corresponding local judgment thresholds. The type anomaly local judgment threshold is set to 0.70, and the pose anomaly local judgment threshold is set to 0.65. If the type anomaly confidence score exceeds the type anomaly local judgment threshold, the error type is confirmed as a material type error, and the image coordinates of the material type anomaly candidate region are recorded as location information. If the confidence score for pose anomaly exceeds the local threshold for pose anomaly determination, the error type is confirmed as a material pose error, and the coordinates of the candidate region for pose offset anomaly in the 3D spatial model are recorded as location information. If both the confidence score for type anomaly and the confidence score for pose anomaly exceed their corresponding local thresholds, the error type is confirmed as a composite error, and both image coordinates and 3D coordinates are recorded as location information. In the specific example of loading automotive gearbox gears, the comprehensive anomaly confidence score obtained from a single detection is 0.82, which exceeds the global determination threshold of 0.75; the confidence score for type anomaly is 0.78, which exceeds the local determination threshold for type anomaly of 0.70; and the confidence score for pose anomaly is 0.60, which does not exceed the local determination threshold for pose anomaly of 0.65. Therefore, the error type is confirmed as a material type error, and the location information is recorded as the image bounding box coordinates (150, 200, 300, 350) of the gear anomaly region.

[0104] It is understandable that the number of sample sets collected when constructing a standard material feature template library directly affects the generalization ability of the template. In the scenario of metal fastener loading, 150 bolt image samples were collected. The extracted standard surface texture feature data is a 256-dimensional local binary pattern histogram vector, and the standard geometric contour feature data is a 1024-dimensional convolutional neural network global feature vector. After statistical averaging and noise reduction processing, the generated standard feature vector and standard pose information are stored in the template library. Data comparison shows that the feature matching stability of the template constructed using 150 samples is 12% higher than that of the template constructed using 50 samples in 1000 consecutive detections. The threshold setting in the error confirmation step can be calibrated according to the material size and the repeatability accuracy of the loading robot. For large-sized materials such as car doors, the global judgment threshold can be set to 0.80 to reduce false alarms caused by slight deformation. For small-sized precision components such as chips, the global judgment threshold can be set to 0.70 to improve detection sensitivity.

[0105] Optionally, the logical judgment for incorrect confirmation can be formally expressed using a decision formula:

[0106]

[0107] in: This represents the final error type output vector. Represents the unit step function. This represents the overall anomaly confidence score. This represents the preset global threshold. Indicates the confidence level of the type anomaly. This indicates the threshold for determining the type of local anomaly. Indicates the confidence level of pose abnormality. The output vector represents the threshold for local pose anomaly detection. The first element being 1 indicates an error, the second element being 1 indicates a material type error, and the third element being 1 indicates a material pose error. A value of [1,1,1] indicates a compound error. In practice, this formula is embedded in the conditional judgment module of the control software. See Table 1 for a comparison of judgment data in a specific example scenario.

[0108] Table 1: Data Table for Judging Errors in Automotive Parts Loading

[0109]

[0110] In some embodiments, the recording format of the location information is compatible with the subsequent processing system. The image coordinates record the coordinates of the smallest bounding rectangle vertex of the candidate region for material type anomalies in pixels. The three-dimensional coordinates record the centroid coordinates and principal axis direction of the candidate region for pose offset anomalies in the world coordinate system of the three-dimensional space model in millimeters. In the example, the three-dimensional coordinates of a composite error are recorded as (102.5, -33.2, 15.8), while the image coordinates are recorded as (220, 180, 310, 260). The standard material feature template library is updated and maintained by periodically collecting new samples and recalculating the feature mean to compensate for material batch differences or slow changes in ambient light.

[0111] It is understandable that the values ​​of the global and local decision thresholds are determined through optimization during the offline testing phase. The optimization process uses a verification dataset containing known correct and incorrect feeding situations. By adjusting the thresholds, the detection accuracy and false alarm rate meet production requirements. The error confirmation step is the final decision-making link in the detection process. Its input depends on the output of all the aforementioned steps, including the comprehensive anomaly confidence score, type anomaly confidence, and pose anomaly confidence.

[0112] See Figure 4This is a bar chart comparing the thresholds for error prevention detection in material handling. It visually illustrates the differences in the three judgment thresholds for different material types, reflecting the accuracy adaptation strategy of the detection system for different materials. The threshold gradients differ for different materials. For example, the difference between the global and type thresholds for automotive parts is 0.05, while the difference for precision machinery is 0.04. This indicates that the system dynamically adjusts the judgment sensitivity for different error types based on the visual complexity of the material. This chart is a core reference for configuring the threshold parameters of the detection system, helping engineers to accurately set thresholds for the process requirements of different materials, avoiding misjudgments or omissions caused by a "one-size-fits-all" approach. The level of the threshold directly corresponds to the production risk level. A high threshold for automotive parts means that the risk of production errors is greatest, requiring the most stringent detection; a low threshold for electronic components indicates that the error risk is relatively controllable. By comparing the threshold gradients for different materials, weaknesses in the detection algorithm can be identified. For example, if the pose anomaly threshold for a certain type of material is significantly low, the robustness of its 3D pose detection can be optimized accordingly.

[0113] In one embodiment of the present invention, after confirming a material feeding error, a real-time alarm is triggered and error information is encapsulated. A structured error description text is generated based on the confirmed error type and location information. The generation of the error description text follows a predefined template, which includes an error serial number, an error occurrence station identifier, an error type code, error location coordinates, and a timestamp field. The error description text, the timestamp that triggered the alarm, and the corresponding keyframe image set or an abnormal area image extracted from it are bound together. The binding process packages the text information, timestamp data, and image binary data into a data packet with a specific header. The data packet header includes the data packet length, version number, and data type identifier.

[0114] In some embodiments, the bound data packet is sent to the production execution system or operator terminal via an industrial communication protocol. The industrial communication protocol adopts the OPCUA protocol. The production execution system is deployed on a server in the workshop and subscribes to alarm topics. The operator terminal is a touch screen human-machine interface installed next to the workstation. After receiving the data packet, the production execution system parses it and records the error event in the database. At the same time, the operator terminal interface pops up a red warning box and displays the error description text and the captured abnormal area image. Data comparison shows that the end-to-end latency from error triggering to operator terminal interface update using the MQTT protocol is less than 500 milliseconds.

[0115] Simultaneously, an emergency stop or pause command is sent to the programmable logic controller (PLC) controlling the loading robotic arm or conveyor line. The command is sent via the industrial Ethernet protocol and includes the target device address and action code. The emergency stop command code "EMERGENCY_STOP" will immediately cut off the power supply to the robotic arm, while the pause command code "PAUSE" will stop the conveyor line and wait for manual intervention. In the example scenario of the electronic chip placement machine, when a chip pose error is detected, the system sends a pause command to the PLC controlling the movement of the placement head. The placement head immediately stops at its current position and waits for the operator to replace the chip.

[0116] It is understandable that the logic for generating structured error description text in the error message encapsulation step can be represented by a text construction formula:

[0117]

[0118] in: The function represents the generated structured error description text string. This represents string concatenation and formatting operations. This represents a unique error serial number string generated by the system. The identifier string representing the workstation where the error occurred. The string represents the error type code. This string represents the coordinates of the recorded error location. The formula represents the timestamp string that triggered the alarm, and defines the order of connection and delimiters for the various parts of the text.

[0119] Optionally, the process of extracting abnormal region images from the keyframe image set is based on the previously recorded image coordinate position information. The rectangular cropping function of the image processing library is used to extract the region of interest. The extracted image is encoded in JPEG format and bound to text information. The data packet structure adopts TLV format, namely type, length, and value, to ensure that the receiver can parse it correctly. The data packet sent to the production execution system will also have an additional priority field. For compound errors or error types involving security risks, a higher priority will be set to ensure that they are processed first.

[0120] In some embodiments, the instructions sent to the programmable logic controller (PLC) include not only action codes, but also prompt codes that require operator confirmation in the pause instructions. These codes trigger the PLC to illuminate the workstation warning lights. In a specific implementation, when the system detects a compound error at the gearbox valve body loading station, the data packet is sent to the production execution system for log recording and quality analysis. At the same time, an emergency stop instruction is sent to the PLC controlling the six-axis loading robot arm. The robot arm immediately stops moving and locks. The operator terminal screen simultaneously displays the error image of the valve body and the abnormal location marked in red in the three-dimensional space model.

[0121] It is understandable that the timestamp for triggering the alarm is accurate to the millisecond level and synchronized with the time source of the image acquisition system to ensure the accuracy of the fault time record. The data packet is sent using a reliable transmission mechanism, such as setting the service quality level to at least once in the MQTT protocol to prevent alarm information loss due to network fluctuations. The information sent to the operator terminal can be scaled proportionally to the bound image according to the terminal screen size to ensure that the image is fully visible. Before sending instructions to the programmable logic controller, the system will calculate the instruction check code, and the receiving programmable logic controller will verify the check code to ensure the integrity and security of the instruction.

[0122] See Figure 5 This is a distribution chart of alarm frequency and command type for each workstation, simultaneously displaying two key indicators: "total alarm frequency" and "average transmission delay" for five production workstations. This data is used to analyze the operational status and transmission performance of the workshop's material handling error prevention detection system. By comparing the differences in indicators across different workstations, the core cause of error triggering can be identified. If a workstation has a high alarm frequency but low delay, the problem stems from material identification accuracy rather than communication performance; conversely, if delay is high but alarm frequency is low, communication link optimization should be prioritized. The lack of a clear linear correlation between alarm frequency and transmission delay indicates that error triggering is primarily determined by the characteristics of the material itself, rather than directly caused by transmission performance. This chart helps production managers locate high-risk workstations and optimize detection algorithms or equipment parameters for nut handling workstations. Optimizing transmission performance involves investigating communication links for workstations with high delays to improve the real-time nature of alarm information. Prioritizing maintenance resources for workstations with high alarm frequency and high transmission delays reduces production risk.

[0123] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A material feeding error prevention detection method based on image processing, characterized in that, include: Acquire a multi-view image sequence of the material feeding area within a preset time interval, and extract a set of keyframe images from it; Visual feature decomposition processing is performed on the keyframe image set to obtain surface texture feature data, geometric contour feature data and spatial pose feature data of the material; The surface texture feature data and geometric contour feature data are compared with the pre-stored standard material feature templates at multiple levels to generate a preliminary difference feature mapping map. Based on the preliminary difference feature mapping map and the spatial pose feature data, a fusion analysis is performed to identify candidate regions for abnormal material types and candidate regions for abnormal pose deviations. The pre-trained anomaly association network is invoked to perform cross-validation between the material type anomaly candidate region and the pose offset anomaly candidate region to generate a comprehensive anomaly confidence score. Based on the comprehensive anomaly confidence score and the preset judgment threshold, the specific type and location information of the material feeding error are confirmed.

2. The material feeding error prevention and detection method based on image processing according to claim 1, characterized in that, The process of acquiring a multi-view image sequence of the material loading area within a preset time interval and extracting a set of keyframe images from it includes: Multiple synchronized industrial cameras, arranged around the material loading area and spatially calibrated, are used to simultaneously trigger image acquisition at preset time intervals, thereby obtaining a sequence of synchronized images covering the entire material loading area. The image sequences from each camera are preprocessed, including brightness equalization, distortion correction, and timestamp alignment, to generate standardized multi-view image sequences; The inter-frame difference method or optical flow method is used to calculate the pixel-level changes or motion vectors between consecutive frames in each viewpoint image sequence, and the frame index where the image content has changed significantly is detected based on the preset change threshold; Based on the detected frame index, initial keyframes are extracted from each viewpoint image sequence, and the structural similarity index between the initial keyframes is calculated. The initial keyframes are deredundant by clustering algorithm. Frames with structural similarity index higher than the merging threshold are grouped together, and the frame with the highest clarity in each group is selected as the representative frame to form an optimized keyframe subset for each viewpoint. Synchronize and align the optimized keyframe subsets from all perspectives based on timestamps, and organize them according to perspective number and time order to construct a multi-view keyframe image set.

3. The material feeding error prevention and detection method based on image processing according to claim 1, characterized in that, The step of performing visual feature decomposition processing on the keyframe image set to obtain surface texture feature data, geometric contour feature data, and spatial pose feature data of the material includes: The set of keyframe images is input into a pre-trained convolutional neural network for multi-scale feature extraction; Local texture descriptors that can characterize the microstructure of the material surface are extracted from the mid-layer feature map of the convolutional neural network to form the surface texture feature data; Global contour descriptors that can characterize the overall shape of the material are extracted from the high-level feature map of the convolutional neural network to form the geometric contour feature data; Using a binocular vision computing model, stereo matching and 3D reconstruction are performed on multi-view images in the keyframe image set to calculate the coordinates and orientation of the material in 3D space and generate the spatial pose feature data.

4. The material feeding error prevention detection method based on image processing according to claim 1, characterized in that, The step of performing multi-level feature comparison between the surface texture feature data, geometric contour feature data, and pre-stored standard material feature templates to generate a preliminary difference feature mapping map includes: The surface texture feature data is compared with the standard texture features in the standard material feature template on a pixel-by-pixel basis. The geometric contour feature data is compared with the standard contour features in the standard material feature template to calculate the shape distance measurement. The similarity calculation result and the shape distance metric calculation result are both normalized to the same dimension; By weighted fusion of the normalized similarity calculation results and the shape distance metric calculation results, a two-dimensional difference intensity matrix is ​​generated; The difference intensity matrix is ​​subjected to region clustering and boundary extraction to form the preliminary difference feature map labeled with the difference intensity level.

5. The material feeding error prevention and detection method based on image processing according to claim 4, characterized in that, The fusion analysis based on the preliminary difference feature mapping map and the spatial pose feature data identifies candidate regions for abnormal material types and candidate regions for abnormal pose shifts, including: Continuous regions with difference intensity exceeding the type anomaly threshold are selected from the preliminary difference feature map and used as initial material type anomaly regions; The initial material type anomaly region is mapped to a three-dimensional spatial model composed of the spatial pose feature data; In the three-dimensional spatial model, the Euclidean distance between the three-dimensional attitude angle of the material corresponding to the initial material type abnormal area and the preset standard attitude angle is calculated; If the Euclidean angular distance exceeds the pose offset threshold, the region is determined to be a composite anomaly candidate region involving both type and pose. If the Euclidean angular distance does not exceed the pose offset threshold, but the difference intensity exceeds another higher independent anomaly threshold, then the region is determined to be a simple candidate region for the material type anomaly. For regions in the preliminary difference feature map where the difference intensity does not exceed the type anomaly threshold, but the Euclidean angular distance exceeds the pose offset threshold, they are determined to be simple pose offset anomaly candidate regions.

6. The material feeding error prevention and detection method based on image processing according to claim 5, characterized in that, The pre-trained anomaly association network is invoked to cross-validate the candidate regions for material type anomalies and the candidate regions for pose offset anomalies, generating a comprehensive anomaly confidence score, including: The difference intensity feature vector of the material type anomaly candidate region is input into the first sub-network of the anomaly association network, and the type anomaly confidence is output. The pose angle distance feature vector of the candidate region of the pose offset anomaly is input into the second sub-network of the anomaly association network, and the pose anomaly confidence is output. The confidence scores for the type anomalies and pose anomalies, along with the mapping vector between the initial material type anomaly region and the three-dimensional spatial model, are input into the third sub-network of the anomaly association network. The third sub-network recalibrates and fuses the two confidence levels by analyzing the symbiotic relationship and causal logic between type anomalies and pose anomalies in spatial location. Output a scalar value between zero and one as the comprehensive anomaly confidence score.

7. The material feeding error prevention detection method based on image processing according to claim 1, characterized in that, Before acquiring the multi-view image sequence of the material loading area within a preset time interval, the method further includes: The steps for building a standard material feature template library include: Collect multiple sets of high-precision image samples of standard materials under standard feeding posture; For each set of high-precision image samples, perform visual feature decomposition processing to extract standard surface texture feature data and standard geometric contour feature data; Statistical averaging and noise reduction are performed on multiple sets of standard surface texture feature data and standard geometric contour feature data to generate robust standard texture features and standard contour features. The standard texture features, the standard contour features, and the three-dimensional coordinates and orientation information under the standard loading posture are packaged and stored as the standard material feature template.

8. The material feeding error prevention and detection method based on image processing according to claim 6, characterized in that, The step of confirming the specific type and location information of the material feeding error based on the comprehensive anomaly confidence score and the preset judgment threshold includes: The comprehensive anomaly confidence score is compared with a preset global judgment threshold; If the overall anomaly confidence score is lower than the global judgment threshold, then the current material feeding is determined to be correct and no error has occurred. If the comprehensive anomaly confidence score is higher than or equal to the global judgment threshold, then it is determined that there is a material feeding error; Further compare the relationship between the confidence scores of the aforementioned types of anomalies and the confidence scores of pose anomalies and their corresponding local judgment thresholds; If the confidence level of the type of anomaly exceeds its local judgment threshold, the error type is confirmed as a material type error, and the image coordinates of the candidate region of the material type anomaly are recorded as location information. If the confidence level of the pose anomaly exceeds its local judgment threshold, the error type is confirmed as material pose error, and the coordinates of the candidate region of the pose offset anomaly in the three-dimensional space model are recorded as position information. If both exceed their local judgment thresholds, the error type is confirmed as a compound error, and both types of location information are recorded simultaneously.

9. The material feeding error prevention detection method based on image processing according to claim 8, characterized in that, After confirming that the material loading error has occurred, the following steps are also included: The steps for triggering real-time alarms and encapsulating error messages specifically include: Based on the confirmed error type and location information, generate a structured error description text; Bind the error description text, the timestamp that triggered the alarm, and the corresponding set of keyframe images or the abnormal area image extracted from them; The bound data packet is sent to the production execution system or operator terminal via industrial communication protocol; At the same time, an emergency stop or pause command is sent to the programmable logic controller that controls the loading robotic arm or conveyor line.

10. The material feeding error prevention and detection method based on image processing according to claim 5, characterized in that, The step of calculating the Euclidean distance between the three-dimensional attitude angle of the material corresponding to the initial material type abnormal region and the preset standard attitude angle in the three-dimensional spatial model includes: The preset standard attitude angle is extracted from the standard material feature template. The preset standard attitude angle is defined in the world coordinate system of the three-dimensional space model in the form of a triplet of yaw angle, pitch angle and roll angle. In the three-dimensional spatial model, for the material point cloud data corresponding to the initial material type anomaly area, the principal component analysis method is used to calculate its three principal directions, and the three principal directions are aligned with the world coordinate axis of the three-dimensional spatial model to solve the current three-dimensional attitude angle of the material corresponding to the initial material type anomaly area. The current three-dimensional attitude angle is also represented in the form of a triplet of yaw angle, pitch angle and roll angle. The triplet of the current three-dimensional attitude angle and the triplet of the preset standard attitude angle are used as two three-dimensional vectors; Calculate the Euclidean distance between the two three-dimensional vectors, and use the calculation result as the Euclidean angular distance, wherein the Euclidean distance is calculated by the square root of the sum of the squares of the differences between the corresponding components of the two vectors.

Images