Manipulator grabbing abnormity real-time detection system integrated with visual identification

Abstract

The invention discloses a manipulator grabbing abnormity real-time detection system integrated with visual identification, and relates to the technical field of manipulator safety detection. The system comprises an image acquisition module, a feature point tracking module, a consistency calculation module, an event triggering module, a space-time positioning module, a map construction module, a feature extraction module, a mode classification module and a control output module, and when a manipulator descends, the system collects a sucker edge image sequence, tracks a feature point motion vector and calculates a consistency coefficient; triggering a contact event when the consistency coefficient changes from above the threshold value to below the threshold value; recognizing contact points, recording the static moment of each feature point, and constructing a contact time sequence graph; and extracting a contact starting deflection angle, a contact diffusion symmetry index and a contact time difference between the suckers, classifying contact modes and outputting corresponding control instructions. When the suction cup makes contact with the plate, the contact mode is recognized, the control instruction is output, adsorption anomaly detection is advanced, and the safety of a carrying system is improved.

Description

Technical Field

[0001] This invention relates to the field of robotic arm safety detection technology, and more specifically, to a real-time detection system for abnormal robotic arm grasping based on integrated visual recognition. Background Technology

[0002] In industrial automated production, vacuum suction cup robots are widely used for handling sheet metal, and the detection of abnormal suction is the key to ensuring safe handling.

[0003] In existing technologies, pressure sensors are typically used to detect the adsorption force after the suction cup contacts the material and establishes a vacuum, or photoelectric sensors are used to detect material detachment. However, by the time such methods detect anomalies, the robot arm has often already entered the interference zone, which can easily lead to damage to the suction cup or equipment malfunction. Chinese patent CN118596201A proposes a method for detecting abnormalities in robot handling, which determines adsorption anomalies by acquiring pressure detection data at a pre-detection position, advancing the detection position to before the handling interference position. However, it still requires the suction cup to contact the material and establish negative pressure before a judgment can be made, resulting in a response lag. Chinese patent application CN202211042051.1 provides a vision-based method for detecting abnormal robot arm positions, which determines whether the robot arm has accurately reset by comparing the positional deviation of preset features in a standard image and a detection image. However, its detection object is the overall position of the robot arm, and it cannot perceive the microscopic posture anomalies at the moment of contact between the suction cup and the material.

[0004] Therefore, existing technologies lack the means to identify contact patterns and predict adsorption anomalies the instant the suction cup contacts the substrate. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a real-time detection system for abnormal gripping by a robotic arm with integrated visual recognition. This system aims to solve the problem that existing vacuum suction cup robotic arms cannot identify the contact pattern at the instant the suction cup contacts the board during the gripping process and thus predict adsorption abnormalities, resulting in delayed detection response and easy interference accidents.

[0006] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a real-time anomaly detection system for robotic arm grasping with integrated visual recognition, comprising: The image acquisition module acquires a sequence of images containing the edge of the suction cup during the descent of the robotic arm; this image sequence is used to provide temporally continuous visual input for subsequent feature point tracking, in order to capture the motion changes of the suction cup edge as it approaches the board.

[0007] The feature point tracking module tracks pre-labeled suction cup edge feature points in the image sequence and calculates the motion vector magnitude of all feature points in each frame image. Specifically, by tracking the pixel displacement of feature points between adjacent frames, the instantaneous motion velocity information of each feature point is obtained, and the change in the magnitude of this motion vector reflects the motion state of each position on the suction cup edge.

[0008] The consistency calculation module calculates the consistency coefficient based on the dispersion of the motion vector magnitude. When the suction cup has not yet contacted the plate, all feature points move rigidly with the robot arm, and the motion vector magnitudes are equal, so the consistency coefficient approaches 1. When a feature point contacts the plate first, its motion vector magnitude drops sharply, resulting in an increase in the overall dispersion and a decrease in the consistency coefficient. This coefficient is used to quantify the degree to which motion consistency is disrupted.

[0009] The event triggering module determines that a contact event has occurred and records the trigger frame when the consistency coefficient changes from above a preset threshold to below a preset threshold. This determination condition uses the falling edge of the consistency coefficient as a marker of contact occurrence, enabling a response to be completed within the same image frame where contact occurs. Compared to detection methods that rely on physical contact signals, this helps to shorten the detection lag.

[0010] The spatiotemporal positioning module determines the feature point with the smallest motion vector magnitude in the trigger frame as the first contact point between the edge of the suction cup and the surface of the board, and records the static moment of each feature point frame by frame. The smallest motion vector magnitude indicates that the point moves the slowest, and its position reflects the orientation of the board and the suction cup when they first come into contact. The static moment records the frame index of each feature point from the moving state to the static state. Together, they constitute a spatiotemporal description of the contact process.

[0011] The map construction module constructs a contact time series map with the circumferential angle of the suction cup contour as the horizontal axis and the time of quiescence as the vertical axis. This map associates the spatial position (circumferential angle) of each feature point with the temporal information (quiescence time) to form a time series curve of contact propagation along the suction cup contour, which facilitates the subsequent extraction of contact pattern features.

[0012] The feature extraction module extracts three features from the contact timing map: contact initiation angle, contact diffusion symmetry index, and contact time difference between suction cups. Among them, the contact initiation angle indicates the orientation of the first contact point on the suction cup, the contact diffusion symmetry index represents the degree of asymmetry in the contact time on both sides of the same suction cup, and the contact time difference between suction cups reflects the deviation in the contact synchronization between different suction cups.

[0013] The pattern classification module classifies the contact pattern into at least one of the following based on the three features: normal parallel contact, plate tilt, local protrusion, or suction cup array skew. The module distinguishes different abnormal types by comparing the extracted feature values ​​with a preset frame number threshold, providing a basis for the selection of subsequent control commands.

[0014] The control output module outputs corresponding control commands based on the classification results. These control commands include allowing vacuum activation, fine-tuning the posture, stopping retraction, or issuing an alarm. Different control commands correspond to different safety response strategies, which are used to intervene in the robot arm's actions or the vacuum system in a timely manner when the contact mode is abnormal.

[0015] Furthermore, the formula for calculating the consistency coefficient by the consistency calculation module is as follows: Let the mean of the motion vector magnitudes of all feature points be... The standard deviation is Then the consistency coefficient Among them, the mean Reflects the overall speed level of motion, standard deviation It reflects the degree of difference in the motion speed of each feature point, and the ratio of the two is transformed to obtain a normalized consistency index.

[0016] Furthermore, the event triggering module determines the condition for the occurrence of a contact event as follows: the consistency coefficient of the current frame is lower than a preset threshold and the consistency coefficient of the previous frame is not lower than the preset threshold, and the current frame is recorded as the trigger frame; wherein, this condition is triggered by the falling edge of the consistency coefficient, avoiding repeated triggering when the consistency coefficient is continuously lower than the threshold, and ensuring that each contact process generates only one trigger event.

[0017] Furthermore, the process of the spatiotemporal positioning module identifying the first contact point is as follows: in the trigger frame, the motion vector magnitude of each feature point is calculated, and the feature point with the smallest motion vector magnitude is taken as the candidate point. If the motion vector magnitude of the candidate point is lower than the preset minimum threshold, it is determined to be the first contact point between the edge of the suction cup and the surface of the board. Otherwise, subsequent frames are taken and the judgment is repeated until the first contact point is determined. In this way, the progressive judgment ensures that the position of the first contact point is locked as soon as possible after the contact occurs, avoiding misjudgment due to single-frame noise.

[0018] Furthermore, the spatiotemporal positioning module records the static moment of the feature point in the following way: starting from the trigger frame, each feature point is analyzed frame by frame. When the motion vector magnitude of a feature point is lower than a preset minimum threshold for three consecutive frames from the current frame, the frame index of the current frame is recorded as the static moment of the feature point. The condition of three consecutive frames is used to filter out misjudgments caused by single-frame noise, ensuring that the recorded static moment corresponds to the stable static state after the actual physical contact.

[0019] Furthermore, the feature extraction module calculates the contact diffusion symmetry index as follows: for each suction cup, the static times of two feature points with an angle difference of 180 degrees on the suction cup contour are subtracted, and the maximum value of this difference among all suction cups is taken as the contact diffusion symmetry index; wherein, this difference reflects the degree of difference in contact time at opposite positions of the suction cup, and the maximum value characterizes the most severe contact asymmetry in the entire suction cup array.

[0020] Furthermore, the feature extraction module calculates the contact time difference between suction cups by: calculating the average value of the static time of all feature points on each suction cup, and taking the difference between the maximum and minimum values ​​of the average values ​​of each suction cup as the contact time difference between suction cups; wherein, this difference reflects the average time deviation of the contact completion between different suction cups, and is used to determine the overall contact synchronization of the suction cup array.

[0021] Furthermore, the classification rules of the pattern classification module are as follows: When the contact diffusion symmetry index is no greater than 1 frame and the contact time difference between suction cups is no greater than 1 frame, it is judged as normal parallel contact; If the contact diffusion symmetry index is not less than 2 frames and the contact time difference between suction cups is not greater than 2 frames, it is determined that the plate is tilted. If the static time of a feature point within a single suction cup is at least 3 frames earlier than the arithmetic mean of the static times of other feature points on that suction cup, and the contact time difference between suction cups is no more than 1 frame, it is determined to be a local protrusion. If the contact time difference between suction cups is not less than 3 frames and the contact diffusion symmetry index is not greater than 1 frame, it is determined that the suction cup array is skewed. The aforementioned frame rate threshold is set based on the camera frame rate. By comparing the quantized feature values ​​with the threshold, the discretization of different contact modes is achieved.

[0022] Furthermore, the control output module outputs the corresponding control commands based on the classification results as follows: When normal parallel contact is determined, a signal allowing vacuum to be turned on is output. When the plate is determined to be tilted, the output carries the contact initiation angle. The attitude fine-tuning command is given and a deceleration descent signal is output simultaneously; When a local protrusion is detected, an instruction is output to stop the adsorption and perform a retraction action; When the suction cup array is determined to be misaligned, a command to prohibit vacuum opening is output and a device maintenance alarm signal is output simultaneously. Among them, the contact start angle in the attitude fine-tuning command The signal is used to instruct the robot arm to adjust the direction of the suction cup posture; the deceleration descent signal is used to reduce the contact speed to reduce impact; the stop retraction command is used to avoid further descent and damage; and the vacuum opening prohibition signal is used to stop the adsorption process.

[0023] Furthermore, the vacuum opening signal output by the control output module serves as the enable signal for the vacuum solenoid valve. The vacuum solenoid valve is activated only upon receiving this signal; otherwise, it remains locked. This logical interlocking mechanism ensures that the vacuum system can only be started when visual detection determines that the system is in normal parallel contact, thus preventing adsorption failure or equipment interference caused by establishing a vacuum in abnormal contact mode.

[0024] The technical effects and advantages of this invention are as follows: This invention constructs a multi-level detection process from motion consistency detection to contact timing graph analysis, enabling instantaneous identification of contact patterns and output of control commands when the suction cup contacts the plate.

[0025] (1) To address the issue of delayed detection response, this invention continuously acquires images of the suction cup edge during the descent of the robotic arm, tracks the motion vector of feature points, and calculates the consistency coefficient. When the consistency coefficient first falls below a preset threshold, a contact event is triggered. This triggering mechanism completes the response within the same image frame where contact occurs. Compared to detection methods that rely on vacuum pressure signals, this helps to advance the detection starting point of adsorption anomalies to the instant of contact, allowing the robotic arm control system time for attitude adjustment or safe stopping.

[0026] (2) To address the problem of inability to distinguish microscopic contact postures, this invention identifies the first contact point and records the static time of each feature point, constructing a contact time sequence map with circumferential angle and static time as coordinates. From this map, the contact diffusion symmetry index and the contact time difference between suction cups are extracted, thereby distinguishing different contact modes such as normal parallel contact, plate tilt, local protrusion, and suction cup array skew. This classification mechanism helps to identify abnormal postures in the very short time after contact is established, providing a basis for the accurate output of subsequent control commands.

[0027] (3) Based on the classification results, this invention outputs commands such as allowing vacuum opening, attitude fine-tuning, stopping retreat, or alarm, and forms a logical interlock with the vacuum solenoid valve. By directly applying the visual detection results to the enable control of the vacuum system, it helps to prevent vacuum establishment when an abnormal contact mode is determined, thereby improving the safety of the handling system and reducing equipment damage caused by abnormal adsorption. Attached Figure Description

[0028] Figure 1 This is a system module framework diagram of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Example 1 As attached Figure 1 The integrated visual recognition robotic arm anomaly detection system shown below, combined with a specific application scenario of a vacuum suction cup robotic arm grasping stamped sheet metal or coated substrate, provides a detailed description of the technical solution of this invention. Those skilled in the art should understand that the embodiments described below are merely illustrative of the implementation of this invention and do not constitute a limitation on the scope of protection of this invention.

[0031] I. System Overall Architecture and Deployment Method The present invention provides an integrated visual recognition robotic arm grasping anomaly real-time detection system, which is deployed on a vacuum suction cup robotic arm.

[0032] The robotic arm's end effector is equipped with multiple silicone vacuum suction cups, each connected to a vacuum generator and a vacuum solenoid valve via an independent air path. An industrial camera is mounted on the robotic arm's end effector bracket, with the camera lens facing downwards and the optical axis perpendicular to the plane of the suction cups. The camera uses a global shutter CMOS sensor with a resolution of 1280×720 pixels and a frame rate of 60 frames per second.

[0033] A ring-shaped LED light source with a color temperature of 5000K is installed in front of the camera lens to ensure clear image acquisition of the suction cup edge under different ambient lighting conditions. The camera and light source are connected to the edge computing unit inside the robot controller via cables. This computing unit runs a real-time operating system and is responsible for executing the functions of the various modules of this invention.

[0034] The system comprises nine functional modules: image acquisition, feature point tracking, consistency calculation, event triggering, spatiotemporal localization, map construction, feature extraction, pattern classification, and control output. Each module is implemented in software as a different thread on the same processor, and data interaction between modules is accomplished through shared memory.

[0035] II. Initialization and Feature Point Calibration Implementation Methods The initialization process is executed after the system is used for the first time or after the suction cup is replaced.

[0036] The robotic arm moves to its initial position under the command of the control system. The initial position is defined as follows: the suction cup plane is perpendicular to the camera's optical axis, the suction cup is 100 mm from the platform below, and the suction cup is completely exposed within the camera's field of view without any obstruction. At this position, the image acquisition module captures a reference image.

[0037] The feature point tracking module processes the reference image. First, the Canny edge detection operator is used to extract edges from the image, with a low threshold of 50 and a high threshold of 150. The input to this operator is the reference image, and the output is a binary edge image.

[0038] Then, the circular outline of each suction cup is identified using the Hough circle transform. The parameters of the Hough circle transform are set as follows: minimum distance is 1.5 times the diameter of the suction cup, param1=100, param2=30. The input of this transform is a binary edge image, and the output is the center coordinates and radius of each suction cup.

[0039] For each suction cup, 12 feature points are sampled at equal intervals along its contour. The sampling starts at 0° (with the horizontal direction to the right of the image as the 0° reference), and is performed at equal angular intervals in a counter-clockwise direction, with an angular interval of 30° between adjacent feature points. The pixel coordinates of each feature point are stored as reference coordinates. The feature point tracking module also records the suction cup number (integers starting from 1) to which each feature point belongs, as well as the circumferential angle value of that feature point on the suction cup contour, with the angle value ranging from 0° to 360°.

[0040] In another implementation, instead of using equal-angle sampling, the Shi-Tomasi corner detection algorithm is employed to extract strong corner points as feature points from the suction cup edge. The algorithm takes a baseline image as input, with a quality level of 0.01 and a minimum spacing of 10 pixels. The output is the coordinates of all detected corner points, sorted by response value from largest to smallest, with the top 12 corner points selected as the feature points of the suction cup. This method is more adaptable to suction cups with irregular shapes or worn edges.

[0041] After initialization, the system enters standby mode, waiting for the robotic arm to begin its descent.

[0042] III. Implementation Methods for Image Acquisition and Feature Point Tracking When the robotic arm control system issues a descent command, the image acquisition module starts continuous image acquisition.

[0043] The camera continuously acquires images at a frame rate of 60 frames per second, with each frame spaced 16.67 milliseconds apart. After each frame is acquired, the image acquisition module sends the image data to the feature point tracking module.

[0044] The feature point tracking module uses the Lucas-Kanade sparse optical flow method to track each feature point frame by frame. The input of the Lucas-Kanade optical flow method is: the previous frame image, the current frame image, and the pixel coordinates of each feature point in the previous frame; the window size is set to 15×15 pixels, and the number of pyramid layers is set to 3; the output is: the pixel coordinates of each feature point in the current frame and a tracking success flag.

[0045] For each successfully tracked feature point The feature point tracking module calculates its motion vector. ,in For feature points in the current frame The pixel coordinate vector, Feature points in the previous frame The pixel coordinate vector. Motion vector. It is a two-dimensional vector.

[0046] Magnitude of motion vector Calculated using Euclidean norm: .

[0047] The feature point tracking module outputs the set of motion vector magnitudes of all successfully tracked feature points in each frame to the consistency calculation module.

[0048] If a feature point fails to be tracked for three consecutive frames (e.g., due to changes in lighting or occlusion), the feature point tracking module marks the feature point as lost and will no longer track it in subsequent frames. If the number of lost feature points exceeds 30% of the total number of feature points (12 × number of suction cups), the system outputs a warning signal, indicating that reinitialization is required. Lost feature points are not included in subsequent consistency coefficient calculations.

[0049] IV. Implementation Method for Calculating the Consistency Coefficient The consistency calculation module receives data from the feature point tracking module.

[0050] Set the current frame Total 1 effective feature point ( (The number of feature points successfully tracked without loss), the magnitude of the motion vector of each feature point is denoted as . .

[0051] The consistency calculation module first calculates the arithmetic mean of these values. : ; Then calculate the standard deviation. : ; Finally, calculate the consistency coefficient. : ; when When the magnitude of the motion vectors of all feature points is 0, the division in the formula is undefined, and the consistency coefficient is... Set it directly to 1.

[0052] The consistency calculation module takes the motion vector magnitude of each feature point as input and outputs the consistency coefficient of the current frame. This module calculates the results for each frame. The value is output to the event triggering module.

[0053] V. Implementation Method for Triggering Contact Events The event triggering module stores a preset threshold internally. .

[0054] The value is determined based on the descent speed of the robotic arm. (Unit: millimeters per second) and camera frame rate (Unit: frames per second) is determined, and the calculation formula is: .

[0055] Regarding the descent speed in this embodiment millimeter per second, frame rate Frames per second, ,Pick .

[0056] The event triggering module also stores the consistency coefficient of the previous frame. .

[0057] For each frame The event triggering module performs the following judgment: If and If so, a contact event is determined to have occurred, and the current frame index is set. Record as trigger frame .

[0058] This condition ensures that the event is triggered only once on the falling edge when the consistency coefficient changes from above the threshold to below the threshold, avoiding triggering in subsequent frames due to... The value remains below the threshold and triggers repeatedly.

[0059] In another implementation, the event triggering module sets up a de-jitter mechanism: requiring two consecutive frames... The values ​​are all lower than And the previous frame Value not lower than Only then is a contact event considered to have occurred. The specific criteria are: and and This method suppresses false triggering caused by single-frame noise.

[0060] After being triggered, the event triggering module will trigger a frame. The image data of the preceding and following three frames (a total of seven frames) are packaged in frame index order and sent to the spatiotemporal positioning module, while the trigger frame is also sent. The index is sent to the graph construction module.

[0061] VI. Implementation Method for First Contact Point Identification After receiving the contact event trigger signal and image sequence, the spatiotemporal positioning module immediately performs the first contact point identification.

[0062] The spatiotemporal positioning module internally stores a preset minimum threshold. , The value is determined based on the descent speed of the robotic arm. Confirmed, the calculation formula is: ,in This is a scaling constant, with a value of 0.005 pixels per frame per millimeter per second. This is the descent speed of the robotic arm (in millimeters per second). This threshold is used to determine whether the feature point is essentially stationary.

[0063] The spatiotemporal localization module processes the received image sequences sequentially according to their frame indices, from smallest to largest. The recognition process is as follows: The first step is to retrieve the trigger frame. Calculate the motion vector magnitude of each feature point in the frame. Find the feature point corresponding to the minimum value and denote its index as . . To meet Feature point numbering.

[0064] The second step is to determine... Is it true? If it is true, then directly determine the feature point. This is the first point of contact between the edge of the suction cup and the surface of the substrate. If this does not occur, proceed to the next frame. Repeat steps one and two. Continue this process, taking subsequent frames in sequence, until the first frame that satisfies the condition is found. frames The minimum point in that frame is determined as the first contact point.

[0065] If no point meeting the conditions is found within 10 frames after the trigger frame, the spatiotemporal positioning module determines that the detection is invalid, outputs an error flag, and reinitializes.

[0066] Once the first contact point is determined, the spatiotemporal positioning module records the suction cup number to which that point belongs. (The suction cups are numbered 1 to...) integers, (total number of suction cups) and the circumferential angle of the feature point on the suction cup profile. . This is the initial contact angle.

[0067] The spatiotemporal localization module takes as input an image sequence and the magnitude of the motion vectors of feature points in each frame, and outputs the suction cup number of the first contact point. and contact initial deflection angle .

[0068] VII. Implementation Method of Static Time Recording After determining the first contact point, the spatiotemporal positioning module continues to record the moment of quiescence for each feature point.

[0069] The static moment is defined as the starting frame index where the feature point first enters and remains static. From the trigger frame... Begin checking the motion vector magnitude of each feature point frame by frame, regardless of whether the first contact point has been identified.

[0070] The specific recording method is as follows: for each feature point... From the trigger frame Begin by checking the magnitude of its motion vectors frame by frame. Define a static counter. The initial value is 0.

[0071] For each frame ,like ,but Increase by 1; otherwise Reset to zero. When Achieve the preset number of consecutive still frames When, the starting frame index of the current frame (i.e. Record the time of staticization of this feature point. .in The frame index of the last frame that satisfies the condition of three consecutive static frames.

[0072] For feature points that fail to meet the condition of three consecutive static frames within 30 frames after the trigger frame, their static time is... Marked as infinity, it indicates that the suction cup position corresponding to this feature point is not in contact with the board.

[0073] The spatiotemporal localization module outputs the static timestamps of all feature points to the atlas construction module. This module takes the motion vector magnitude of each feature point in each frame as input and outputs the static timestamp of each feature point. .

[0074] VIII. Implementation Method for Constructing Contact Time Series Maps The map construction module receives the static time data of each feature point.

[0075] For each suction cup, the mapping module performs the following operations: The circumferential angle of the suction cup's outline The x-axis represents the x-axis, and its value range is [value range missing]. to . (Based on the static moment) The vertical axis is used, with the unit being the frame index, to trigger the frame. As a reference for frame 0 (i.e., the static time corresponding to the trigger frame is 0).

[0076] Each feature point belonging to the suction cup is determined by its circumferential angle. and the time of stillness Plot these points as a single point in the coordinate system. Connecting the smallest to largest values ​​sequentially forms a broken line, called the contact timing curve. .

[0077] At the data storage level, the map construction module stores the data point set of each suction cup in the form of an array. Each element of the array contains three fields: suction cup ID (integer), circumferential angle, and circumferential angle. (Floating-point number, in degrees), time to quiescent (Integer, frame index). This array is stored in shared memory and can be directly read by the feature extraction module.

[0078] The input to the map construction module is the static time of each feature point. and its corresponding suction cup number and circumferential angle The output is an array storing the contact timing map.

[0079] IX. Feature Extraction Implementation Methods The feature extraction module extracts three features from the data output by the map construction module: contact initiation angle, contact diffusion symmetry index, and contact time difference between suction cups.

[0080] First, the initial contact angle This value is obtained directly from the output of the spatiotemporal positioning module and does not require further calculation.

[0081] Second, the contact diffusion symmetry index The feature extraction module iterates through all feature point pairs on the contour of each suction cup, selecting those with angular differences. The two points, i.e., the angle and If both points have a time of quiescence (neither of which is infinite), then calculate... .

[0082] Iterate through all suction cups and all angle pairs, and retrieve all... The maximum value in is used as If there is no usable angle pair on a suction cup, then that suction cup will not participate. The calculation. The unit is a frame.

[0083] Third, the contact time difference between suction cups The feature extraction module first calculates the average static time for each suction cup. For each suction cup... Suppose that it has If there are 1 effective feature point (the static time is not infinite), then the average static time is denoted as . .

[0084] Then, take the maximum value of the average static time of all suction cups. and minimum value Calculate the difference If the number of valid feature points of a certain suction cup is 0, then that suction cup will not participate. The calculation triggers an alarm on the device. The unit is a frame.

[0085] In another implementation, the contact time difference between suction cups is calculated using the difference of the median times at which each suction cup comes to a standstill. For suction cups... Take the median of the static times of all valid feature points on it. ,but This method reduces the impact of outliers.

[0086] The feature extraction module takes an array of contact time series maps as input and outputs three feature values. .

[0087] 10. Implementation Methods of Pattern Classification The pattern classification module internally stores a classification rule table, based on feature values. Determine the contact mode. The classification rules are as follows: Rule 1: If and If so, it is determined to be normal parallel contact.

[0088] Rule 2: If and If so, it is determined that the board is tilted.

[0089] Rule 3: If a suction cup exists The static time of a feature point on it. It is at least 3 frames earlier than the arithmetic mean of the static times of all other feature points on the suction cup, and If so, it is determined to be a local protrusion.

[0090] Rule 4: If and If so, it is determined that the suction cup array is skewed.

[0091] The frame rate thresholds (1 frame, 2 frames, 3 frames) in the above rules are related to the camera frame rate. For a system with a frame rate of 60 frames per second, 1 frame corresponds to 16.67 milliseconds. The thresholds are set to fixed values ​​and do not change with operating conditions.

[0092] The input to the pattern classification module is three feature values. The output is the judgment result (enumerated values: normal parallel contact, plate tilt, local protrusion, suction cup array skew).

[0093] XI. Implementation Method of Control Command Output Based on the determination result of the mode classification module, the control output module outputs the corresponding control commands. These commands are sent to the robot controller and vacuum solenoid valve via digital output ports or a fieldbus. The fieldbus uses either the EtherCAT or Profinet protocol.

[0094] When normal parallel contact is detected, the control output module outputs a signal allowing vacuum to open. This signal is active high for 100 milliseconds, after which it automatically goes low. Upon receiving this high-level signal, the enable terminal of the vacuum solenoid valve remains in an active state within this 100-millisecond window. During this time, other outputs of the robot controller can normally control the solenoid valve to open. If a vacuum has not been established by the end of the 100-millisecond window, the solenoid valve automatically closes.

[0095] When the plate is determined to be tilted, the control output module outputs a contact initiation angle. It provides attitude fine-tuning commands and simultaneously outputs deceleration and descent signals.

[0096] The attitude fine-tuning command uses analog output, and the output voltage is related to... A linear relationship is established: 0V corresponds to 0°, 10V corresponds to 360°, and the output voltage accuracy is 0.01V. In another implementation, the attitude fine-tuning command is sent in digital form via the EtherCAT fieldbus, and the command data frame contains the contact start angle. Floating-point values.

[0097] The deceleration signal is a digital pulse with a pulse width of 50 milliseconds, triggering the robot controller to reduce the descent speed to 50% of the current speed or to a preset safe speed of 30 millimeters per second. The robot then... Adjust the suction cup orientation to make the suction cup plane nearly parallel to the board plane.

[0098] When a localized protrusion is detected, the control output module outputs a command to abort adsorption and execute a retraction action. This command first triggers the robot arm's emergency stop function, immediately halting the descent; then, it outputs an ascending retraction command, causing the robot arm to rise to its initial position at a speed of 200 mm / s. Simultaneously, the control output module sends an anomaly record to the production management system via Ethernet, containing the current board number and position coordinates, and outputs a board anomaly alarm signal.

[0099] When the suction cup array is determined to be misaligned, the control output module outputs a command to prohibit vacuum opening, and simultaneously outputs a device maintenance alarm signal. The command to prohibit vacuum opening is a continuous low-level signal, which remains until the system is reset or the fault is manually cleared by the operator or the next initialization is completed. This signal directly locks the enable terminal of the vacuum solenoid valve, preventing it from being opened by any signal. The device maintenance alarm signal triggers an audible and visual alarm and displays fault code E401 on the human-machine interface, prompting "Suction cup array misaligned, please check suction cup installation."

[0100] The input to the control output module is the pattern classification result, and the output is the control command signals mentioned above.

[0101] 12. Implementation Methods of Logical Interlocks The output terminal of the control output module and the control circuit of the vacuum solenoid valve are logically interlocked by software logic.

[0102] Specifically, in the ladder logic program of the robot controller, the vacuum opening permission signal output by the control output module is connected in series as a normally open contact in the output coil circuit of the vacuum solenoid valve. The output coil circuit of the vacuum solenoid valve is controlled by two conditions connected in series: the first condition is the vacuum opening command output by the robot controller, and the second condition is the vacuum opening permission signal output by the control output module. Both conditions are active high.

[0103] Therefore, the solenoid valve will only be energized and open when both the vacuum opening signal and the vacuum opening command are high. If the vacuum opening signal is low, the solenoid valve will remain closed regardless of changes in the vacuum opening command.

[0104] In another implementation, the logical interlock is achieved through hardware circuitry: the drive circuit of the vacuum solenoid valve is controlled by two series-connected solid-state relays. The control terminal of the first relay is connected to the vacuum opening command output of the robot controller, and the control terminal of the second relay is connected to the vacuum opening permission signal output of the control output module. This implementation also achieves the effect of hardware interlocking.

[0105] XIII. Explanation of References to Prior Art Content In the embodiments of the present invention, the following prior art content is involved, and the specific implementation methods are not described in detail. Only the input-output relationship and the calling sequence in this solution are explained: Image acquisition: Implemented using a software development kit (SDK) provided by the camera manufacturer. The SSD takes a trigger signal as input (triggered by the robot arm's descent command) and outputs grayscale image data. This module is continuously invoked at a rate of 60 frames per second after the robot arm begins its descent, until adsorption is complete or an abnormal termination occurs.

[0106] Optical flow feature point tracking: This is implemented using the `calcOpticalFlowPyrLK` function from the open-source computer vision library. The function takes the coordinates of the feature points from the previous and following frames as input, and outputs the coordinates of the feature points in the current frame along with a tracking success flag. This function is called once per frame.

[0107] Edge detection and Hough circle transform: Implemented using the Canny and HoughCircles functions from the OpenCV library. The Canny function takes the baseline image as input and outputs a binary edge image. The HoughCircles function takes the binary edge image as input and outputs the center coordinates and radius of the suction cup. These two functions are called only once during system initialization.

[0108] Robotic arm motion control: Utilizing the robotic arm controller's built-in motion instruction library. This library takes speed and position commands as inputs and outputs the actual movement of the robotic arm. This module is invoked upon receiving instructions from the control output module.

[0109] Vacuum solenoid valve control: A digital output module using a programmable logic controller (PLC) or robotic arm controller is employed. The module's inputs are an enable signal (from the control output module) and a switch signal (from the robotic arm controller), with the output being the opening and closing of the solenoid valve. This module responds in real-time to changes in the input signals.

[0110] Based on the above description, combined with common knowledge and existing technology, those skilled in the art can fully implement the technical solution of the present invention without creative effort.

[0111] XIV. System Workflow Summary The complete workflow of the integrated visual recognition robotic arm anomaly real-time detection system of the present invention is as follows: After the robotic arm begins its descent, the image acquisition module continuously acquires images. The feature point tracking module calculates the motion vector magnitude of each feature point frame by frame. The consistency calculation module calculates the mean value frame by frame. Standard deviation Consistency coefficient .

[0112] Event triggering module monitoring The change in value, when From higher Become lower than Record the trigger frame at that time. .

[0113] The spatiotemporal positioning module processes the received image sequence according to the frame index order, identifies the first contact point in the trigger frame and subsequent frames, and obtains the contact initiation deflection angle. and from the trigger frame Begin examining the motion vector magnitude of each feature point frame by frame, and record the static moment of each feature point. (Motion vector size is lower than 3 consecutive frames) (Time record).

[0114] The map construction module uses a circumferential angle. The horizontal axis represents the time of staticization. A contact time-series map of each suction cup is constructed with the ordinate as the vertical axis. The feature extraction module calculates the contact diffusion symmetry index from the map. Contact time difference between the suction cup and the suction cup .

[0115] The pattern classification module is based on The contact mode is determined according to the rules. The control output module outputs corresponding control commands based on the determination result and controls the enable state of the vacuum solenoid valve through a logic interlock mechanism.

[0116] Through the above embodiments, the present invention realizes the instantaneous identification of the contact mode and output of control commands when the suction cup contacts the board, and advances the detection of adsorption anomalies from after the vacuum is established to the moment of contact, thus reserving sufficient response time for the robot control system and effectively avoiding suction cup damage, board detachment and equipment interference accidents caused by adsorption anomalies.

[0117] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time anomaly detection system for robotic arm grasping with integrated visual recognition, characterized in that, include: The image acquisition module acquires a sequence of images, including the edge of the suction cup, continuously collected during the descent of the robotic arm; The feature point tracking module tracks pre-labeled suction cup edge feature points in the image sequence and calculates the motion vector magnitude of all feature points in each frame of the image; The consistency calculation module calculates the consistency coefficient based on the degree of dispersion of the motion vector magnitude; The event triggering module determines that a contact event has occurred and records the trigger frame when the consistency coefficient changes from being higher than a preset threshold to being lower than a preset threshold. The spatiotemporal positioning module determines the feature point with the smallest motion vector in the trigger frame as the first contact point between the edge of the suction cup and the surface of the board, and records the moment of stillness of each feature point frame by frame. The map construction module constructs a contact time sequence map with the circumferential angle of the suction cup outline as the horizontal axis and the time of stasis as the vertical axis. The feature extraction module extracts three features from the contact timing map: contact initiation angle, contact diffusion symmetry index, and contact time difference between suction cups. The pattern classification module classifies the contact pattern into at least one of the following based on the three characteristics: normal parallel contact, plate tilt, local protrusion, or suction cup array skew. The control output module outputs corresponding control commands based on the classification results. These control commands include enabling vacuum opening, fine-tuning the attitude, stopping the retraction, or issuing an alarm.

2. The integrated visual recognition robotic arm anomaly detection system according to claim 1, characterized in that, The formula for calculating the consistency coefficient by the consistency calculation module is: Let the mean of the motion vector magnitudes of all feature points be... The standard deviation is Then the consistency coefficient .

3. The integrated visual recognition robotic arm anomaly real-time detection system according to claim 1, characterized in that, The event triggering module determines the occurrence of a contact event based on the following conditions: the consistency coefficient of the current frame is lower than a preset threshold and the consistency coefficient of the previous frame is not lower than the preset threshold, and the current frame is recorded as the trigger frame.

4. The integrated visual recognition robotic arm anomaly real-time detection system according to claim 1, characterized in that, The process of the spatiotemporal positioning module identifying the first contact point is as follows: in the trigger frame, the motion vector magnitude of each feature point is calculated, and the feature point with the smallest motion vector magnitude is taken as the candidate point. If the motion vector magnitude of the candidate point is lower than the preset minimum threshold, it is determined to be the first contact point between the edge of the suction cup and the surface of the board. Otherwise, the subsequent frames are taken and the judgment is repeated until the first contact point is determined.

5. The integrated visual recognition robotic arm real-time detection system for grasping anomalies according to claim 1, characterized in that, The spatiotemporal positioning module records the static moment of feature points by analyzing each feature point frame by frame from the trigger frame. When the motion vector magnitude of a feature point is lower than the preset minimum threshold for three consecutive frames from the current frame, the frame index of the current frame is recorded as the static moment of the feature point.

6. The integrated visual recognition robotic arm anomaly real-time detection system according to claim 1, characterized in that, The feature extraction module calculates the contact diffusion symmetry index by subtracting the static times of two feature points on the suction cup contour that differ in angle by 180 degrees from each other, and taking the maximum value of this difference among all suction cups as the contact diffusion symmetry index.

7. The integrated visual recognition robotic arm anomaly real-time detection system according to claim 1, characterized in that, The feature extraction module calculates the contact time difference between suction cups by calculating the average value of the static time of all feature points on each suction cup, and taking the difference between the maximum and minimum values ​​of the average values ​​of each suction cup as the contact time difference between suction cups.

8. The integrated visual recognition robotic arm real-time detection system for grasping anomalies according to claim 1, characterized in that, The classification rules of the pattern classification module are as follows: When the contact diffusion symmetry index is no greater than 1 frame and the contact time difference between suction cups is no greater than 1 frame, it is judged as normal parallel contact; If the contact diffusion symmetry index is not less than 2 frames and the contact time difference between suction cups is not greater than 2 frames, it is determined that the plate is tilted. If the static time of a feature point within a single suction cup is at least 3 frames earlier than the arithmetic mean of the static times of other feature points on that suction cup, and the contact time difference between suction cups is no more than 1 frame, it is determined to be a local protrusion. If the contact time difference between suction cups is not less than 3 frames and the contact diffusion symmetry index is not greater than 1 frame, it is determined that the suction cup array is skewed.

9. The integrated visual recognition robotic arm real-time detection system for grasping anomalies according to claim 8, characterized in that, The control output module outputs the corresponding control command based on the classification result: When normal parallel contact is determined, a signal allowing vacuum to be turned on is output. When the plate is determined to be tilted, the output carries the contact initiation angle. The attitude fine-tuning command is given and a deceleration descent signal is output simultaneously; When a local protrusion is detected, an instruction is output to stop the adsorption and perform a retraction action; When the suction cup array is determined to be misaligned, a command to prohibit vacuum operation is output, and a device maintenance alarm signal is output simultaneously.

10. The integrated visual recognition robotic arm anomaly real-time detection system according to claim 1, characterized in that, The vacuum opening signal output by the control output module serves as the enable signal for the vacuum solenoid valve. The vacuum solenoid valve is activated only when it receives this signal; otherwise, it remains locked.

Images