Pneumatic manipulator motion implementation method and system based on industrial vision

Abstract

The invention relates to the technical field of industrial vision and robot control, and discloses a pneumatic manipulator motion implementation method and system based on industrial vision, and the method comprises the steps: carrying out the image collection of a to-be-sorted article, judging whether visual abnormality exists or not, and transmitting a visual abnormality signal to a motion controller of a pneumatic manipulator if the visual abnormality exists; the current operation of the pneumatic manipulator is interrupted, preset non-contact probing movement is executed, and multiple frames of images of the to-be-sorted articles in the non-contact probing movement process are collected; analyzing and determining the optical reflection characteristics of the to-be-sorted articles in the multiple frames of images, and correcting the multiple frames of images to generate an accurate feature map of the to-be-sorted articles; and according to the accurate feature map, the accurate position and posture information of the to-be-sorted articles is calculated, a grabbing instruction is generated according to the accurate position and posture information and sent to the motion controller, and the pneumatic manipulator is controlled to execute the grabbing action. The problem that the precision of a traditional method is reduced in a complex environment is solved.

Description

Technical Field

[0001] This application relates to the fields of industrial vision and robot control technology, and more specifically, to a method and system for realizing the motion of a pneumatic manipulator based on industrial vision. Background Technology

[0002] In the field of modern industrial automation, high-speed sorting operations place extremely high demands on the motion precision and continuity of pneumatic robots. Traditional control methods struggle to adapt to subtle changes in the working environment or the characteristics of the processed items, leading to performance degradation. On a high-speed automated sorting production line, pneumatic robots are guided by a sophisticated industrial vision system equipped with a high-resolution camera and stable lighting, and undergo rigorous calibration.

[0003] However, during the long-term operation of the production line, the upstream raw material supplier adjusted the source of the raw materials supplied for the items to be sorted, resulting in a loss of uniform surface gloss in the new batches of items. When the industrial vision system continued to use its preset fixed lighting scheme to acquire images of these new batches of items, the lighting light illuminating the enhanced specular reflection areas on the item surface would reflect a large amount of light directly back to the industrial camera lens, causing local oversaturation or bright whitening in these areas in the image. This whitening effect would make the key feature points of the item blurry or even completely lost in the image, seriously affecting the vision system's complete extraction of the item's feature information, making the image data no longer an accurate mapping of the object's true physical state.

[0004] When the image recognition module receives images containing locally highlighted or blurred areas, its internally preset criteria for distinguishing targets from the background become inapplicable. When large areas of whitening appear in the image, the brightness gradient information is disrupted, edges become blurred, or artifacts appear, making it difficult for the system to accurately delineate the true outline of the object. This directly leads to significant deviations in the calculation of the target object's position coordinates. The system may even misidentify some whitening areas of an object as part of the background, or identify a complete object as an incomplete target, thus outputting incorrect grasping position and posture information.

[0005] The motion control program of the pneumatic robot generates inaccurate grasping paths and posture commands based on these erroneous object position coordinates provided by the vision system. The gripper may not be able to completely cover the item, causing the item to slip, tilt, or even collide unexpectedly with the end effector during grasping, or it may completely miss the gripper, causing the item to fall off the conveyor belt or be pushed away from the intended position.

[0006] With the frequent occurrence of such misidentification and grasping failures, the number of defective products on the production line has increased significantly, directly impacting production efficiency and product qualification rate. A deeper problem is that repeated unexpected collisions or multiple grasping attempts subject the robotic arm's joint bearings and internal cylinder seals to impacts and friction exceeding normal limits, accelerating the wear of these critical physical components. This wear, in turn, further affects the long-term stability and motion accuracy of the robotic arm, creating a vicious cycle that leads to a continuous decline in the overall performance of the robotic arm.

[0007] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this application provides a method and system for realizing the movement of a pneumatic manipulator based on industrial vision, in order to solve problems such as vision system recognition errors, manipulator grasping failures, and accelerated equipment wear caused by uneven surface characteristics of items in existing industrial automated sorting operations.

[0009] Firstly, this application provides a method for realizing the motion of a pneumatic manipulator based on industrial vision, including: Images of items to be sorted on the conveyor belt are captured, and visual anomalies are determined based on the captured images. If visual anomalies are found, a visual anomaly signal is sent to the motion controller of the pneumatic manipulator. In response to visual anomaly signals, the current operation of the pneumatic manipulator is interrupted, and the pneumatic manipulator is controlled to perform a preset non-contact probing motion, and multiple frames of images of the items to be sorted are acquired during the process of the pneumatic manipulator performing the preset non-contact probing motion. The optical reflection characteristics of the items to be sorted in multiple frames of images are analyzed and determined. Based on the optical reflection characteristics, the multiple frames of images are corrected to generate accurate feature maps of the items to be sorted. Based on the precise feature map, the accurate position and orientation information of the items to be sorted are calculated. Based on the accurate position and orientation information, a gripping instruction is generated and sent to the motion controller, so that the motion controller controls the pneumatic robot to perform the gripping action according to the gripping instruction.

[0010] This technical solution effectively addresses the visual recognition problem caused by changes in the surface characteristics of objects. By using non-contact exploration and image correction, it obtains accurate object feature maps, thereby enabling pneumatic robotic arms to accurately grasp objects and solving the problem of decreased accuracy in complex environments using traditional methods.

[0011] Furthermore, the preset non-contact probing motion includes controlling the pneumatic manipulator to move along a preset non-contact path, so that its end effector sweeps over the items to be sorted to form a dynamic light and shadow area on the surface of the items to be sorted.

[0012] This technical solution utilizes changes in dynamic light and shadow areas to more comprehensively capture the optical reflection characteristics of an object's surface, providing a richer data foundation for subsequent image correction.

[0013] Furthermore, the steps for analyzing and determining the optical reflectance properties of the items to be sorted in multiple frames of images include: Based on multiple frames of images, key feature points of the items to be sorted are identified by a feature point detection algorithm, and the changes in brightness, contrast and position of the key feature points in dynamic light and shadow areas are calculated as optical reflection characteristics.

[0014] This technical solution can accurately quantify the optical response of an object under dynamic lighting conditions, providing accurate input parameters for subsequent image correction and improving the targeting and effectiveness of the correction.

[0015] Based on the above, this application further proposes a step of correcting multiple frames of images based on optical reflection characteristics to generate accurate feature maps of items to be sorted, including: Based on optical reflection characteristics, image fusion or de-reflection algorithms are used to remove or compensate for local oversaturated or blurred areas in multiple frames of images caused by fixed lighting and uneven gloss, thereby generating accurate feature maps.

[0016] In some preferred embodiments, the step of determining whether a visual abnormality exists based on the acquired image includes: Determine whether there are local areas in the acquired image where the pixel brightness value reaches or exceeds a preset saturation threshold, and / or whether the edge sharpness of key features of objects in the image is lower than a preset sharpness threshold.

[0017] As a technological improvement, it also includes the following before performing the preset non-contact probing motion: Based on the preset non-contact path, the geometric model of the end effector of the pneumatic manipulator, and the fixed lighting parameters on the conveyor belt, the predicted sequence of light and shadow features of the dynamic light and shadow area changing over time under ideal conditions is pre-calculated and stored. The light and shadow feature prediction sequence is compared with the optical reflection characteristics of multiple frames of images, and the multiple frames of images are preprocessed based on the comparison results. Based on optical reflection characteristics, the pre-processed multi-frame images are corrected to generate accurate feature maps of the items to be sorted.

[0018] Based on this, the steps for comparing the light and shadow feature prediction sequence with the optical reflectance characteristics of multiple frames of images include: By comparing the optical reflection characteristics with the light and shadow feature prediction sequence, random high-frequency noise caused by environmental vibration and / or systematic deviations caused by deviations in the actual movement trajectory of the pneumatic manipulator can be identified. Preprocessing of multiple frames of images based on random high-frequency noise and / or systematic bias.

[0019] To improve the solution, the preprocessing steps for multiple frames of images based on random high-frequency noise and / or systematic bias include: For random high-frequency noise, a time-domain filtering method is used to smooth multiple frames of images; and / or, for systematic deviations, a preset image analysis model is adjusted to compensate for the light and shadow of multiple frames of images.

[0020] In practical applications, the grasping command includes the best approach path, the grasping point, and the grasping posture.

[0021] Secondly, this application also discloses a pneumatic manipulator motion realization system based on industrial vision, the system comprising: The anomaly detection module is used to acquire images of the items to be sorted on the conveyor belt and determine whether there are visual anomalies based on the acquired images. If there are visual anomalies, a visual anomaly signal is sent to the motion controller of the pneumatic manipulator. The detection and image acquisition module is used to interrupt the current operation of the pneumatic manipulator in response to visual abnormality signals, control the pneumatic manipulator to perform a preset non-contact detection motion, and acquire multiple frames of images of the items to be sorted during the pneumatic manipulator's preset non-contact detection motion. The image correction processing module is used to analyze and determine the optical reflection characteristics of the items to be sorted in multiple frames of images. Based on the optical reflection characteristics, the multiple frames of images are corrected to generate accurate feature maps of the items to be sorted. The gripping operation execution module is used to calculate the accurate position and posture information of the items to be sorted based on the precise feature map, generate gripping instructions based on the accurate position and posture information, and send them to the motion controller so that the motion controller controls the pneumatic manipulator to perform gripping actions according to the gripping instructions.

[0022] This application provides a system that can implement the above-mentioned method through this technical solution. The modular design ensures the collaborative work of each functional unit, thereby achieving effective processing and accurate capture of complex visual anomalies.

[0023] In summary, this application provides a method and system for realizing the motion of a pneumatic robot based on industrial vision. This method effectively solves the problems of vision system recognition errors, robot gripping failures, and accelerated equipment wear caused by uneven surface characteristics of objects in existing technologies. Specifically, when the vision system detects visual anomalies such as local oversaturation or blurred areas in the image, the system no longer blindly grips based on erroneous images. Instead, it triggers the pneumatic robot to execute a preset non-contact probe motion. By acquiring multiple frames of images and analyzing the optical reflection characteristics of the object under dynamic lighting, it can accurately identify and compensate for image distortion caused by uneven illumination and changes in surface gloss. Based on the corrected and accurate feature map, the system can calculate the accurate position and posture information of the item to be sorted, thereby generating precise gripping instructions to guide the pneumatic robot to perform accurate gripping actions. This method avoids gripping deviations, empty grips, or collisions caused by inaccurate image information in traditional methods, significantly improving the success rate and efficiency of sorting operations, while reducing unexpected wear on the robot and extending the service life of the equipment. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a method for realizing the motion of a pneumatic manipulator based on industrial vision, provided in an embodiment of this application.

[0025] Figure 2 This is a schematic diagram of a pneumatic manipulator motion realization system based on industrial vision, provided as an embodiment of this application.

[0026] Labeling Explanation: 210, Anomaly Detection Module; 220, Execution Exploration and Image Acquisition Module; 230, Image Correction and Processing Module; 240, Capture Operation Execution Module. Detailed Implementation

[0027] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0028] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0029] In modern industrial automation, high-speed sorting operations place extremely high demands on the motion precision and continuity of pneumatic manipulators. Traditional control methods struggle to adapt to subtle changes in the working environment or the characteristics of the processed items, leading to performance degradation. For example, during long-term production line operation, upstream raw material suppliers may adjust the source of raw materials supplied for sorting, resulting in a loss of uniform surface gloss in new batches of items. This surface inhomogeneity is a significant change in the eyes of industrial vision systems. When the industrial vision system continues to use its preset fixed lighting scheme to acquire images of these new batches of items, the illumination light hitting the enhanced specular reflection areas on the item surface will reflect a large amount of light directly back to the industrial camera lens, causing local oversaturation or bright whitening in these areas of the image. This whitening effect makes the key feature points of the item blurry or even completely lost in the image, severely affecting the vision system's complete extraction of item feature information, making the image data no longer an accurate mapping of the object's true physical state. After receiving images containing these locally bright or blurred areas, the image recognition module's internal preset criteria for distinguishing targets from the background become inapplicable. When large areas of whitewash appear in an image, the brightness gradient information is destroyed, edges become blurred, or artifacts appear, making it difficult for the system to accurately delineate the true outline of the object. This directly leads to significant deviations in the calculation of the target object's position coordinates. The system may even misidentify the whitewashed areas of part of the object as part of the background, or identify a complete object as an incomplete target, thus outputting incorrect gripping position and posture information. The motion control program of the pneumatic robot generates inaccurate gripping paths and posture commands based on these erroneous object position coordinates provided by the vision system. In high-speed sorting operations, although the control system attempts to fine-tune the robot's movements through pneumatic feedback, when the target position itself is incorrect, these fine-tuning adjustments may actually cause the actual gripping point to deviate further from the ideal gripping point of the object. The gripper may not be able to completely cover the object, causing the object to slip, tilt, or even collide unexpectedly with the end gripper of the robot during gripping, or completely miss, causing the object to fall off the conveyor belt or be pushed away from the intended position. With the frequent occurrence of such misidentification and grasping failures, the number of defective products on the production line has increased significantly, directly impacting production efficiency and product qualification rate. A deeper problem is that repeated unexpected collisions or multiple grasping attempts subject the robotic arm's joint bearings and internal cylinder seals to impacts and friction exceeding normal limits, accelerating the wear of these critical physical components. This wear, in turn, further affects the long-term stability and motion accuracy of the robotic arm, creating a vicious cycle that leads to a continuous decline in the overall performance of the robotic arm.

[0030] In this regard, firstly, referring to Figure 1This application proposes a method for realizing the motion of a pneumatic manipulator based on industrial vision, including: S1. Collect images of the items to be sorted on the conveyor belt and determine whether there are visual anomalies based on the collected images. If there are visual anomalies, send a visual anomaly signal to the motion controller of the pneumatic manipulator. S2. In response to a visual anomaly signal, interrupt the current operation of the pneumatic manipulator, control the pneumatic manipulator to perform a preset non-contact probing motion, and collect multiple frames of images of the items to be sorted during the pneumatic manipulator's preset non-contact probing motion. S3. Analyze and determine the optical reflection characteristics of the items to be sorted in multiple frames of images. Based on the optical reflection characteristics, perform correction processing on the multiple frames of images to generate accurate feature maps of the items to be sorted. S4. Based on the precise feature map, calculate the accurate position and posture information of the items to be sorted, generate a gripping instruction based on the accurate position and posture information, and send it to the motion controller so that the motion controller controls the pneumatic robot to perform the gripping action according to the gripping instruction.

[0031] A pneumatic robot is an automated device driven by compressed air to perform operations such as gripping and handling. Its motion controller is an electronic unit responsible for receiving instructions and controlling the movement of each joint of the robot. Visual anomalies refer to situations in conventional image acquisition where image quality deteriorates due to factors such as changes in the surface characteristics of the object or uneven ambient lighting, making it impossible to accurately identify the object's features. Examples include local oversaturation, blurring, or loss of feature points in the image. Non-contact detection motion refers to the pneumatic robot moving along a preset path without touching the object to be sorted, to acquire multiple frames of images of the object under dynamic lighting conditions. Optical reflection characteristics refer to the physical properties of an object's surface, such as reflection, absorption, and scattering of light. These characteristics affect the brightness, contrast, and texture of the object in the image. Accurate feature maps are image data that, after correction processing, accurately reflect the key features of the object's geometry, edges, and texture, providing a reliable basis for subsequent position and orientation calculations. This method is typically implemented on an industrial automated sorting production line, which includes a conveyor belt, industrial cameras, lighting devices, a pneumatic robot, and a control system and data processing unit connecting the various components.

[0032] The proposed industrial vision-based pneumatic manipulator motion implementation method forms a closed-loop intelligent vision guidance system by introducing visual anomaly detection, non-contact exploration motion, and image correction processing based on optical reflection characteristics. When the system detects visual anomalies in routine image acquisition, it no longer blindly processes defective images but actively triggers the pneumatic manipulator's non-contact exploration motion. During the exploration process, by acquiring multiple frames of images and combining them with dynamic light and shadow changes, the optical reflection characteristics of the object can be captured more comprehensively and accurately. These optical reflection characteristics are used to correct the original image, thereby generating accurate feature maps that are unaffected by uneven lighting or changes in surface characteristics. Finally, based on these accurate feature maps, the pneumatic manipulator can obtain accurate position and posture information and perform precise grasping actions. The entire process effectively solves the problem of recognition errors and grasping failures caused by image quality degradation when dealing with objects with complex surface characteristics in traditional vision systems, significantly improving the reliability and efficiency of sorting operations.

[0033] The core innovation of this application lies in its breakthrough overcoming the limitations of traditional industrial vision systems, especially when dealing with objects with uneven surface gloss or complex optical reflection characteristics. Traditional methods often rely on fixed lighting conditions and preset image processing models. Once the characteristics of the object or the ambient lighting changes, problems such as image oversaturation, blurring, or loss of feature points can easily occur, leading to a decrease in recognition accuracy and thus affecting the success rate of the robotic arm's grasping.

[0034] In contrast, this application, by introducing a visual anomaly detection mechanism, can proactively identify image quality issues, avoiding the use of defective images for subsequent processing. More importantly, this application innovatively proposes non-contact probing motion and image correction processing based on optical reflection characteristics. By controlling the pneumatic manipulator to perform a preset non-contact probing motion and acquiring multiple frames of images under dynamic lighting conditions, the system can obtain the reflection information of the object under different lighting angles, thereby gaining a more comprehensive understanding of its optical characteristics. Subsequently, based on these optical reflection characteristics, the multiple frames of images are corrected, effectively removing or compensating for image defects caused by uneven lighting, generating high-precision feature maps. This proactive probing and intelligent correction strategy allows the system to proactively acquire and process high-quality visual data, rather than passively accepting defective images. As a result, the pneumatic manipulator can obtain more accurate object position and posture information, thereby performing more precise and reliable grasping actions, significantly reducing the grasping failure rate, improving production efficiency, and extending the manipulator's service life. This method not only solves the problem of identification and grasping caused by changes in the surface characteristics of objects in existing technologies, but also provides a more robust and adaptable vision guidance solution for the field of industrial automation.

[0035] In some embodiments described above, an industrial vision-based pneumatic manipulator motion implementation method is proposed. This method analyzes the optical reflection characteristics of items to be sorted by executing a preset non-contact probing motion and acquiring multiple frames of images. However, in practical applications, if the preset non-contact probing motion fails to sufficiently excite or capture the optical response of the item's surface—for example, when the item has a complex geometry, uneven surface gloss, or excessive local reflection—simple non-contact motion may not be sufficient to obtain rich and recognizable image data, thus affecting the accuracy of subsequent optical reflection characteristic analysis and potentially leading to inaccurate generation of precise feature maps.

[0036] In this regard, this application further proposes that the aforementioned preset non-contact detection motion includes controlling the pneumatic manipulator to move along a preset non-contact path, so that its end effector sweeps over the items to be sorted, thereby forming a dynamic light and shadow area on the surface of the items to be sorted.

[0037] Specifically, a pre-planned non-contact path refers to a trajectory designed to ensure that the end effector of a pneumatic robot does not make physical contact with the items to be sorted during its movement. This path is typically designed to allow the end effector to cover the area above the items in a controlled manner. The end effector sweeping over the items can be understood as the gripping components or tools of the pneumatic robot scanning above or to the side of the items. In practical applications, when the end effector moves over the items, it blocks some ambient light or fixed lighting, thus casting a moving shadow on the surface of the items or changing the intensity and direction of light on the surface, thereby creating a dynamic light and shadow area that changes over time. The purpose is to stimulate optical responses such as reflection, scattering, or absorption of light in different areas of the item's surface through this controlled lighting change, so that these dynamic changes can be captured in subsequent multi-frame images.

[0038] This application's solution effectively solves the problem of comprehensively capturing the optical characteristics of items under traditional fixed lighting by creating a dynamic light and shadow area above the items to be sorted using the end effector of a pneumatic manipulator. The dynamic light and shadow area allows different areas of the item's surface to be illuminated by varying intensities and directions at different times. This dynamic change reveals details on the item's surface that might be obscured, overexposed, or underexposed under fixed lighting conditions. For example, for items with highly reflective properties, localized oversaturated areas may appear under fixed lighting, leading to the loss of feature information; however, when the dynamic light and shadow area passes over these oversaturated areas, the lighting conditions change, allowing the previously saturated pixel values ​​to recover to a recognizable range, thus revealing their true optical reflective properties. Furthermore, for items with complex textures or uneven gloss, the dynamic light and shadow changes can more comprehensively reflect their surface response to light, providing a richer and more accurate data foundation for subsequent analysis of brightness, contrast, and positional changes in multi-frame images.

[0039] Through the aforementioned technical solution, the pre-set non-contact probing motion is no longer merely a simple spatial movement, but is endowed with the function of actively creating a dynamic lighting environment. This significantly improves the information richness and quality of the acquired multi-frame images, making subsequent analysis of the optical reflectivity of the items to be sorted more accurate and comprehensive. This method is particularly suitable for handling items with challenging optical characteristics, such as those with high reflectivity, low contrast, or complex surface textures, effectively avoiding inaccurate feature extraction caused by initial visual anomalies. Ultimately, it ensures that the accurate position and orientation information of the items to be sorted can be precisely calculated, thereby improving the success rate and robustness of the pneumatic robotic arm's grasping operation.

[0040] In some preferred embodiments, it is assumed that the item to be sorted is a metal part with irregular reflective areas on its surface. During the initial image acquisition, due to the influence of fixed lighting, some areas of the metal part may exhibit strong specular reflection, leading to localized oversaturation of the image and making it impossible to clearly identify its edges and surface texture. At this time, a pneumatic manipulator is controlled to perform a preset non-contact probing motion, with its end effector sweeping over the metal part along a preset non-contact path. As the end effector moves, a dynamic shadow area forms and moves on the surface of the metal part. The vision system continuously acquires multiple frames of images during this process. In these multiple frames, the originally oversaturated areas gradually reveal their true brightness distribution and texture details as the shadow moves, because the lighting conditions of this area change in different frames. By analyzing these dynamic changes, such as the fluctuation range of brightness values ​​and the change pattern of contrast, the system can accurately determine the optical reflection characteristics of the metal part, thereby effectively removing or compensating for localized oversaturation caused by specular reflection in subsequent image correction processing, ultimately generating a clear and accurate feature map that reflects the characteristics of the metal part.

[0041] Specifically, the steps of analyzing and determining the optical reflection characteristics of the items to be sorted in the multi-frame images include: based on the multi-frame images, identifying the key feature points of the items to be sorted through a feature point detection algorithm, and calculating the changes in brightness, contrast, and position of the key feature points in the dynamic light and shadow area as optical reflection characteristics.

[0042] Feature point detection algorithms can be understood as an image processing technique aimed at automatically identifying pixel regions with stable visual information such as unique textures, corners, or edges in an image. In practical applications, feature point detection algorithms can be any one or a combination of algorithms such as Scale Invariant Feature Transform (SIFT), Speed-Up Robust Feature Transform (SURF), Oriented Fast and Rotationally Reducing (ORB), etc., to ensure that key feature points can still be stably identified and tracked under different viewing angles, scales, and lighting conditions. Key feature points refer to points on the surface of items to be sorted that have significant visual characteristics, effectively representing the local geometry and material properties of the items.

[0043] Furthermore, the dynamic light and shadow area refers to the region of light and shadow variation formed on the surface of the item during the aforementioned pre-defined non-contact probing motion, as the end effector of the pneumatic manipulator sweeps over the item to be sorted. The light and shadow variations in this area reflect the three-dimensional geometry and optical reflection characteristics of the item's surface. Changes in brightness, contrast, and position refer to the changes in pixel brightness values, local contrast, and their positions in the image coordinate system of key feature points in different frame images under the dynamic light and shadow area. These changes, once quantified, can serve as the optical reflection characteristics of the item to be sorted, characterizing the absorption, reflection, and scattering behavior of the item's surface, such as specular reflection and diffuse reflection.

[0044] This application's solution overcomes the limitations of traditional methods in accurately capturing the true visual information of objects under complex lighting conditions or when the surface characteristics of the object are uneven. Specifically, when a pneumatic manipulator performs a preset non-contact probing motion, the dynamic light and shadow area formed by its end effector on the surface of the object to be sorted can stimulate optical responses in different areas of the object's surface. Key feature points identified by the feature point detection algorithm can stably capture these local responses. For example, on a high-gloss surface, the brightness change of key feature points is more drastic when light and shadow pass over them, the contrast may momentarily increase or decrease, and their position in the image may produce parallax due to the movement of the highlight area. Conversely, for diffuse surfaces, the brightness change is relatively gradual. By quantifying these changes in brightness, contrast, and position, the optical reflection characteristics of the object to be sorted can be accurately constructed, thus providing a reliable basis for subsequent image correction processing.

[0045] Specifically, the steps described above, which involve correcting multiple frames of images based on optical reflection characteristics to generate accurate feature maps of the items to be sorted, include: Based on optical reflection characteristics, image fusion or de-reflection algorithms are used to remove or compensate for local oversaturated or blurred areas in multiple frames of images caused by fixed lighting and uneven gloss, thereby generating accurate feature maps.

[0046] Image fusion algorithms combine information from multiple images to generate a single image containing more complete and accurate information. In practical applications, techniques such as multi-exposure fusion, multi-focus fusion, or multispectral fusion can be employed. For example, when oversaturation or blurring exists in different regions of multiple frames, image fusion algorithms can select the best-quality regions from each frame for combination, thus eliminating the limitations of a single image. Dereflection algorithms use specific image processing techniques to separate and remove interference caused by reflected light from an image, obtaining the object's inherent color and texture information. Specifically, methods based on physical models, such as using the Lambertian reflection model and specular reflection model, can be employed; or deep learning-based methods can be used to train neural networks to identify and remove reflective components. The aim is to eliminate or significantly reduce the negative impact of reflection phenomena such as highlights and glare on image feature extraction.

[0047] Localized oversaturation or blurring areas caused by fixed lighting and uneven gloss refer to situations in industrial vision inspection environments where, due to fixed light source positions, uneven light intensity, and differences in gloss levels of the surface materials of the items to be sorted (such as metal and plastic), certain areas in the image may have excessively high (oversaturated) or low pixel brightness values. This makes key features such as texture and edges of the items unclear, thus affecting the accuracy of subsequent feature recognition and positioning. Removal or compensation operations aim to address these defective areas. Removal typically refers to directly eliminating or replacing severely affected pixels or areas; compensation refers to adjusting the pixel values ​​of these areas using algorithms to restore them to the normal range, thereby revealing the occluded or distorted features.

[0048] This application's solution effectively addresses the problem of localized oversaturation or blurry areas in multi-frame images under fixed lighting and uneven gloss conditions by introducing image fusion or de-reflection algorithms. Specifically, when the surface of the item to be sorted has high gloss reflection or uneven illumination, a single image often fails to accurately capture all its features. Image fusion algorithms can integrate effective information from different frames; for example, an oversaturated area in one frame may appear normal in another. Fusion can capture the best representation, generating a more comprehensive image. Simultaneously, de-reflection algorithms can directly and specifically separate and remove reflective components from the image, clearly presenting the item's inherent features and avoiding interference from reflected light on feature extraction. Therefore, regardless of whether image fusion or de-reflection algorithms are used, based on established optical reflection characteristics, refined correction processing of multiple frames can be performed, ensuring that the final generated accurate feature map accurately reflects the true geometric and texture information of the item to be sorted.

[0049] Specifically, the steps for determining the presence of visual abnormalities based on the acquired images can be broken down into the following methods: Determine whether there are local areas in the acquired image where the pixel brightness value reaches or exceeds a preset saturation threshold, and / or whether the edge sharpness of key features of objects in the image is lower than a preset sharpness threshold.

[0050] The determination of whether there are localized areas in the acquired image where pixel brightness values ​​reach or exceed a preset saturation threshold involves analyzing the acquired image to identify regions where pixel brightness values ​​are abnormally high, reaching or exceeding the pre-set saturation threshold. This phenomenon typically indicates overexposure, strong reflection, or highlight clipping in a localized area of ​​the image, potentially leading to the loss or severe distortion of item feature information in that area. The preset saturation threshold can be empirically set based on the actual industrial lighting environment, the material characteristics of the items to be sorted, and the dynamic range of the image sensor, or determined through learning and training on a large amount of normal and abnormal image data.

[0051] Furthermore, determining that the edge sharpness of key features of an item in an image is below a preset sharpness threshold means processing the acquired image to assess whether the edges of key features (such as their contours, textures, or specific geometric shapes) of the item to be sorted are sufficiently sharp. When the edge sharpness is below the preset threshold, it indicates that the image may have problems such as blurriness, out-of-focus issues, motion blur, or insufficient lighting, making it difficult to accurately identify and locate the boundaries and details of the item. In practical applications, edge sharpness can be quantified using various image processing algorithms, such as using gradient operators (e.g., Sobel, Canny) to calculate edge intensity, or using Fourier transform to analyze the high-frequency components of the image. The preset sharpness threshold can be adjusted according to the requirements for grasping accuracy and image quality.

[0052] This application's solution makes the identification of visual anomalies more accurate and automated by setting specific judgment criteria. When the pixel brightness value of a local area in an image is too high, reaching or exceeding a preset saturation threshold, it indicates that the area may have strong reflection or overexposure, causing the true color, texture, and other information of the object to be obscured, thus affecting the accuracy of subsequent feature extraction and positioning. Simultaneously, when the edge sharpness of key features of an object is lower than a preset sharpness threshold, it means that the image quality is poor, possibly due to blurring, defocusing, or insufficient lighting, making it difficult to accurately identify the object's outline and details. By combining these two judgment criteria, visual anomalies that may lead to the failure of subsequent grasping operations can be effectively identified, providing a triggering basis for subsequent non-contact exploration movements.

[0053] In some embodiments described above in this application, multi-frame images are corrected by analyzing the optical reflection characteristics of the items to be sorted to generate accurate feature maps. However, in actual industrial environments, factors such as fixed lighting on the conveyor belt, environmental vibration, and slight deviations in the actual movement trajectory of the pneumatic manipulator may cause additional interference to the acquired multi-frame images. This makes it difficult for correction processing based solely on optical reflection characteristics to completely eliminate these complex effects, thereby affecting the quality of the generated accurate feature maps and the accuracy of subsequent gripping actions.

[0054] In this regard, this application further proposes that, before performing the aforementioned preset non-contact probing motion, it also includes: Based on the preset non-contact path, the geometric model of the end effector of the pneumatic manipulator, and the fixed lighting parameters on the conveyor belt, the predicted sequence of light and shadow features of the dynamic light and shadow area changing over time under ideal conditions is pre-calculated and stored. The light and shadow feature prediction sequence is compared with the optical reflection characteristics of multiple frames of images, and the multiple frames of images are preprocessed based on the comparison results. Based on optical reflection characteristics, the pre-processed multi-frame images are corrected to generate accurate feature maps of the items to be sorted.

[0055] Specifically, pre-calculating and storing the predicted sequence of light and shadow features of dynamic light and shadow regions under ideal conditions over time refers to simulating and predicting how dynamic light and shadow regions will form on the surface of the items to be sorted under ideal, undisturbed conditions, and how the brightness, shape, and position of these light and shadow regions will change over time, using known motion paths, the geometry of the end effector, and ambient lighting conditions, through physical optical models or simulation software. This predicted data is stored as a benchmark for subsequent comparisons. Comparing the predicted light and shadow feature sequence with the optical reflection characteristics of multiple frames of images involves comparing the optical reflection characteristics of the items to be sorted (e.g., brightness, contrast, changes in the position of key feature points, etc.) reflected in the actually acquired multiple frames of images with the pre-calculated ideal light and shadow feature prediction sequence. This comparison aims to identify differences between the actual and ideal conditions, which may originate from environmental vibrations, minor deviations in the pneumatic manipulator's trajectory, etc. In practical applications, preprocessing the multiple frames of images based on the comparison results involves making preliminary adjustments or corrections to the originally acquired multiple frames of images based on the differences identified in the comparison results. For example, if the comparison results show random noise, noise reduction can be performed; if systematic biases exist, lighting compensation or geometric correction can be performed. The aim is to eliminate or reduce the impact of non-ideal factors on image quality, providing a cleaner input for subsequent precise correction processing. Furthermore, based on optical reflection characteristics, the preprocessed multi-frame images are corrected to generate precise feature maps of the items to be sorted. This refers to using the optical reflection characteristics (such as material and surface texture) of the items themselves after preprocessing to perform refined image correction. Preprocessing solves the problem of external interference, while this correction processing focuses on the visual characteristics of the items themselves, ensuring that the final precise feature map accurately reflects the geometric shape and surface details of the items, unaffected by uneven lighting or reflective properties.

[0056] In some embodiments described above, a method is proposed to pre-calculate and store a predicted sequence of light and shadow features of a dynamic light and shadow region under ideal conditions, based on a preset non-contact path, the geometric model of the end effector of a pneumatic manipulator, and fixed lighting parameters on a conveyor belt. This predicted sequence is then compared with the optical reflection characteristics of multiple frames of images, and the images are preprocessed based on the comparison results to generate an accurate feature map. However, in real industrial environments, slight deviations in environmental vibrations and the movement trajectory of the pneumatic manipulator can lead to random noise and systematic biases in the comparison results, affecting the accuracy of preprocessing and the quality of the final accurate feature map. Without addressing these interfering factors, even with preprocessing, the generated accurate feature map may still contain errors, thus affecting the accuracy of subsequent calculations of item position and orientation information.

[0057] In response, this application further proposes a step of comparing the above-mentioned light and shadow feature prediction sequence with the optical reflection characteristics of the above-mentioned multi-frame images, including: By comparing the optical reflection characteristics with the light and shadow feature prediction sequence, random high-frequency noise caused by environmental vibration and / or systematic deviations caused by deviations in the actual movement trajectory of the pneumatic manipulator can be identified. Preprocessing of multiple frames of images based on random high-frequency noise and / or systematic bias.

[0058] Specifically, comparing the aforementioned optical reflection characteristics with the aforementioned light and shadow feature prediction sequence involves comparing the changes in brightness, contrast, and position of the actual items to be sorted in the dynamic light and shadow area (i.e., optical reflection characteristics) with the pre-calculated light and shadow feature prediction sequence under ideal conditions, frame by frame or feature point by feature point. During this comparison, random high-frequency noise caused by environmental vibration and / or systematic deviations caused by deviations in the actual movement trajectory of the pneumatic manipulator can be identified. Random high-frequency noise caused by environmental vibration typically manifests as rapid, irregular fluctuations in pixel values ​​across multiple frames, which may be due to minor vibrations in the image acquisition device or the items to be sorted themselves within a short period. Identifying this noise can be done through frequency domain analysis, statistical methods (e.g., calculating the variance of inter-frame differences), or machine learning-based models. Systematic deviations caused by deviations in the actual movement trajectory of the pneumatic manipulator refer to subtle and continuous deviations between the actual movement trajectory and the preset path when the pneumatic manipulator executes a preset non-contact path due to mechanical errors, control precision limitations, or load variations. These deviations result in systematic differences between the shape, position, or intensity of the dynamic light and shadow area and the prediction sequence. Such biases can be identified by comparing the actual lighting characteristics with the long-term trend, average offset, or using state estimation algorithms such as Kalman filtering. Once these random high-frequency noises and / or systematic biases are identified, targeted preprocessing of multiple frames can be performed based on these identification results. For example, for random high-frequency noise, smoothing filters (such as mean filtering, Gaussian filtering) or median filtering can be used to reduce its impact; for systematic biases, image registration, geometric correction, or illumination compensation can be performed to eliminate or reduce lighting distortion caused by trajectory deviations.

[0059] However, in practical applications, simply identifying these noises and biases is not enough to completely eliminate their negative impact on image quality and subsequent feature extraction. Without targeted processing measures, the preprocessing effect may be poor, which in turn affects the final grasping accuracy.

[0060] In this regard, this application further proposes preprocessing steps for the aforementioned multi-frame images, including: To address random high-frequency noise, a time-domain filtering method is used to smooth multiple frames of images. And / or, to address systematic biases, perform light and shadow compensation on multiple frames of images by adjusting a preset image analysis model.

[0061] Specifically, the above steps for preprocessing multiple frames of images based on random high-frequency noise and / or systematic bias include: To address random high-frequency noise, a time-domain filtering method is used to smooth multiple frames of images. And / or, to address systematic biases, perform light and shadow compensation on multiple frames of images by adjusting a preset image analysis model.

[0062] Random high-frequency noise typically manifests as irregular, rapidly changing pixel value fluctuations in an image, which may be caused by transient interference in the environment, random errors in the sensor itself, or accidental distortions during data transmission. To effectively suppress this type of noise, temporal filtering methods can be used to smooth multiple frames of the image. Temporal filtering refers to processing continuously acquired image frames over time; for example, techniques such as moving average filtering, median filtering, or Gaussian filtering can be employed. By weighted averaging or statistically processing the pixel values ​​of adjacent frames, the impact of random noise on a single frame can be effectively reduced, resulting in a smoother image and preservation of detail.

[0063] Furthermore, systematic bias refers to the persistent deviation between the actual movement trajectory of the pneumatic manipulator and the preset trajectory, or the regular shift in image features caused by factors such as lighting conditions and surface characteristics of objects. This type of bias may lead to systematic differences in the shape, position, or brightness of the light and shadow areas compared to the predicted sequence. To compensate for this systematic bias, a preset image analysis model can be adjusted to perform light and shadow compensation on multiple frames of images. Specifically, the image analysis model can be a model based on physical optics principles, used to describe the process of light reflection on the surface of an object and shadow formation. By comparing the actually acquired optical reflection characteristics with the predicted sequence of light and shadow features, the magnitude and direction of the systematic bias can be quantified, and then the relevant parameters in the image analysis model (such as light source position, intensity, and surface reflectance coefficient of the object) can be adaptively adjusted. This adjustment allows the model to more accurately simulate the light and shadow distribution in the current actual environment, thereby performing precise light and shadow compensation on multiple frames of images and eliminating light and shadow distortion caused by systematic bias.

[0064] In some preferred embodiments, it is assumed that during the non-contact exploration motion of the pneumatic manipulator, vibrations from other equipment in the workshop cause random pixel jitter in the image acquisition device within a short period of time, resulting in random high-frequency noise. Simultaneously, due to slight wear on the joints of the pneumatic manipulator after prolonged operation, a small, continuous, systematic deviation occurs between the actual motion trajectory and the preset trajectory, leading to a regular difference between the shape and brightness of the dynamic light and shadow area and the ideal prediction.

[0065] At this point, the preprocessing method of this application first performs temporal median filtering on the acquired multi-frame images to address random high-frequency noise. Specifically, for each pixel in each frame, its new pixel value is determined by the median of the corresponding pixel values ​​in the current frame and several frames before and after it (e.g., the previous and next frames). This processing can effectively remove salt-and-pepper noise or transient bright spots in the image, making the image background and object edges smoother.

[0066] Subsequently, to address systematic biases, the system compares the optical reflectance characteristics of the multi-frame images after temporal filtering with a pre-stored sequence of predicted light and shadow features. This comparison identifies persistent positional shifts in the light and shadow regions and overall differences in brightness. Based on this bias information, the system adjusts the light source position parameters and ambient light intensity parameters in the preset image analysis model, enabling the adjusted model to more accurately simulate actual lighting and motion conditions. Using this adjusted model, light and shadow compensation is applied to the multi-frame images, for example, through local brightness correction or contrast enhancement algorithms, to eliminate light and shadow distortions caused by systematic biases, thereby generating more accurate feature maps of the items to be sorted.

[0067] Specifically, the above-mentioned grasping instructions can be further refined to include more precise motion control information.

[0068] The grab command includes the best approach path, grab point, and grab posture.

[0069] The optimal approach path refers to the best trajectory for the pneumatic robot's end effector to move from its current position to above or near the gripping point before contacting the item to be sorted. Its purpose is to avoid collisions with the surrounding environment or items, while ensuring a smooth and efficient gripping process. For example, it can be calculated using path planning algorithms (such as RRT, PRM, etc.) based on the robot's kinematic model, workspace constraints, and item distribution. The gripping point is the specific location where the pneumatic robot's end effector contacts the item to be sorted and applies gripping force. This point is typically the most stable, easiest to grip, and least likely to be damaged area on the item. Its purpose is to ensure a high success rate of gripping and prevent the item from slipping or deforming during the gripping process. For example, it can be determined based on the item's geometry, center of gravity, and material properties through visual recognition and mechanical analysis. The gripping posture refers to the spatial orientation and angle of the pneumatic robot's end effector relative to the item to be sorted at the gripping point. Its purpose is to allow the gripping tool (such as a suction cup, gripper, etc.) to cooperate with the item in the most suitable way for a stable grip. For example, the orientation can be determined using an attitude estimation algorithm based on the shape of the item, the type of gripping tool, and the requirements for subsequent sorting or placement.

[0070] Secondly, referring to Figure 2 The specific embodiments of this application also disclose a pneumatic manipulator motion realization system based on industrial vision, the system comprising: The anomaly detection module 210 is used to acquire images of the items to be sorted on the conveyor belt and determine whether there is a visual anomaly based on the acquired images. If there is a visual anomaly, a visual anomaly signal is sent to the motion controller of the pneumatic manipulator. The detection and image acquisition module 220 is used to interrupt the current operation of the pneumatic manipulator in response to a visual abnormality signal, control the pneumatic manipulator to perform a preset non-contact detection motion, and acquire multiple frames of images of the items to be sorted during the pneumatic manipulator's preset non-contact detection motion. The image correction processing module 230 is used to analyze and determine the optical reflection characteristics of the items to be sorted in multiple frames of images, and to correct the multiple frames of images based on the optical reflection characteristics in order to generate an accurate feature map of the items to be sorted. The gripping operation execution module 240 is used to calculate the accurate position and posture information of the items to be sorted based on the precise feature map, generate gripping instructions based on the accurate position and posture information, and send them to the motion controller so that the motion controller controls the pneumatic manipulator to perform gripping actions according to the gripping instructions.

[0071] This system, through its modular design, aims to solve the problem that traditional industrial vision systems suffer from decreased image recognition accuracy when faced with uneven surface characteristics of items to be sorted or changes in ambient lighting, which in turn leads to the failure of pneumatic robotic arms to grasp the items.

[0072] The anomaly detection module 210 first performs a preliminary analysis of the acquired images, proactively identifying potential visual anomalies to avoid using defective images for subsequent processing. Once an anomaly is detected, the detection and image acquisition module 220, in conjunction with the pneumatic manipulator, performs a non-contact detection motion, dynamically acquiring multiple frames of images of the object under different lighting conditions to capture its true optical reflection characteristics. Subsequently, the image correction and processing module 230 intelligently corrects the multiple frames of images using these optical reflection characteristics, generating a high-precision feature map to ensure the reliability of the visual data. Finally, the grasping operation execution module 240 calculates the accurate position and posture information of the object based on the corrected feature map and generates precise grasping commands to guide the pneumatic manipulator to perform efficient and stable grasping actions. Through the close collaboration of these modules, this system can significantly improve the robustness and accuracy of the pneumatic manipulator in complex industrial environments, thereby increasing production efficiency and extending equipment life.

[0073] Specifically, the anomaly detection module 210 is configured to acquire images of the items to be sorted on the conveyor belt and determine whether visual anomalies exist based on the acquired images. If a visual anomaly is found, the module sends a visual anomaly signal to the motion controller of the pneumatic manipulator. This module can be implemented as a separate image processing unit, for example, an embedded processor running a preset image analysis algorithm, or a host computer software module connected to an industrial camera, responsible for receiving image data in real time and performing anomaly detection. The specific methods for determining visual anomalies, such as detecting whether the pixel brightness value of a local area reaches or exceeds a preset saturation threshold, and / or whether the edge sharpness of key features of the item is lower than a preset sharpness threshold, have been described in detail in the above embodiments and will not be repeated here.

[0074] Furthermore, the detection and image acquisition module 220 is configured to interrupt the current operation of the pneumatic manipulator in response to a visual anomaly signal, control the pneumatic manipulator to perform a preset non-contact detection motion, and acquire multiple frames of images of the items to be sorted during the preset non-contact detection motion. This module can be implemented by a subroutine in the manipulator control system or by a separate motion planning and vision acquisition coordinator. For example, when a visual anomaly signal is received, the module sends a command to the motion controller to move the pneumatic manipulator along a preset non-contact path, causing its end effector to pass over the items to be sorted. Simultaneously, the module coordinates an industrial camera to continuously acquire multiple frames of images at a high frame rate. A detailed description of this dynamic acquisition process has been provided in the above embodiments.

[0075] Furthermore, the image correction processing module 230 is configured to analyze and determine the optical reflectance characteristics of the items to be sorted in multiple frames of images. Based on these optical reflectance characteristics, the module corrects the multiple frames of images to generate an accurate feature map of the items to be sorted. This module can be a high-performance image processing server or a dedicated image processor integrated into a vision system. Its working principle includes detecting feature points in multiple frames of images and calculating the changes in brightness, contrast, and position of key feature points in dynamic lighting and shadow areas to determine the optical reflectance characteristics of the items. Subsequently, based on these optical reflectance characteristics, the module can remove or compensate for local oversaturated or blurred areas in the multiple frames of images caused by fixed lighting and uneven gloss through image fusion or dereflection algorithms, ultimately generating an accurate feature map. The details of these image processing algorithms have been described in the above embodiments.

[0076] Finally, the gripping operation execution module 240 is configured to calculate the accurate position and orientation information of the item to be sorted based on the precise feature map, generate gripping instructions based on the accurate position and orientation information, and send them to the motion controller. This allows the motion controller to control the pneumatic manipulator to perform the gripping action according to the gripping instructions. This module can function as a high-level functional module of the motion controller or as an independent task planning and execution unit. It uses algorithms based on feature matching or deep learning to extract information such as the geometric center and principal axis direction of the item from the precise feature map, thereby calculating its accurate position and orientation in three-dimensional space. Subsequently, this module generates gripping instructions containing detailed information such as the optimal approach path, gripping point, and gripping orientation, and sends them to the motion controller to ensure that the pneumatic manipulator can complete the gripping task in the optimal manner.

[0077] The system of this application, by introducing an anomaly detection module 210, can proactively identify image quality problems, avoiding the use of defective images for subsequent processing. Furthermore, the collaborative work of the detection and image acquisition module 220 and the image correction processing module 230 enables the system to proactively acquire multi-frame images of the object under dynamic lighting conditions through non-contact detection motion, and intelligently correct them based on their optical reflection characteristics, thereby generating a high-precision feature map. As a result, the grasping operation execution module 240 can obtain more accurate object position and posture information and execute more precise and reliable grasping actions. This system architecture of proactive perception, intelligent correction, and precise execution significantly improves the robustness and success rate of pneumatic manipulators in complex and variable industrial environments, effectively reduces the defect rate of the production line, and extends the service life of the equipment, demonstrating significant technological progress.

[0078] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for realizing the motion of a pneumatic manipulator based on industrial vision, characterized in that, include: Images of items to be sorted on the conveyor belt are captured, and visual anomalies are determined based on the captured images. If visual anomalies are found, a visual anomaly signal is sent to the motion controller of the pneumatic manipulator. In response to the visual anomaly signal, the current operation of the pneumatic manipulator is interrupted, the pneumatic manipulator is controlled to perform a preset non-contact probing motion, and multiple frames of images of the items to be sorted are acquired during the pneumatic manipulator's preset non-contact probing motion. The optical reflection characteristics of the items to be sorted in the multi-frame images are analyzed and determined. Based on the optical reflection characteristics, the multi-frame images are corrected to generate an accurate feature map of the items to be sorted. Based on the precise feature map, the accurate position and orientation information of the item to be sorted are calculated. A gripping command is generated based on the accurate position and orientation information and sent to the motion controller, so that the motion controller controls the pneumatic manipulator to perform a gripping action according to the gripping command.

2. The method for realizing the motion of a pneumatic manipulator based on industrial vision according to claim 1, characterized in that, The preset non-contact detection motion includes controlling the pneumatic manipulator to move along a preset non-contact path, so that its end effector sweeps over the items to be sorted, thereby forming a dynamic light and shadow area on the surface of the items to be sorted.

3. The method for realizing the motion of a pneumatic manipulator based on industrial vision according to claim 2, characterized in that, The step of analyzing and determining the optical reflectance characteristics of the items to be sorted in the multi-frame images includes: Based on the multi-frame images, the key feature points of the items to be sorted are identified by the feature point detection algorithm, and the changes in brightness, contrast and position of the key feature points under the dynamic light and shadow area are calculated as the optical reflection characteristics.

4. The method for realizing the motion of a pneumatic manipulator based on industrial vision according to claim 1, characterized in that, The step of correcting the multi-frame images based on the optical reflection characteristics to generate a precise feature map of the items to be sorted includes: Based on the optical reflection characteristics, the precise feature map is generated by removing or compensating for local oversaturated or blurred areas in the multi-frame images caused by fixed lighting and uneven gloss through image fusion or dereflection algorithms.

5. The method for realizing the motion of a pneumatic manipulator based on industrial vision according to claim 1, characterized in that, The step of determining whether there is a visual abnormality based on the acquired image includes: Determine whether there are local areas in the acquired image where the pixel brightness value reaches or exceeds a preset saturation threshold, and / or whether the edge clarity of key features of objects in the image is lower than a preset clarity threshold.

6. The method for realizing the motion of a pneumatic manipulator based on industrial vision according to claim 1, characterized in that, Before performing the preset non-contact probing motion, the following is also included: Based on the preset non-contact path, the geometric model of the end effector of the pneumatic manipulator, and the fixed lighting parameters on the conveyor belt, the predicted sequence of light and shadow features of the dynamic light and shadow area changing over time under ideal conditions is pre-calculated and stored. The light and shadow feature prediction sequence is compared with the optical reflection characteristics of the multi-frame images, and the multi-frame images are preprocessed based on the comparison results. Based on the optical reflection characteristics, the preprocessed multi-frame images are corrected to generate accurate feature maps of the items to be sorted.

7. The method for realizing the motion of a pneumatic manipulator based on industrial vision according to claim 6, characterized in that, The step of comparing the predicted light and shadow feature sequence with the optical reflectance characteristics of the multi-frame images includes: By comparing the optical reflection characteristics with the light and shadow feature prediction sequence, random high-frequency noise caused by environmental vibration and / or systematic deviations caused by deviations in the actual movement trajectory of the pneumatic manipulator can be identified. The multi-frame images are preprocessed based on the random high-frequency noise and / or the systematic bias.

8. The method for realizing the motion of a pneumatic manipulator based on industrial vision according to claim 7, characterized in that, The step of preprocessing the multi-frame images based on the random high-frequency noise and / or the systematic bias includes: To address the random high-frequency noise, the multi-frame images are smoothed using a time-domain filtering method; and / or, to address the systematic deviation, the multi-frame images are compensated for lighting and shadow by adjusting a preset image analysis model.

9. The method for realizing the motion of a pneumatic manipulator based on industrial vision according to claim 1, characterized in that, The grabbing command includes the optimal approach path, grabbing point, and grabbing posture.

10. A pneumatic manipulator motion realization system based on industrial vision, characterized in that, The system includes: The anomaly detection module is used to acquire images of items to be sorted on the conveyor belt and determine whether there is a visual anomaly based on the acquired images. If the visual anomaly exists, a visual anomaly signal is sent to the motion controller of the pneumatic manipulator. The detection and image acquisition module is used to interrupt the current operation of the pneumatic manipulator in response to the visual abnormality signal, control the pneumatic manipulator to perform a preset non-contact detection movement, and acquire multiple frames of images of the items to be sorted during the pneumatic manipulator's preset non-contact detection movement. The image correction processing module is used to analyze and determine the optical reflection characteristics of the items to be sorted in the multi-frame images, and to perform correction processing on the multi-frame images based on the optical reflection characteristics to generate an accurate feature map of the items to be sorted. The grasping operation execution module is used to calculate the accurate position and posture information of the item to be sorted based on the precise feature map, generate a grasping command based on the accurate position and posture information, and send it to the motion controller so that the motion controller controls the pneumatic manipulator to perform a grasping action according to the grasping command.

Images