A vision-guided method and system for a robotic arm

Abstract

This invention provides a visual guidance method and system for a robotic arm, relating to the field of robotic arm technology. This invention combines binocular stereo depth maps for correlation analysis to obtain fused depth, effectively reducing single-camera ranging errors and improving measurement accuracy in three-dimensional space. Simultaneously, based on the fused three-dimensional spatial coordinates and grayscale information, it dynamically calculates the illumination deviation and uniformity of the working area and generates a dynamic supplementary lighting formula. Based on this formula, by adjusting the brightness and angle of the supplementary light source, the illumination of the working area is optimized in real time, avoiding the impact of local overexposure and insufficient illumination on visual guidance. This significantly improves the ranging accuracy and image quality of the robotic arm vision system in complex lighting environments, enhancing the stability and reliability of visual guidance.

Description

Technical Field

[0001] This invention relates to the field of robotic arm technology, specifically to a visual guidance method and system for a robotic arm. Background Technology

[0002] A robotic arm is a robotic device composed of multi-degree-of-freedom joints that can simulate the movement functions of a human arm, enabling various automated operations such as grasping, handling, assembly, and welding. It typically consists of actuators, sensors, a control system, and actuators, possessing high precision, high repeatability, and high load capacity, and is widely used in industrial manufacturing, medical surgery, and logistics warehousing. Robotic arms often rely on vision systems for judgment during operation, using these systems to determine working conditions and perform baseline calibration, forming the foundational system of the robotic arm.

[0003] In visual guidance and detection tasks, the uniformity and stability of illumination conditions directly affect the image quality and the accuracy of subsequent algorithms. Typical robotic arm visual brightness guidance often relies on fixed light sources or simple manual adjustments, lacking a dynamic adaptive adjustment mechanism for illumination uniformity and brightness changes. Although some systems have adjustment functions, they usually use a single brightness standard or fixed parameters, making it difficult to simultaneously consider the uniformity of local brightness and the deviation of overall brightness. This results in unsatisfactory supplementary lighting effects, affecting the robustness and accuracy of visual recognition.

[0004] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a visual guidance method and system for a robotic arm to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A vision-guided method for a robotic arm, comprising the following steps:

[0008] S1. Set up three industrial cameras and fill lights at the end of the robotic arm. Determine the intrinsic and extrinsic parameters of the industrial cameras using the Zhang Zhengyou checkerboard calibration method. Construct a three-dimensional coordinate system based on the intrinsic and extrinsic parameters.

[0009] S2. Based on three industrial cameras, RGB images and binocular stereo depth maps of the working area of ​​the robotic arm are acquired simultaneously. The RGB images are corrected using a binocular stereo vision method. Correlation analysis is performed based on the binocular stereo depth map to generate a fused depth. The fused depth is used to improve the ranging accuracy.

[0010] S3. Perform data processing based on RGB image and fusion depth, correspond RGB image pixels to spatial coordinates in a three-dimensional spatial coordinate system, and calculate the single-channel grayscale of each spatial coordinate. The single-channel grayscale is used to reflect the grayscale at the corresponding spatial coordinate.

[0011] S4. Perform data processing based on single-channel grayscale to obtain local average grayscale, perform numerical analysis based on local average grayscale, generate deviation, compare with a set threshold, and output whether supplementary lighting is needed.

[0012] S5. When supplementary lighting is required for the output, a correlation analysis is performed based on the local average gray level and the single-channel gray level to generate uniformity. The uniformity is used to reflect the uniformity of the brightness in the working area. A dynamic supplementary lighting formula is generated based on the uniformity and deviation. The supplementary lighting device performs supplementary lighting based on the dynamic supplementary lighting formula.

[0013] S6. After the supplementary adjustments are completed, the robotic arm continues to perform operations based on vision-guided technology.

[0014] Furthermore, the expression for the intrinsic parameter matrix K is:

[0015] ;

[0016] in, , These represent the focal lengths of the corresponding industrial cameras in the horizontal and vertical directions. The horizontal and vertical coordinates of the principal point of the industrial camera;

[0017] The expression for the extrinsic parameter matrix T is:

[0018] ;

[0019] Where R is the rotation matrix, which reflects the spatial rotation of the industrial camera's camera coordinate system relative to the world coordinate system, and t is the translation vector, which reflects the spatial translation of the industrial camera's camera coordinate system relative to the world coordinate system.

[0020] Furthermore, the two industrial cameras are designated as Industrial Camera A and Industrial Camera B, respectively indexed by subscripts A and B. The data collected by Industrial Camera A and Industrial Camera B are projected onto the target surface via a projector, displaying coded optical patterns. The three industrial cameras capture images with optical codes from different perspectives. Using pattern decoding, stereo matching, and triangulation calculation methods, binocular stereo depth maps for the corresponding industrial cameras are obtained. The pixel coordinates of the binocular stereo depth maps are... This indicates that the stereo depth map includes depth data from industrial camera A. Depth data from industrial camera B Correlation analysis is performed on the stereo depth maps to generate fused depth maps. The formula used is:

[0021] ;

[0022] in, and These are the weighting coefficients for industrial camera A and industrial camera B, respectively. , , Let A be the standard deviation of depth measurement for industrial camera A. The depth measurement standard deviation is used for industrial camera B, and the fused depth is used to improve ranging accuracy.

[0023] Furthermore, data processing is performed based on the RGB image and the fusion depth to generate corresponding pixel points. The spatial coordinates P in the three-dimensional coordinate system are based on the following formula:

[0024] ;

[0025] in, The intrinsic parameter matrix is ​​the inverse, and the spatial coordinates P are used to reflect the corresponding pixel points. The corresponding point in a three-dimensional coordinate system.

[0026] Furthermore, correlation analysis is performed on the red, blue, and green channel values ​​of the spatial coordinates in a three-dimensional spatial coordinate system to output single-channel grayscale values. The formula used is:

[0027] ;

[0028] in, , , The red, green, and blue color components are the result of fusing the corresponding pixels of the RGB images from three industrial cameras.

[0029] Furthermore, data processing is performed based on single-channel grayscale to obtain the local average grayscale. The formula used is:

[0030] ;

[0031] in, The local average grayscale value represents the working area of ​​the robotic arm and reflects the average grayscale value within that area. This represents the total number of coordinates of points within the work area.

[0032] Numerical analysis is performed based on local average gray levels, and a set threshold is set. The comparison is performed, and the deviation L is output based on the following formula:

[0033] ;

[0034] When the deviation L is less than 1, it indicates that the local illumination is insufficient and the output needs to be supplemented with light.

[0035] Furthermore, correlation analysis is performed based on local average grayscale and single-channel grayscale to generate uniformity. The formula used is:

[0036] ;

[0037] Uniformity is used to reflect the average brightness of the working area; the smaller the value, the more uniform the brightness.

[0038] The dynamic supplementary lighting formula is generated based on uniformity and deviation:

[0039] ;

[0040] in, The deviation response coefficient has a range of values ​​of 100. , This is the uniformity suppression coefficient, with a value range of [value range missing]. , This represents the current brightness percentage range of the supplementary light, with a value range of [value range missing]. , The percentage range of fill light brightness for the fill light device. This is the update amount for the fill light angle. For the range of adjustment of the fill light angle, ,gradient This represents the change in uniformity when the fill light is adjusted per unit angle at the current angle. The learning rate, used to control the adjustment step size, is set to 0.5. For different angles and brightness environments, a series of uniformity data are pre-collected to establish a neural network model of angle and uniformity. This neural network model is then used to estimate the gradient of the supplementary light at the current angle. , This represents the angle value of the supplementary light from the previous moment to the current moment. This is the current angle value of the fill light. This is the angle value after the fill light has been adjusted;

[0041] The dynamic supplemental lighting formula executes a self-loop, and when the formula is satisfied... The supplemental lighting guidance is terminated at the appropriate time to complete the supplemental lighting process. This is the allowable error value for illumination. These are the allowable error values ​​for illumination uniformity, all of which are set values.

[0042] The present invention also provides a vision guidance system for a robotic arm, used in a vision guidance method for a robotic arm, comprising:

[0043] The setup module is used to set up three industrial cameras and fill lights at the end of the robotic arm. The intrinsic and extrinsic parameter matrices of the industrial cameras are determined by Zhang Zhengyou's checkerboard calibration method, and a three-dimensional coordinate system is constructed based on the intrinsic and extrinsic parameter matrices.

[0044] The fusion depth module is used to simultaneously acquire RGB images and binocular stereo depth maps of the robotic arm's working area using three industrial cameras, correct the RGB images using binocular stereo vision methods, perform correlation analysis based on the binocular stereo depth maps, and generate fusion depth.

[0045] The single-channel grayscale module is used to perform data processing based on RGB images and fusion depth, mapping RGB image pixels to spatial coordinates in a three-dimensional coordinate system and calculating the single-channel grayscale of each spatial coordinate.

[0046] The data processing module is used to perform data processing based on single-channel grayscale, obtain local average grayscale, perform numerical analysis based on local average grayscale, generate deviation, compare it with a set threshold, and output whether supplementary lighting is needed.

[0047] When supplemental lighting is required, the control module performs correlation analysis based on the local average gray level and single-channel gray level to generate uniformity. The uniformity is used to reflect the uniformity of the brightness in the working area. A dynamic supplemental lighting formula is generated based on the uniformity and deviation. The supplemental lighting device performs supplemental lighting based on the dynamic supplemental lighting formula. After the supplemental lighting is completed, the robotic arm continues to perform operations based on vision guidance technology.

[0048] Compared with the prior art, the beneficial effects of the present invention are:

[0049] This invention combines binocular stereo depth maps with correlation analysis to obtain fused depth, effectively reducing single-camera ranging errors and improving the measurement accuracy in three-dimensional space. Simultaneously, based on the fused three-dimensional spatial coordinates and grayscale information, it dynamically calculates the illumination deviation and uniformity of the working area and generates a dynamic supplementary lighting formula. Based on this formula, by adjusting the brightness and angle of the supplementary light source, the illumination of the working area is optimized in real time, avoiding the impact of local overexposure and insufficient illumination on visual guidance. This significantly improves the ranging accuracy and image quality of the robotic arm vision system in complex lighting environments, enhancing the stability and reliability of visual guidance. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the overall method flow of the present invention;

[0051] Figure 2 This is a schematic diagram of the overall system flow of the present invention;

[0052] Figure 3 This is a line graph showing the deviation of the fill light device before and after adjustment according to the present invention;

[0053] Figure 4 This is a line graph showing the uniformity of the filler light device before and after adjustment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0055] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0056] Example:

[0057] Please see Figure 1 The present invention provides a technical solution:

[0058] A vision-guided method for a robotic arm, comprising the following steps:

[0059] Step 1: Set up three industrial cameras and fill lights at the end of the robotic arm. Determine the intrinsic and extrinsic parameter matrices of the industrial cameras using the Zhang Zhengyou checkerboard calibration method. Construct a three-dimensional coordinate system based on the intrinsic and extrinsic parameter matrices.

[0060] The expression for the intrinsic parameter matrix K is:

[0061] ;

[0062] in, , These represent the focal lengths of the corresponding industrial cameras in the horizontal and vertical directions. The horizontal and vertical coordinates of the principal point of the industrial camera;

[0063] The expression for the extrinsic parameter matrix T is:

[0064] ;

[0065] Where R is the rotation matrix, which reflects the spatial rotation relationship between the camera coordinate system of the industrial camera and the world coordinate system, and t is the translation vector, which reflects the spatial translation relationship between the camera coordinate system of the industrial camera and the world coordinate system.

[0066] Step 2: Simultaneously acquire RGB images and binocular stereo depth maps of the robotic arm's working area using three industrial cameras, correct the RGB images using binocular stereo vision, perform correlation analysis based on the binocular stereo depth maps, and generate a fused depth. The fused depth is used to improve ranging accuracy.

[0067] Binocular stereo vision refers to a method that uses two cameras with known positions and parameters to simultaneously capture the same scene from different angles, and then analyzes the positional differences of the same object in the two images to recover the three-dimensional spatial depth information of each point in the scene.

[0068] The two industrial cameras are designated as Industrial Camera A and Industrial Camera B. Data collected by Industrial Camera A and Industrial Camera B are indexed by subscripts A and B, respectively. Encoded optical patterns are projected onto the target surface via a projector. The three industrial cameras capture images with optical encoding from different perspectives. Using pattern decoding, stereo matching, and triangulation calculation methods, binocular stereo depth maps for the corresponding industrial cameras are obtained. The pixel coordinates of the binocular stereo depth maps are... This indicates that the stereo depth map includes depth data from industrial camera A. Depth data from industrial camera B Correlation analysis is performed on the stereo depth maps to generate fused depth maps. The formula used is:

[0069] ;

[0070] in, and These are the weighting coefficients for industrial camera A and industrial camera B, respectively. , , Let A be the standard deviation of depth measurement for industrial camera A. The depth measurement standard deviation is used for industrial camera B, and the fused depth is used to improve ranging accuracy.

[0071] The standard deviation of depth measurement reflects the magnitude of measurement noise. A smaller standard deviation indicates a more concentrated distribution of results from multiple measurements, smaller error fluctuations, and more reliable measurement results. Therefore, the noise is lower and the information content is greater. Since the error in depth measurement follows a zero-mean Gaussian distribution, its information content can be measured by the reciprocal of its variance. Weights are designed to represent the proportion of total information from each industrial camera, and weights are allocated according to this proportion, i.e., the output... and By fusing depth measurement information from industrial cameras and assigning weights based on the proportion of information, the resulting data can be analyzed and significantly improved in depth ranging accuracy compared to using a single camera.

[0072] Step 3: Perform data processing based on the RGB image and fusion depth, map the RGB image pixels to the spatial coordinates of the three-dimensional coordinate system, and calculate the single-channel grayscale of each spatial coordinate. The single-channel grayscale is used to reflect the grayscale at the corresponding spatial coordinate.

[0073] Data processing is performed based on the RGB image and fusion depth to generate corresponding pixels. The spatial coordinates P in the three-dimensional coordinate system are based on the following formula:

[0074] ;

[0075] in, The intrinsic parameter matrix is ​​the inverse, and the spatial coordinates P are used to reflect the corresponding pixel points. The corresponding point in the three-dimensional coordinate system. Based on the fusion depth and camera model, the two-dimensional pixels captured by the industrial camera are mapped back to three-dimensional space.

[0076] Correlation analysis is performed on the red, blue, and green channel values ​​of spatial coordinates based on a three-dimensional spatial coordinate system, and single-channel grayscale is output. The formula used is:

[0077] ;

[0078] in, , , This represents the red, green, and blue color components obtained by fusing corresponding pixels from RGB images of three industrial cameras. Single-channel grayscale values ​​are fundamental to image processing algorithms, simplifying calculations while preserving brightness information for subsequent illumination uniformity analysis.

[0079] Step 4: Perform data processing based on single-channel grayscale to obtain local average grayscale, perform numerical analysis based on local average grayscale to generate deviation, compare it with a set threshold, and output whether supplementary lighting is needed.

[0080] Data processing is performed based on single-channel grayscale to obtain local average grayscale. The formula used is:

[0081] ;

[0082] in, The set of all spatial points within the working area of ​​the robotic arm. The total number of coordinates of points in the working area is represented by the local average gray level, which reflects the average gray level within the working area and comprehensively reflects the overall brightness level of the area. The larger the local average gray level, the higher the overall brightness level of the area.

[0083] Numerical analysis is performed based on local average gray levels, and a set threshold is set. The comparison is performed, with the threshold being the ideal target brightness calibrated during the robotic arm's operation, and the output deviation L based on the following formula:

[0084] ;

[0085] The deviation L reflects the deviation of the current brightness level from the ideal value. The closer the average gray level is to the threshold, the closer the deviation is to 1. When the deviation L is less than 1, it indicates that the local illumination is insufficient and the output needs to be supplemented with light.

[0086] Step 5: When supplementary lighting is required, perform correlation analysis based on local average gray level and single-channel gray level to generate uniformity. The uniformity is used to reflect the uniformity of the brightness in the working area. A dynamic supplementary lighting formula is generated based on the uniformity and deviation. The supplementary lighting device performs supplementary lighting based on the dynamic supplementary lighting formula.

[0087] Correlation analysis is performed based on local average grayscale and single-channel grayscale to generate uniformity. The formula used is:

[0088] ;

[0089] Uniformity is used to reflect the average brightness of the working area. The smaller the value, the more uniform the brightness. Uniformity is used to determine whether the light distribution is uniform, providing a basis for adjusting the angle and brightness of the fill light.

[0090] The dynamic supplementary lighting formula is generated based on uniformity and deviation:

[0091] ;

[0092] in, This is the deviation response coefficient, used to control the sensitivity of the supplementary light's brightness adjustment to the deviation, determining the magnitude of the change in supplementary light brightness with the deviation, and its value range is... The larger the value, the more sensitive it is. This is the uniformity suppression coefficient, used to control the weight of the fill light adjustment on the uniformity index suppression, suppressing local overexposure caused by uneven lighting, and ensuring the uniformity of the fill light. Its value range is... The larger the value, the greater the weight of the brightness adjustment of the supplementary light on the uniformity index suppression. The deviation response coefficient and uniformity suppression coefficient determine the adjustment speed, stability, and quality of the supplementary light. Through experimental calibration, This represents the current brightness percentage range of the supplementary light, with a value range of [value range missing]. , The percentage range of fill light brightness for the fill light device. This is a cutoff function used to limit the angle adjustment range of the fill light. This is the update amount for the fill light angle. These are the minimum and maximum values ​​for the fill light angle adjustment range. ,gradient This represents the change in uniformity when the fill light is adjusted per unit angle at the current angle. The learning rate, used to control the adjustment step size, is set to 0.5. For different angles and brightness environments, a series of uniformity data are pre-collected to establish a neural network model of angle and uniformity. This neural network model is then used to estimate the gradient of the supplementary light at the current angle. , This is the current angle value of the fill light. This is the angle value after the fill light has been adjusted;

[0093] According to the dynamic supplemental lighting formula, when the deviation is less than 1, the brightness of the working area is insufficient, requiring supplemental lighting. Therefore, by... Increasing the brightness and uniformity of the supplementary light results in a more uneven brightness distribution, leading to localized overexposure and preventing visual guidance from capturing image information. Therefore, by... Reducing brightness and avoiding overly bright areas helps to suppress overexposure caused by excessive brightness; angle adjustment is based on gradient. The adjustment is made in the direction of reducing uniformity, with a momentum term of 0.6 to ensure convergence stability. Among them, the deviation L is negatively correlated with the brightness of the supplementary light. The smaller the deviation L, the weaker the illumination, and the more supplementary light is needed, that is, the more brightness needs to be increased. The uniformity is negatively correlated with the brightness of the supplementary light. That is, the larger the uniformity, the brighter the local points in the working area, which poses a risk of overexposure, and therefore the brightness is reduced.

[0094] The dynamic supplemental lighting formula executes a self-loop, and when the formula is satisfied... The supplementary lighting guidance is terminated at the designated time to complete the supplementary lighting process. During the supplementary lighting process, the brightness is adjusted first, followed by the angle. If the dynamic supplementary lighting formula is not met, the brightness and angle are adjusted again, with linked fine-tuning, until both brightness and uniformity meet the standards. If the standards are not met after several cycles, an alarm is triggered to alert the staff. This is the allowable error value for illumination. These are the allowable error values ​​for illumination uniformity, all of which are set values.

[0095] Table 1: Illumination Data Sheet for Fill Light Devices

[0096]

[0097] Reference Figure 3 and Figure 4 During the vision-guided process of the robotic arm, the data in the table above reflects the dynamic adjustment of the supplementary lighting, ensuring that the brightness of the working area remains within the preset standard range. Before adjustment, the uniformity was mostly concentrated between 0.02 and 0.06, with some not meeting the preset uniformity threshold. After adjustment, it was less than 0.02. Before adjustment, the deviation was either too low or too high, but after adjustment, it met the deviation threshold range, indicating that the supplementary lighting effectively compensated for insufficient or excessive illumination.

[0098] Step 6: After the supplementary adjustments are completed, the robotic arm continues to perform operations based on vision-guided technology.

[0099] The present invention also provides a vision guidance system for a robotic arm, used in a vision guidance method for a robotic arm, comprising:

[0100] The setup module is used to set up three industrial cameras and fill lights at the end of the robotic arm. The intrinsic and extrinsic parameter matrices of the industrial cameras are determined by Zhang Zhengyou's checkerboard calibration method, and a three-dimensional coordinate system is constructed based on the intrinsic and extrinsic parameter matrices.

[0101] The fusion depth module is used to simultaneously acquire RGB images and binocular stereo depth maps of the robotic arm's working area using three industrial cameras, correct the RGB images using binocular stereo vision methods, perform correlation analysis based on the binocular stereo depth maps, and generate fusion depth.

[0102] The single-channel grayscale module is used to perform data processing based on RGB images and fusion depth, mapping RGB image pixels to spatial coordinates in a three-dimensional coordinate system and calculating the single-channel grayscale of each spatial coordinate.

[0103] The data processing module is used to perform data processing based on single-channel grayscale, obtain local average grayscale, perform numerical analysis based on local average grayscale, generate deviation, compare it with a set threshold, and output whether supplementary lighting is needed.

[0104] When supplemental lighting is required, the control module performs correlation analysis based on the local average gray level and single-channel gray level to generate uniformity. The uniformity is used to reflect the uniformity of the brightness in the working area. A dynamic supplemental lighting formula is generated based on the uniformity and deviation. The supplemental lighting device performs supplemental lighting based on the dynamic supplemental lighting formula. After the supplemental lighting is completed, the robotic arm continues to perform operations based on vision guidance technology.

[0105] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0106] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0107] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0108] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A visual guidance method for a robotic arm, characterized in that, The specific steps include: S1. Set up three industrial cameras and fill lights at the end of the robotic arm. Determine the intrinsic and extrinsic parameters of the industrial cameras using the Zhang Zhengyou checkerboard calibration method. Construct a three-dimensional coordinate system based on the intrinsic and extrinsic parameters. S2. Based on three industrial cameras, RGB images and binocular stereo depth maps of the working area of ​​the robotic arm are acquired simultaneously. The RGB images are corrected using a binocular stereo vision method. Correlation analysis is performed based on the binocular stereo depth map to generate a fused depth. The fused depth is used to improve the ranging accuracy. S3. Perform data processing based on RGB image and fusion depth, correspond RGB image pixels to spatial coordinates in a three-dimensional spatial coordinate system, and calculate the single-channel grayscale of each spatial coordinate. The single-channel grayscale is used to reflect the grayscale at the corresponding spatial coordinate. S4. Perform data processing based on single-channel grayscale to obtain local average grayscale, perform numerical analysis based on local average grayscale, generate deviation, compare with a set threshold, and output whether supplementary lighting is needed. S5. When supplementary lighting is required for the output, a correlation analysis is performed based on the local average gray level and the single-channel gray level to generate uniformity. The uniformity is used to reflect the uniformity of the brightness in the working area. A dynamic supplementary lighting formula is generated based on the uniformity and deviation. The supplementary lighting device performs supplementary lighting based on the dynamic supplementary lighting formula. S6. After the supplementary adjustments are completed, the robotic arm continues to perform operations based on vision-guided technology; The dynamic supplementary lighting formula is generated based on uniformity and deviation: in, The deviation response coefficient has a range of values ​​of 100. , This is the uniformity suppression coefficient, with a value range of [value range missing]. , This represents the current brightness percentage range of the supplementary light, with a value range of [value range missing]. , The percentage increase in brightness after supplemental lighting. This is the update amount for the fill light angle. For the range of adjustment of the fill light angle, ,gradient This represents the change in uniformity when the fill light is adjusted per unit angle at the current angle. The learning rate, used to control the adjustment step size, is set to 0.

5. For different angles and brightness environments, a series of uniformity data are pre-collected to establish a neural network model of angle and uniformity. This neural network model is then used to estimate the gradient of the supplementary light at the current angle. , This represents the change in the angle of the supplementary light at the previous moment. This is the current angle value of the fill light. This is the angle value after the fill light has been adjusted. To output the deviation, For uniformity; The dynamic supplemental lighting formula executes a self-loop, and when the formula is satisfied... The supplemental lighting guidance is terminated at the appropriate time to complete the supplemental lighting process. This is the allowable error value for illumination. These are the allowable error values ​​for illumination uniformity, all of which are set values.

2. The visual guidance method for a robotic arm according to claim 1, characterized in that: The expression for the intrinsic parameter matrix K is: in, , This refers to the focal length of the industrial camera in both the horizontal and vertical directions. The horizontal and vertical coordinates of the principal point of the industrial camera; The expression for the extrinsic parameter matrix T is: Where R is the rotation matrix, which reflects the spatial rotation of the industrial camera's camera coordinate system relative to the world coordinate system, and t is the translation vector, which reflects the spatial translation of the industrial camera's camera coordinate system relative to the world coordinate system.

3. The visual guidance method for a robotic arm according to claim 2, characterized in that: The three industrial cameras are paired into two groups of two, namely Industrial Camera Group A and Industrial Camera Group B. The industrial camera groups acquire stereo depth maps using a binocular stereo vision method. The pixel coordinates of the stereo depth maps are used... This indicates that the stereo depth map includes depth data acquired by industrial camera group A. Depth data acquired by industrial camera group B Correlation analysis is performed on the stereo depth maps to generate fused depth maps. The formula used is: in, and These are the weighting coefficients for industrial camera group A and industrial camera group B, respectively. , , Let A be the standard deviation of depth measurement for industrial camera group A. To determine the standard deviation of depth measurement for industrial camera group B, we comprehensively analyze two sets of depth data acquired by three industrial cameras to generate a fused depth and improve ranging accuracy.

4. The visual guidance method for a robotic arm according to claim 3, characterized in that: Data processing is performed based on the RGB image and fusion depth to generate corresponding pixels. The spatial coordinates P in the three-dimensional coordinate system are based on the following formula: in, The intrinsic parameter matrix is ​​the inverse, and the spatial coordinates P are used to reflect the corresponding pixel points. The corresponding point in a three-dimensional coordinate system.

5. The visual guidance method for a robotic arm according to claim 4, characterized in that: Correlation analysis is performed on the red, blue, and green channel values ​​of spatial coordinates based on a three-dimensional spatial coordinate system, and single-channel grayscale is output. The formula used is: in, , , The red, green, and blue color components are the result of fusing the corresponding pixels of the RGB images from three industrial cameras.

6. The visual guidance method for a robotic arm according to claim 5, characterized in that: Data processing is performed based on single-channel grayscale to obtain local average grayscale. The formula used is: in, The local average grayscale value represents the working area of ​​the robotic arm and reflects the average grayscale value within that area. This represents the total number of coordinates of points within the work area. Numerical analysis is performed based on local average gray levels, and a set threshold is set. The comparison is performed, and the deviation L is output based on the following formula: When the deviation L is less than 1, it indicates that the local illumination is insufficient and the output needs to be supplemented with light.

7. The visual guidance method for a robotic arm according to claim 6, characterized in that: Correlation analysis is performed based on local average grayscale and single-channel grayscale to generate uniformity. The formula used is: Uniformity is used to reflect the average brightness of the working area; the smaller the value, the more uniform the brightness.

8. A vision guidance system for a robotic arm, used to execute the vision guidance method for a robotic arm according to any one of claims 1-7, characterized in that, include: The setup module is used to set up three industrial cameras and fill lights at the end of the robotic arm. The intrinsic and extrinsic parameter matrices of the industrial cameras are determined by Zhang Zhengyou's checkerboard calibration method, and a three-dimensional coordinate system is constructed based on the intrinsic and extrinsic parameter matrices. The fusion depth module is used to simultaneously acquire RGB images and binocular stereo depth maps of the robotic arm's working area using three industrial cameras, correct the RGB images using binocular stereo vision methods, perform correlation analysis based on the binocular stereo depth maps, and generate fusion depth. The single-channel grayscale module is used to perform data processing based on RGB images and fusion depth, mapping RGB image pixels to spatial coordinates in a three-dimensional coordinate system and calculating the single-channel grayscale of each spatial coordinate. The data processing module is used to perform data processing based on single-channel grayscale, obtain local average grayscale, perform numerical analysis based on local average grayscale, generate deviation, compare it with a set threshold, and output whether supplementary lighting is needed. When supplemental lighting is required, the control module performs correlation analysis based on the local average gray level and single-channel gray level to generate uniformity. The uniformity is used to reflect the uniformity of the brightness in the working area. A dynamic supplemental lighting formula is generated based on the uniformity and deviation. The supplemental lighting device performs supplemental lighting based on the dynamic supplemental lighting formula. After the supplemental lighting is completed, the robotic arm continues to perform operations based on vision guidance technology.

Images