Hoisting mechanical arm guiding method based on monocular machine vision

Abstract

The invention relates to a hoisting mechanical arm guiding method based on monocular machine vision. The method comprises the steps that real-time images are collected through a monocular industrial camera, recognition is completed through a recognition algorithm, communication between a guiding algorithm and a PLC is achieved, control over the action of a mechanical arm is completed, and accurate guiding of the hoisting mechanical arm is achieved. The method comprises the following steps: acquiring image data by using an industrial camera, and extracting position and attitude information of a target object in combination with image processing and a feature recognition algorithm; the motion trail of the mechanical arm is calculated through a control algorithm, and the position and posture of the mechanical arm are adjusted through the PLC to achieve accurate positioning. In addition, the invention further comprises a monocular camera image size measurement algorithm, the actual size of an object is calculated through camera calibration and depth estimation, and therefore the working precision of the mechanical arm is improved. The method simplifies hardware configuration, reduces equipment cost, has the advantages of high efficiency, intelligence and accuracy, and is suitable for the fields of automatic lifting and carrying, industrial automation, robot technology and the like.

Description

Technical Field

[0001] The present invention relates to the field of machine vision and lifting machinery, and in particular to a method for guiding a lifting mechanical arm using monocular machine vision, which is applicable to automated lifting and handling, industrial automation and robotics. Specifically, the present invention provides a technical solution for achieving precise guidance and positioning of a lifting mechanical arm by combining hardware devices such as a monocular industrial camera, a PLC controller and a graphics card with image processing, recognition algorithms, deep learning and guidance algorithms. Background Art

[0002] With the improvement of industrial automation level, lifting and handling are gradually developing towards intelligence and refinement. Traditional lifting and handling operations usually rely on manual operation or simple mechanical control systems, which cannot effectively cope with complex and changing working environments, and have problems such as low precision, slow response, poor flexibility and frequent manual intervention. With the development of machine vision and artificial intelligence technology, vision-based guidance methods have gradually become one of the key technologies for the automation of crane arms and an important technical direction for the automation transformation of the industrial field.

[0003] In the prior art, visual guidance methods rely on complex multi-camera systems, lidar, ultrasonic sensors, depth cameras and other external sensor devices. Although these solutions have improved accuracy to a certain extent, they also bring a series of problems such as high system complexity, high hardware cost, poor environmental adaptability, and difficulty in integration. Multi-camera systems also require high-precision synchronization and calibration, and have extremely high requirements for the real-time and stability of the system. The size and cost of the equipment also make this technology difficult to be widely used in small and medium-sized enterprises or high-cost sensitive scenarios. For example: the domestically disclosed patent CN119217385A uses four cameras plus anti-interference components to complete product recognition; patent CN119260374A uses multiple cameras plus hinges to locate the door position; both of these disclosed patents use multiple cameras plus other auxiliary facilities to complete positioning, and both have complex equipment, large recognition calculation workload, and high cost.

[0004] Therefore, how to solve the shortcomings of traditional vision methods in object positioning, 3D reconstruction, precision control and data exchange delay on the basis of simplifying the hardware configuration of multiple cameras, especially in the framework of monocular vision system; using monocular camera to guide the operation of crane mechanical arm with the required precision, robustness and real-time performance has become a key issue that needs to be solved urgently. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a method for guiding a crane arm based on monocular machine vision, the method comprising:

[0006] A monocular industrial camera is set up and fixed on the front end of the lifting mechanical arm to obtain real-time images of objects. The camera uses a commonly used RGB (Red, Green, Blue) color camera to support high-speed real-time data transmission. The camera collects image data of the mechanical arm operating area to obtain visual information related to the lifting operation. According to actual needs, a camera with higher resolution and frame rate can be selected.

[0007] Image processing and feature recognition: collect image data of the crane operation site through industrial cameras, including spatial images, target object images, etc., and transmit them to computer algorithm programs for analysis and calculation; the image needs to be preprocessed in the early stage, which includes image denoising and image enhancement:

[0008] Image denoising uses Gaussian filtering and convolution operation function:

[0009]

[0010] Where δ is the standard deviation of the Gaussian kernel and k is the size of the filter;

[0011] For image enhancement, a new adaptive multi-domain joint enhancement algorithm is proposed. This algorithm integrates spatial domain enhancement, frequency domain enhancement and pyramid enhancement technologies. Through the adaptive weight allocation mechanism, the enhancement effect is dynamically optimized to meet the needs of different scenarios. The following is the fusion enhancement feature function:

[0012] I'(x,y)=ws·Is'(x,y)+wf·If'(x,y)+wp·Ip'(x,y)

[0013] ws+wf+wp=1

[0014]

[0015] Among them, w s 、w f 、w p They are spatial domain enhancement, frequency domain enhancement and pyramid enhancement adaptive weights.

[0016] After image preprocessing, convolutional neural networks are used for image recognition and self-learning. Extracting image features is an important task in image recognition. The key point of image feature extraction is the local extreme point in the scale space. The positioning function of the scale space L(x, y, δ) is:

[0017] L(x,y,σ)=Gaussian(x,y,σ)×I(x,y)

[0018] Among them, Gaussian(x,y,δ) is the Gaussian filter function, and I(x,y) is the collected image.

[0019] Calculate the 3D depth information in 2D images. Since 2D images lack direct 3D spatial information, a new method using geometric modeling, machine learning and image feature extraction is proposed. This method can efficiently calculate the depth information of 2D images and provide robust and high-precision depth estimation capabilities in complex scenes.

[0020] First, depth calculation is performed based on monocular vision and geometric constraints. For a monocular system, the target size is known or obtained through image priors. Through the parallel view geometry model, the depth can be estimated through the projection relationship:

[0021]

[0022] Among them, Z is the depth information, H is the height of the target object in the real world, h is the projection height of the target object in the image (in pixels), and f is the focal length of the camera;

[0023] Then, a deep prediction model based on convolutional neural network is used:

[0024] Z(x,y)=Φ(I,θ)

[0025] Where Z(x,y) is the depth value of the pixel (x,y) in the image, I is the input image, Φ is the neural network model function, and θ is the network parameter obtained through training;

[0026] Furthermore, using depth estimation based on optical flow field, for a moving camera, the depth information can be calculated by analyzing the optical flow field of the image. The calculation formula for the depth in the optical flow field is:

[0027]

[0028] Among them, v t is the vertical velocity of the target on the image plane, u is the horizontal displacement of the target in the image, and f is the focal length;

[0029] Finally, combining the advantages of the above three depth information algorithms, a weighted fusion depth estimation formula is constructed:

[0030] Z(x,y)=w1·Zgeometry+w2·Zlearning+w3·Zoptical

[0031] w1+w2+w3=1

[0032] Among them, Z geometry is the depth estimation based on geometric constraints, Z learning Depth estimation based on machine learning, Z optical Depth estimation based on optical flow and blur, w1, w2, w3 are weights.

[0033] The recognition algorithm guides PLC control. First, real-time two-way communication between the guidance algorithm and the PLC controller is established. Then, the recognized target position and posture information, the calculated motion trajectory and angle adjustment parameters of the robot arm are sent to the PLC controller, and the progress and results of the PLC controller are obtained. The actual operating position and angle of the robot arm are obtained according to the camera, and the errors of the calculation results and the execution results are compared. Based on the error feedback of image recognition, the motion path of the robot arm is adjusted to ensure the operation accuracy. Finally, it is sent to the action execution mechanism of the crane arm. Here, the error control mechanism is introduced to evaluate and optimize the error of each lifting operation to improve the accuracy and efficiency of the overall operation. The error calibration function is:

[0034] e(t)=xi-xj

[0035]

[0036] Among them, x i is the target position, x j is the current position of the robot arm, u(t) is the control data after real-time correction, K p is the ratio, K i is the integral gain, K d is the differential gain.

[0037] The robotic arm executes the action instructions issued by the PLC controller. The lifting mechanical arm is composed of multi-degree-of-freedom robotic arms, which are responsible for performing lifting and handling operations. It has a high-precision position and angle control system and operates according to (x, y, z) coordinate values. It should be noted that the robotic arm will obtain new position instructions during operation. This is a process of continuous cycle correction to the target position.

[0038] The PLC controller interacts with the visual guidance system in real time. The PLC controls the crane arm to perform actions, cyclically receives control guidance signals from the computer, and feeds back progress signals of real-time execution. The electric, pneumatic and other devices of the arm complete all actions of the crane arm until the task is completed. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flow chart of a crane arm guidance method based on monocular machine vision provided by an embodiment of the present invention;

[0040] Figure 2 An architectural diagram of a crane arm guidance method based on monocular machine vision provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0042] Method Example:

[0043] A specific embodiment of the present invention discloses a method for guiding a crane arm based on monocular machine vision, which realizes the use of a monocular industrial camera to guide the crane arm to lift and carry goods. Figure 1 As shown, the method includes:

[0044] Step S1, setting a monocular industrial camera, fixing the camera on the front end of the lifting mechanical arm, acquiring an object image and an environment image, and the camera continuously collects image data of the operating area during the operation of the mechanical arm to obtain visual information related to the lifting operation;

[0045] Among them, the monocular camera adopts RGB (Red, Green, Blue) color camera, which supports high-speed data real-time transmission. According to the needs of the actual scene, a camera with higher resolution and frame rate can be selected to improve the image quality and acquisition speed. Preferably, an industrial camera with a shooting speed of more than 90 frames per second is used.

[0046] Step S2, after collecting the image, directly read the image from the camera, perform image processing and feature recognition; synchronously, store the image in a specified location of the memory for secondary call by the algorithm; collect real-time image data of the crane operation site through an industrial camera, including spatial images, images of the lifted objects and markings, etc., and transmit them to a computer, which is analyzed by the computer's algorithm program and calculated by a graphics card; pre-process the collected image, which includes image denoising using Gaussian filtering and image enhancement using a neural network;

[0047] First, preprocess the image and use Gaussian filtering to remove noise and convolution operation function:

[0048]

[0049] Where δ is the standard deviation of the Gaussian kernel and k is the size of the filter;

[0050] Secondly, for image enhancement, a new adaptive multi-domain joint enhancement algorithm is proposed. This algorithm integrates spatial domain enhancement, frequency domain enhancement and pyramid enhancement technologies. Through the adaptive weight allocation mechanism, the enhancement effect is dynamically optimized to meet the needs of different scenarios. The following is the fusion enhancement feature function:

[0051] I'(x,y)=ws·Is'(x,y)+wf·If'(x,y)+wp·Ip'(x,y)

[0052] ws+wf+wp=1

[0053]

[0054] Among them, w s 、w f 、w p They are the adaptive weights for spatial domain, frequency domain and pyramid enhancement respectively.

[0055] Step S3, after image preprocessing is completed, image recognition is performed; a convolutional neural network is used to extract local features of the image, and features such as texture and edge are captured through the convolution kernel. Feature descriptors are generated for the extracted feature points for subsequent matching and classification; the core of feature extraction is to locate local extreme points with stability from the scale space, extract key points to describe the significant features of the image, and the positioning function of the scale space L(x, y, δ) is:

[0056] L(x,y,σ)=Gaussian(x,y,σ)×I(x,y)

[0057] Among them, Gaussian(x,y,δ) is the Gaussian filter function, and I(x,y) is the collected image.

[0058] Step S4, calculating the three-dimensional depth information in the two-dimensional image. Due to the lack of direct three-dimensional spatial information in the two-dimensional image, a new method using geometric modeling, machine learning and image feature extraction is proposed. This method can efficiently calculate the depth information of the two-dimensional image and has the ability to provide robust and high-precision depth estimation in complex scenes;

[0059] First, depth calculation is performed based on monocular vision and geometric constraints. For a monocular system, the target size is known or obtained through image priors. Through the parallel view geometry model, the depth can be estimated through the projection relationship:

[0060]

[0061] Among them, Z is the depth information, H is the height of the target object in the real world, h is the projection height of the target object in the image (in pixels), and f is the focal length of the camera;

[0062] Then, a deep prediction model based on convolutional neural network is used:

[0063] Z(x,y)=Φ(I,θ)

[0064] Where Z(x,y) is the depth value of the pixel (x,y) in the image, I is the input image, Φ is the neural network model function, and θ is the network parameter obtained through training;

[0065] Furthermore, using depth estimation based on optical flow field, for a moving camera, the depth information can be calculated by analyzing the optical flow field of the image. The calculation formula for the depth in the optical flow field is:

[0066]

[0067] Among them, v t is the vertical velocity of the target on the image plane, u is the horizontal displacement of the target in the image, and f is the focal length;

[0068] Finally, combining the advantages of the above three depth information algorithms, a weighted fusion depth estimation formula is constructed:

[0069] Z(x,y)=w1·Zgeometry+w2·Zlearning+w3·Zoptical

[0070] w1+w2+w3=1

[0071] Among them, Z geometry is the depth estimation based on geometric constraints, Z learning Depth estimation based on machine learning, Z optical Depth estimation based on optical flow and blur, w1, w2, w3 are weights.

[0072] Step S5, the recognition algorithm guides the PLC control. First, establish real-time two-way communication between the guidance algorithm and the PLC controller. The communication delay is required to be less than 10ms, and the guidance algorithm updates and sends instructions for less than 50ms. Then, the identified target position and posture information, the calculated motion trajectory and angle adjustment parameters of the robot arm are sent to the PLC controller, and the progress and results of the PLC controller are obtained. The actual operating position and angle of the robot arm are obtained according to the camera, and the errors of the calculation results and the execution results are compared. Based on the error feedback of image recognition, the motion path of the robot arm is adjusted to ensure the operation accuracy. Finally, the instruction is sent to the action execution mechanism of the crane arm. Here, the error control mechanism is introduced to evaluate and optimize the error of each lifting and handling operation to improve the accuracy and efficiency of the overall operation. The error calibration function is:

[0073] e(t)=xi-xj

[0074]

[0075] Among them, x i is the target position, x jis the current position of the robot arm, u(t) is the control data after real-time correction, K p is the ratio, K i is the integral gain, K d is the differential gain.

[0076] Step S6, the robotic arm executes the action instructions issued by the PLC controller. The lifting robotic arm is composed of a multi-degree-of-freedom robotic arm, which is responsible for performing lifting and handling operations. It has a high-precision position and angle control system and operates according to the (x, y, z) coordinate system. It should be noted that the robotic arm will continuously obtain new position instructions during operation, which is a process of constantly correcting and running to the target position.

[0077] Step S7, visual guidance real-time shooting of the running image, update and send the three-dimensional position guidance information to the PLC controller, the real-time update time is no more than 100ms, until the robot arm runs to the target, gives the next action instruction, and works in a cycle.

[0078] In step S8, the PLC controller interacts with the visual guidance program in real time, and the time interval between each interaction is less than 10ms; the PLC controls the crane arm to execute actions, cyclically receives control guidance signals from the computer, and feeds back real-time progress signals; the electric, pneumatic and other devices of the arm complete all actions of the crane arm until the task is completed.

[0079] System Example

[0080] Another embodiment of the present invention discloses a system example of a method for guiding a crane arm based on monocular machine vision, comprising:

[0081] Crane arm structure module, such as Figure 2 As shown in Figure 1, it includes the supporting structure frame of the crane arm, the parallel movement, up and down lifting and lifting units of the multi-axis motion robot arm, and is composed of multiple sets of motors, reducers, air pumps, sensors and brake units. This module is an important structure and execution module for completing the lifting and handling of items.

[0082] Preferably, the monocular industrial camera is installed at the front end of the robotic arm, such as Figure 2 As shown in Figure 2; the camera uses a 6-megapixel color global shutter industrial camera, which provides images at no less than 90 frames per second. The camera lens is flush with the front end of the robotic arm to meet the needs of real-time image capture. At the same time, an LED lighting source is set on the upper part of the robotic arm. The LED light source has been optically designed to ensure the lighting requirements of the on-site camera for image collection.

[0083] Boom execution module, such as Figure 2As shown in 3, the crane arm can complete horizontal, vertical and up and down movements, and can precisely execute three-dimensional space (x, y, z) coordinate movements according to instructions to complete the loading, lifting and unloading of objects.

[0084] PLC controller, PLC controller and visual guidance system implement data interaction, each interaction time interval is 10ms; PLC controls the crane arm to execute actions, cyclically receives control guidance signals from the computer, and feeds back the real-time position, height, load and other information of the robot arm.

[0085] Preferably, the vision guidance algorithm includes algorithms such as image denoising and enhancement preprocessing, image recognition, two-dimensional to three-dimensional depth calculation, dimension measurement, feature extraction, and deep learning; at the same time, it also includes algorithms such as information interaction with the PLC controller and motion error correction, which sends three-dimensional (x, y, z) coordinates to the PLC controller in real time to guide the robotic arm execution unit to perform actions, thereby achieving high-precision, real-time, and highly robust unmanned automatic guidance operations.

[0086] Compared with the prior art, the present invention proposes a method for guiding a crane arm based on monocular machine vision. First, by using a monocular camera and advanced image processing algorithms, the position, shape and posture of the object can be accurately identified, and then the crane arm can be accurately guided to perform automated operations. Compared with the traditional control method based on manual operation or simple sensor system, the present invention can greatly improve the lifting and handling accuracy and reduce the risks and losses caused by human error. Secondly, monocular vision combined with multi-scale fusion enhancement, image denoising, depth estimation and other algorithms enables the present invention to work adaptively in complex and dynamic environments, including lighting changes, background interference and target deformation. Through algorithm optimization, the system can still maintain high recognition accuracy and robustness in adverse environments. Finally, compared with the traditional system that relies on multi-camera and multi-sensor systems for target positioning and control, the present invention adopts a monocular vision system, and through improved image processing and deep learning algorithms, it realizes precise guidance of the robot arm at low cost.

[0087] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for guiding a crane arm based on monocular machine vision, characterized in that: The system includes: collecting image data of the working area where the target object is located through a monocular industrial camera; extracting the size, position and posture information of the target object by combining image processing and feature recognition algorithms; calculating the motion trajectory and angle adjustment parameters of the lifting mechanical arm according to the size and position information of the target object; transmitting the calculation results to the mechanical arm actuator through a PLC controller, adjusting the movement of the mechanical arm so that it is accurately aligned with the target object to complete the lifting action.

2. The monocular vision crane arm guidance method according to claim 1 is characterized in that: The image processing and feature recognition algorithms include edge detection, contour extraction and deep learning algorithms, which are used for machine self-learning and automatic extraction of the shape, position and posture of the target object.

3. The crane arm guiding method according to claim 1, characterized in that: The algorithm for calculating the position information of the target object includes kinematics and trajectory planning algorithms for accurate calculation.

4. The crane arm guiding method according to claim 1, characterized in that: The PLC controller adjusts the position, posture, speed or acceleration of the robot arm according to the obtained target position and posture information.

5. The crane arm guiding method according to claim 1, characterized in that: The method further includes acquiring multiple images through an industrial camera and optimizing the position information of the target object using an error correction algorithm.

6. An object size measurement algorithm based on a monocular industrial camera, comprising obtaining the camera's intrinsic parameter matrix through camera calibration; calculating the actual size of the target object using the size of the target object in the image and the camera's focal length; and further optimizing the actual size measurement based on the distance between the object and the camera through a depth estimation algorithm.

7. The object size measurement algorithm according to claim 6, characterized in that: The depth estimation algorithm calculates the distance between the object and the camera by knowing the actual size of the object and the size of the object in the image.

8. The object size measurement algorithm according to claim 6, characterized in that: The object size optimization adopts multiple measurements combined with an error correction method to improve the final measurement size accuracy.

9. The crane arm guidance method and object size measurement algorithm according to claims 1 and 6, characterized in that: The method is applicable to the fields of automated lifting and handling, industrial automation, robotics, etc., and can achieve efficient and precise operation control.

Images