A method and system for controlling an overhead crane based on machine vision

Abstract

The present invention discloses a method and system for controlling an overhead crane based on machine vision, comprising: S1: installing a binocular camera, calibrating it and connecting it to a network; S2: collecting and preprocessing the image data of the overhead crane to obtain the preprocessed image data of the overhead crane; S3: obtaining the image data of a heavy object based on the preprocessed image data of the overhead crane, extracting features, calculating the number of required feature point pairs, and counting the number of SIFT feature point pairs; S4: calculating the binocular matching feature point pairs and their descriptors, and calculating the binocular depth of the heavy object and the horizontal and vertical coordinates of the center of the heavy object in the camera coordinate system; S5: calculating the hook lowering distance and the hook safety range radius function; giving an alarm when an obstacle is detected within the hook safety radius; S6: calculating the control parameters of the overhead crane and controlling the overhead crane. The present invention can solve the problems of insufficient three-dimensional positioning accuracy and limited obstacle avoidance capability of the traditional method for controlling an overhead crane based on machine vision.

Description

Technical Field

[0001] The present invention relates to the technical field of overhead crane control, and in particular to an overhead crane control method and system based on machine vision. Background Art

[0002] In the field of industrial automation, overhead cranes (also known as bridge cranes) are essential material handling equipment and are widely used in various industrial scenarios. Machine vision-based overhead crane control systems integrate high-definition cameras and image processing algorithms to capture images of the crane's operating area, identify, locate, and track objects within the images, and thus guide the crane's motion trajectory and operations, enabling real-time monitoring and precise control of the crane's operating environment. Three-dimensional positioning and obstacle avoidance are two key components of the crane control system, crucial for ensuring safe and efficient operation.

[0003] Traditional overhead crane control systems primarily rely on mechanical sensors, such as encoders and limit switches, and simple image processing techniques for 3D positioning. These methods often struggle to ensure high-precision 3D positioning in complex and changing industrial environments. Mechanical sensors are susceptible to wear and interference, affecting positioning accuracy. Simple image processing techniques struggle to handle image noise, occlusion, and lighting variations, leading to inaccurate positioning results.

[0004] Furthermore, traditional overhead crane control systems rely primarily on preset safety distances for obstacle avoidance. However, in a dynamic industrial environment, obstacles can change in position, shape, and speed, making it difficult for traditional methods to achieve real-time, accurate obstacle detection and early warning, increasing safety risks in overhead crane operations. Summary of the Invention

[0005] In view of this, the present invention aims to provide a method and system for controlling an overhead crane based on machine vision, aiming to solve the problems of insufficient three-dimensional positioning accuracy and limited obstacle avoidance capability of traditional overhead crane control methods based on machine vision.

[0006] A method for controlling an overhead crane based on machine vision comprises the following steps:

[0007] S1: Install the binocular camera, calibrate the binocular camera and connect it to the network;

[0008] S2: Using a binocular camera to collect and preprocess the overhead crane image data to obtain preprocessed overhead crane image data;

[0009] S3: Perform heavy object detection on the pre-processed overhead crane image data, crop the heavy object after detection, and obtain heavy object image data; perform feature extraction on the heavy object image data and calculate the required number of feature point pairs; then perform SIFT feature point matching on the heavy object image data and count the number of SIFT feature point pairs;

[0010] S4: If the number of required feature point pairs is greater than the number of SIFT feature point pairs, then the heavy object features are combined to calculate the supplementary feature point pairs and their descriptors, and the supplementary feature point pairs and their descriptors are merged with the SIFT feature point pairs and their descriptors to obtain the binocular matching feature point pairs and their descriptors; if the number of required feature point pairs is less than the number of SIFT feature point pairs, then the heavy object features are combined to delete the SIFT feature point pairs to obtain the binocular matching feature point pairs and their descriptors; based on the binocular matching feature point pairs, the binocular depth of the heavy object and the horizontal and vertical coordinates of the center of the heavy object in the camera coordinate system are calculated;

[0011] S5: Perform hook target detection on the pre-processed overhead crane image data to obtain a hook target frame, crop the hook image data, and calculate the hook lowering distance; perform obstacle target detection on the pre-processed overhead crane image data and extract obstacle features; combine the hook lowering distance and obstacle features to calculate the hook safety range radius function; and issue an alarm when an obstacle is detected within the hook safety radius.

[0012] S6: Based on the binocular depth of the heavy object, the horizontal and vertical coordinates of the center of the heavy object in the camera coordinate system, and the hook target frame, calculate the overhead crane control parameters, and control the overhead crane according to the overhead crane control parameters.

[0013] Furthermore, the S1 step includes:

[0014] S11: Install a binocular camera on the trolley in front of the overhead crane. The straight-line distance between the binocular camera and the hook should be no less than 1.5 meters.

[0015] S12: Use Zhang Zhengyou calibration method to calibrate the binocular camera;

[0016] S13: Connect the binocular camera to the industrial computer through a gigabit switch and configure the camera's network communication parameters.

[0017] Furthermore, the S2 step includes:

[0018] S21: capturing image data of the overhead crane in the operating area of ​​the overhead crane in real time using a binocular camera, wherein the image data includes left image data and right image data, and the left image data and the right image data are respectively acquired by two different cameras of the binocular camera;

[0019] S22: performing denoising processing on the overhead crane image data using Gaussian filtering to obtain denoised overhead crane image data;

[0020] S23: Using the Bouguet algorithm to perform stereo correction on the left image data and the right image data in the denoised overhead crane image data to ensure that the two images are on the same plane, thereby obtaining preprocessed overhead crane image data; the preprocessed overhead crane image data includes preprocessed left image data and preprocessed right image data.

[0021] Furthermore, the S3 step includes:

[0022] S31: Perform heavy object detection on the pre-processed overhead crane image data and output the horizontal position of the heavy object according to the center coordinates of the heavy object. The calculation method is: ;

[0023] in, are the horizontal and vertical coordinates of the center of the weight in the pre-processed overhead crane image data, is a rectangular heavy object target frame consisting of 4 vertex coordinates; for Network operations, is the pre-processed overhead crane image data;

[0024] After detecting the heavy object, the pre-processed overhead crane image data is cropped with the rectangular heavy object target frame as the boundary to obtain the heavy object image data, which is recorded as ;

[0025] S32: Extract features from the heavy object image data to obtain heavy object features, and calculate the required number of feature point pairs based on the heavy object features. The calculation method is:

[0026] ;

[0027] in, For heavy object characteristics, For convolutional neural network operations, is the probability distribution of the number of feature point pairs, is the softmax function, Operates for multi-layer perceptrons;

[0028] Pick The category with the largest probability of feature point pairs is used as the required number of feature point pairs, denoted as ;

[0029] During overhead crane hoisting operations, accurate feature extraction of heavy objects is crucial for ensuring the accuracy of subsequent deep learning calculations and control. Traditional methods often rely on a fixed number of feature points or simple feature extraction algorithms, which are unable to adapt to varying loads and operating environments. Step S32 uses a convolutional neural network (CNN) and a multi-layer perceptron (MLP) combined with a softmax classification model to achieve adaptive feature extraction of heavy objects and intelligent selection of the number of feature points.

[0030] Here, we choose to use the classification model to calculate the number of feature point pairs instead of directly using the multi-layer perceptron to generate a value as the number of feature points. This can avoid insufficient or excessive number of feature points, which in turn affects the accuracy of subsequent depth calculation and control. For the classification model, the number of feature points can be set in advance. The number of feature points generated by the classification model has both adaptive capabilities and does not produce overly extreme values.

[0031] S33: Perform SIFT feature point matching on the heavy object image data and count the number of SIFT feature point pairs. The calculation method is:

[0032] ;

[0033] in, is a SIFT feature point pair, is the number of SIFT feature point pairs, is a SIFT feature point pair descriptor, each SIFT feature point pair corresponds to a SIFT feature point pair descriptor, It is the SIFT feature point matching algorithm.

[0034] Furthermore, the S4 step includes:

[0035] S41: If the number of required feature point pairs is greater than the number of SIFT feature point pairs, then the heavy object features are combined to calculate the supplementary feature point pairs and their descriptors, and the supplementary feature point pairs and their descriptors are merged with the SIFT feature point pairs and their descriptors to obtain the binocular matching feature point pairs and their descriptors. The calculation method is:

[0036] ;

[0037] in, To supplement the features of feature point pairs, is the sigmoid function, To supplement the feature point pairs, To supplement the descriptor of feature point pairs, For splicing operation, is the binocular matching feature point pair, Descriptor for binocular matching feature point pairs;

[0038] S42: If the number of required feature point pairs is less than the number of SIFT feature point pairs, SIFT feature point pairs are deleted in combination with the heavy object features. The calculation method is:

[0039] ;

[0040] in, The features of the SIFT feature point pair that need to be deleted, is the probability distribution of SIFT feature point pairs that need to be deleted;

[0041] Calculate After that, sort the SIFT feature point pairs according to the probability of deletion from large to small, and delete SIFT feature point pairs and their descriptors, the SIFT feature point pairs and their descriptors that are not deleted are binocular matching feature point pairs and their descriptors, which are respectively recorded as ;

[0042] When the number of required feature point pairs is greater than the number of SIFT feature point pairs, step S41 calculates supplementary feature point pairs by combining the heavy object features and SIFT feature point information, and merges them with the original SIFT feature point pairs; when the number of required feature point pairs is less than the number of SIFT feature point pairs, step S42 intelligently selects the feature point pairs that need to be deleted by combining the heavy object features and SIFT feature point information, achieving a balance between the amount of calculation, interference information and feature expression ability, so that the algorithm can cope with heavy objects of different shapes, sizes and textures;

[0043] S43: Based on binocular matching feature point pairs , calculate the binocular depth of the heavy object and the horizontal and vertical coordinates of the center of the heavy object in the camera coordinate system. The calculation process is:

[0044] S431: For each pair of binocular matching feature points, calculate the disparity using the following method:

[0045] ;

[0046] in, According to the i The disparity calculated by binocular matching feature point pairs, i is the index of binocular matching feature point pairs, For the i The horizontal coordinates of the binocular matching feature point pairs in the preprocessed left image data, For the i The horizontal coordinates of the binocular matching feature point pairs in the pre-processed right image data;

[0047] S432: Calculate the depth of each binocular matching feature point pair using the following calculation method:

[0048] ;

[0049] in, For the i The depth of the binocular matching feature point pair, f is the camera focal length, and B is the baseline distance;

[0050] S433: Calculate the weights of the binocular matching feature point pairs, and calculate the binocular depth of the heavy object and the horizontal and vertical coordinates of the center of the heavy object in the camera coordinate system based on the depth of each binocular matching feature point pair. The calculation method is:

[0051] ;

[0052] in, For the i The weight of the binocular matching feature point pairs, For the attention mechanism operation, Binocular depth for heavy objects; are the horizontal and vertical coordinates of the center of the weight in the camera coordinate system respectively.

[0053] By calculating the difference between the horizontal coordinates of the binocular matching feature point pairs in the left and right images, step S43 accurately obtains the disparity information of each feature point pair. After obtaining the disparity information, step S433 introduces an attention mechanism to dynamically adjust the weights of the feature points based on their importance and reliability. This enables the algorithm to more effectively screen out useful feature points in complex environments (such as lighting changes, occlusion, noise, etc.), thereby improving the accuracy and stability of binocular measurement.

[0054] Furthermore, the step S5 includes:

[0055] S51: Perform hook target detection on the pre-processed overhead crane image data to obtain the hook target frame. The calculation method is:

[0056] ;

[0057] in, The hook target frame is a rectangle consisting of 4 vertex coordinates;

[0058] Then, the pre-processed crane image data is cropped with the hook target frame as the boundary to obtain the hook image data, which is recorded as ;

[0059] S52: Count the number of pixels of the hook in the hook image data, and calculate the hook lowering distance by combining the number of pixels of the hook and the hook cable scale. The calculation method is:

[0060] ;

[0061] in, is the number of pixels of the hook in the hook image data, To sum the statistical functions, is the hook lowering distance, Scale for hook and cable;

[0062] In overhead crane hoisting operations, traditional methods often rely on steel cable scales or sensors to measure the hook's lowering distance. However, over long periods of use, the steel cable may gradually become loose due to stress, wear, or environmental factors, resulting in inaccurate scale and, in turn, affecting the measurement accuracy of the hook's distance. Step S52 uses deep learning to count the number of hook pixels in the hook image data. The proportional relationship between the number of pixels and the actual size of the hook is used to more accurately calculate the hook's lowering distance.

[0063] S53: Perform obstacle detection and extract obstacle features on the pre-processed overhead crane image data. The calculation method is:

[0064] ;

[0065] in, is a rectangular obstacle target frame consisting of 4 vertex coordinates. is the preliminary obstacle feature, For VGGNet network operation, is the obstacle feature, are the horizontal, vertical and vertical coordinates of the obstacle in the camera coordinate system respectively; according to Calculated;

[0066] S54: Calculate the hook safety range radius function based on the hook lowering distance and obstacle characteristics. The calculation method is:

[0067] ;

[0068] in, is the hook safety range radius function;

[0069] Traditional fixed safety radius methods often generate unnecessary false alarms in open areas, but the hook safety range radius function effectively reduces this through adaptive adjustment. The hook safety range radius function proposed in this invention can adaptively adjust the safety radius based on the hook's lowering distance and the characteristics of obstacles in the surrounding environment. When the hook is at a high position and there are few obstacles around, the function automatically sets a larger safety radius to allow for more flexible operating space. As the hook is lowered, especially when entering an area with dense obstacles, the function dynamically reduces the safety radius to ensure timely warning of potential collisions.

[0070] S55: When an obstacle is detected within the hook safety radius, an alarm is issued.

[0071] Furthermore, the step S6 includes:

[0072] S61: Combine the horizontal and vertical coordinates of the weight center in the camera coordinate system, the binocular depth of the weight, and the hook target frame to output the overhead crane control parameters. The calculation method is:

[0073] ;

[0074] in, Control parameters for the overhead crane, including the movement of the trolley, carriage, and hook;

[0075] S62: Generate a recommended control instruction based on the calculated overhead crane control parameters. The calculation method is:

[0076] ;

[0077] in, The probability distribution of the recommended control instructions is used to select the recommended control instruction with the highest probability as the recommended control instruction; the recommended control instructions include: trolley to the left, trolley to the right, trolley forward, trolley backward, hook up, hook down, and pause operation.

[0078] The present invention also discloses a machine vision-based overhead crane control system, comprising:

[0079] Image collection and preprocessing module: uses a binocular camera to collect overhead crane image data and preprocesses it to obtain preprocessed overhead crane image data;

[0080] Feature point pair number calculation module: performs heavy object target detection on the pre-processed overhead crane image data, crops the heavy objects after detection, and obtains heavy object image data; performs feature extraction on the heavy object image data and calculates the required number of feature point pairs; performs SIFT feature point matching on the heavy object image data and counts the number of SIFT feature point pairs;

[0081] Binocular matching feature point pair calculation module: Calculates the binocular matching feature point pairs and their descriptors based on the required number of feature point pairs and the number of SIFT feature point pairs, and calculates the binocular depth of the heavy object and the horizontal and vertical coordinates of the center of the heavy object in the camera coordinate system;

[0082] Hook safety range radius function calculation module: performs hook target detection on the pre-processed overhead crane image data to obtain the hook target frame, and then crops the hook image data to calculate the hook lowering distance; performs obstacle target detection on the pre-processed overhead crane image data and extracts obstacle features; combines the hook lowering distance and obstacle features to calculate the hook safety range radius function; and issues an alarm when an obstacle is detected within the hook safety radius.

[0083] Overhead crane control module: Calculates the overhead crane control parameters based on the binocular depth of the weight, the horizontal and vertical coordinates of the weight center in the camera coordinate system, and the hook target frame, and controls the overhead crane according to the overhead crane control parameters.

[0084] Compared with the prior art, the advantages of the present invention are:

[0085] (1) The present invention adopts a binocular camera to extract the binocular depth of heavy objects based on the idea of ​​line of sight calculation; at the same time, the target detection algorithm is combined with the feature extraction network to improve the matching effect of feature point pairs, thereby improving the three-dimensional positioning capability of the system; in addition, the present invention calculates the radius function of the hook safety range and dynamically adjusts the safety radius according to the hook lowering distance and obstacle characteristics, and issues obstacle avoidance warnings in a timely manner, thereby effectively improving the obstacle avoidance capability.

[0086] (2) The present invention uses a convolutional neural network and a multi-layer perceptron, combined with a softmax classification model, and utilizes the idea of ​​the classification model to adaptively calculate the number of feature point pairs, thereby avoiding insufficient or excessive number of feature points, which in turn affects the accuracy of subsequent depth calculation and control.

[0087] (3) The present invention proposes a hook safety range radius function that can adaptively adjust the safety radius according to the hook's lowering distance and the characteristics of obstacles in the surrounding environment; when the hook is at a higher position and there are few obstacles around it, the function will automatically set a larger safety radius to allow for more flexible operating space; as the hook is lowered, especially when entering an area with dense obstacles, the function will dynamically reduce the safety radius to ensure timely warning of potential collisions.

[0088] (4) In order to solve the problem that during overhead crane hoisting operations, the steel cable is gradually loosened due to factors such as force and wear, which leads to inaccurate measurement of the hook lowering distance, the present invention is based on deep learning, which counts the number of pixels of the hook in the hook image data, and calculates the hook lowering distance more accurately through the proportional relationship between the number of pixels and the actual size of the hook. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 A schematic diagram of the process of the overhead crane method based on machine vision provided by the present invention;

[0090] Figure 2 Schematic diagram of the algorithm flow for calculating the required feature point pairs and the number of SIFT feature point pairs provided by the present invention;

[0091] Figure 3 This is a schematic diagram of the algorithm flow for calculating the hook safety range radius function provided by the present invention. DETAILED DESCRIPTION

[0092] The present invention will be further described below with reference to the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention fall within the scope of protection of the present invention.

[0093] Example 1: A method for controlling an overhead crane based on machine vision, such as Figure 1 As shown, the following steps are included:

[0094] S1: Install the binocular camera, calibrate the binocular camera and connect it to the network;

[0095] Two Flea3GigE Vision cameras from Point Grey, Canada, and MO814-MP2 lenses from Computar, Japan, were used to form a binocular camera.

[0096] S11: Install a binocular camera on the trolley in front of the overhead crane. The straight-line distance between the binocular camera and the hook should be no less than 1.5 meters.

[0097] S12: Use Zhang Zhengyou calibration method to calibrate the binocular camera;

[0098] S13: Connect the binocular camera to the industrial computer through a gigabit switch and configure the camera's network communication parameters.

[0099] S2: Using a binocular camera to collect and preprocess the overhead crane image data to obtain preprocessed overhead crane image data;

[0100] S21: capturing image data of the overhead crane in the operating area of ​​the overhead crane in real time using a binocular camera, wherein the image data includes left image data and right image data, and the left image data and the right image data are respectively acquired by two different cameras of the binocular camera;

[0101] S22: performing denoising processing on the overhead crane image data using Gaussian filtering to obtain denoised overhead crane image data;

[0102] S23: Using the Bouguet algorithm to perform stereo correction on the left image data and the right image data in the denoised overhead crane image data to ensure that the two images are on the same plane, thereby obtaining preprocessed overhead crane image data; the preprocessed overhead crane image data includes preprocessed left image data and preprocessed right image data.

[0103] S3: Perform heavy object detection on the pre-processed overhead crane image data, crop the heavy object after detection, and obtain heavy object image data; perform feature extraction on the heavy object image data and calculate the required number of feature point pairs; then perform SIFT feature point matching on the heavy object image data and count the number of SIFT feature point pairs;

[0104] S31: Perform heavy object detection on the pre-processed overhead crane image data and output the horizontal position of the heavy object according to the center coordinates of the heavy object. The calculation method is: ;

[0105] in, are the horizontal and vertical coordinates of the center of the weight in the pre-processed overhead crane image data, is a rectangular heavy object target frame consisting of 4 vertex coordinates; for Network operations, is the pre-processed overhead crane image data;

[0106] After detecting the heavy object, the pre-processed overhead crane image data is cropped with the rectangular heavy object target frame as the boundary to obtain the heavy object image data, which is recorded as ;

[0107] S32: Extract features from the heavy object image data to obtain heavy object features, and calculate the required number of feature point pairs based on the heavy object features. The calculation method is:

[0108] ;

[0109] in, For heavy object characteristics, For convolutional neural network operations, is the probability distribution of the number of feature point pairs, is the softmax function, Operates for multi-layer perceptrons;

[0110] Pick The category with the largest probability of feature point pairs is used as the required number of feature point pairs, denoted as ;

[0111] During overhead crane hoisting operations, accurate feature extraction of heavy objects is crucial for ensuring the accuracy of subsequent deep learning calculations and control. Traditional methods often rely on a fixed number of feature points or simple feature extraction algorithms, which are unable to adapt to varying loads and operating environments. Step S32 uses a convolutional neural network (CNN) and a multi-layer perceptron (MLP) combined with a softmax classification model to achieve adaptive feature extraction of heavy objects and intelligent selection of the number of feature points.

[0112] Here, we choose to use the classification model to calculate the number of feature point pairs instead of directly using the multi-layer perceptron to generate a value as the number of feature points. This can avoid insufficient or excessive number of feature points, which in turn affects the accuracy of subsequent depth calculation and control. For the classification model, the number of feature points can be set in advance. The number of feature points generated by the classification model has both adaptive capabilities and does not produce overly extreme values.

[0113] S33: Perform SIFT feature point matching on the heavy object image data and count the number of SIFT feature point pairs. The calculation method is:

[0114] ;

[0115] in, is a SIFT feature point pair, is the number of SIFT feature point pairs, is a SIFT feature point pair descriptor, each SIFT feature point pair corresponds to a SIFT feature point pair descriptor, It is the SIFT feature point matching algorithm.

[0116] The algorithm flow diagram for calculating the required feature point pairs and the number of SIFT feature point pairs can be shown as follows: Figure 2 shown.

[0117] S4: Calculate the binocular matching feature point pairs and their descriptors based on the required number of feature point pairs and the number of SIFT feature point pairs, and calculate the binocular depth of the heavy object and the horizontal and vertical coordinates of the center of the heavy object in the camera coordinate system;

[0118] S41: If the number of required feature point pairs is greater than the number of SIFT feature point pairs, then the heavy object features are combined to calculate the supplementary feature point pairs and their descriptors, and the supplementary feature point pairs and their descriptors are merged with the SIFT feature point pairs and their descriptors to obtain the binocular matching feature point pairs and their descriptors. The calculation method is:

[0119] ;

[0120] in, To supplement the features of feature point pairs, is the sigmoid function, To supplement the feature point pairs, To supplement the descriptor of feature point pairs, For splicing operation, is the binocular matching feature point pair, Descriptor for binocular matching feature point pairs;

[0121] S42: If the number of required feature point pairs is less than the number of SIFT feature point pairs, SIFT feature point pairs are deleted in combination with the heavy object features. The calculation method is:

[0122] ;

[0123] in, The features of the SIFT feature point pair that need to be deleted, is the probability distribution of SIFT feature point pairs that need to be deleted;

[0124] Calculate After that, sort the SIFT feature point pairs according to the probability of deletion from large to small, and delete SIFT feature point pairs and their descriptors, the SIFT feature point pairs and their descriptors that are not deleted are binocular matching feature point pairs and their descriptors, which are respectively recorded as ;

[0125] When the number of required feature point pairs is greater than the number of SIFT feature point pairs, step S41 calculates supplementary feature point pairs by combining the heavy object features and SIFT feature point information, and merges them with the original SIFT feature point pairs; when the number of required feature point pairs is less than the number of SIFT feature point pairs, step S42 intelligently selects the feature point pairs that need to be deleted by combining the heavy object features and SIFT feature point information, achieving a balance between the amount of calculation, interference information and feature expression ability, so that the algorithm can cope with heavy objects of different shapes, sizes and textures;

[0126] S43: Based on binocular matching feature point pairs , calculate the binocular depth of the heavy object and the horizontal and vertical coordinates of the center of the heavy object in the camera coordinate system. The calculation process is:

[0127] S431: For each pair of binocular matching feature points, calculate the disparity using the following method:

[0128] ;

[0129] in, According to the i The disparity calculated by binocular matching feature point pairs, i is the binocular matching feature point pair index, For the iThe horizontal coordinates of the binocular matching feature point pairs in the preprocessed left image data, For the i The horizontal coordinates of the binocular matching feature point pairs in the pre-processed right image data;

[0130] S432: Calculate the depth of each binocular matching feature point pair using the following calculation method:

[0131] ;

[0132] in, For the i The depth of the binocular matching feature point pair, f is the camera focal length, and B is the baseline distance;

[0133] S433: Calculate the weights of the binocular matching feature point pairs, and calculate the binocular depth of the heavy object and the horizontal and vertical coordinates of the center of the heavy object in the camera coordinate system based on the depth of each binocular matching feature point pair. The calculation method is:

[0134] ;

[0135] in, For the i The weight of the binocular matching feature point pairs, For the attention mechanism operation, Binocular depth for heavy objects; are the horizontal and vertical coordinates of the center of the weight in the camera coordinate system respectively.

[0136] By calculating the difference between the horizontal coordinates of the binocular matching feature point pairs in the left and right images, step S43 accurately obtains the disparity information of each feature point pair. After obtaining the disparity information, step S433 introduces an attention mechanism to dynamically adjust the weights of the feature points based on their importance and reliability. This enables the algorithm to more effectively screen out useful feature points in complex environments (such as lighting changes, occlusion, noise, etc.), thereby improving the accuracy and stability of binocular measurement.

[0137] S5: Perform hook target detection on the pre-processed overhead crane image data to obtain a hook target frame, crop the hook image data, and calculate the hook lowering distance; perform obstacle target detection on the pre-processed overhead crane image data and extract obstacle features; combine the hook lowering distance and obstacle features to calculate the hook safety range radius function; and issue an alarm when an obstacle is detected within the hook safety radius.

[0138] S51: Perform hook target detection on the pre-processed overhead crane image data to obtain the hook target frame. The calculation method is:

[0139] ;

[0140] in, The hook target frame is a rectangle consisting of 4 vertex coordinates;

[0141] Then, the pre-processed crane image data is cropped with the hook target frame as the boundary to obtain the hook image data, which is recorded as ;

[0142] S52: Count the number of pixels of the hook in the hook image data, and calculate the hook lowering distance by combining the number of pixels of the hook and the hook cable scale. The calculation method is:

[0143] ;

[0144] in, is the number of pixels of the hook in the hook image data, To sum the statistical functions, is the hook lowering distance, Scale for hook and cable;

[0145] In overhead crane hoisting operations, traditional methods often rely on steel cable scales or sensors to measure the hook's lowering distance. However, over long periods of use, the steel cable may gradually become loose due to stress, wear, or environmental factors, resulting in inaccurate scale and, in turn, affecting the measurement accuracy of the hook's distance. Step S52 uses deep learning to count the number of hook pixels in the hook image data. The proportional relationship between the number of pixels and the actual size of the hook is used to more accurately calculate the hook's lowering distance.

[0146] S53: Perform obstacle detection and extract obstacle features on the pre-processed overhead crane image data. The calculation method is:

[0147] ;

[0148] in, is a rectangular obstacle target frame consisting of 4 vertex coordinates. is the preliminary obstacle feature, For VGGNet network operation, is the obstacle feature, are the horizontal, vertical and vertical coordinates of the obstacle in the camera coordinate system respectively; according to Calculated;

[0149] S54: Calculate the hook safety range radius function based on the hook lowering distance and obstacle characteristics. The calculation method is:

[0150] ;

[0151] in, is the hook safety range radius function; the algorithm flow diagram for calculating the hook safety range radius function is as follows Figure 3 As shown;

[0152] Traditional fixed safety radius methods often generate unnecessary false alarms in open areas, but the hook safety range radius function effectively reduces this through adaptive adjustment. The hook safety range radius function proposed in this invention can adaptively adjust the safety radius based on the hook's lowering distance and the characteristics of obstacles in the surrounding environment. When the hook is at a high position and there are few obstacles around, the function automatically sets a larger safety radius to allow for more flexible operating space. As the hook is lowered, especially when entering an area with dense obstacles, the function dynamically reduces the safety radius to ensure timely warning of potential collisions.

[0153] S55: An alarm is issued when an obstacle is detected within the hook safety radius.

[0154] In particular, when there is a moving target within the hook range, the intention of the moving target is identified and the calculation method is:

[0155] ;

[0156] in, is the trajectory of the moving target, is the KTL target tracking algorithm, is the position coordinate of the moving target in the next frame, It is a gated recurrent unit operation;

[0157] If the moving target is outside the hook safety radius, or moves outside the hook safety radius within the next 5 frames, the moving target will be ignored;

[0158] If the moving target is within the hook safety radius, or moves within the hook safety radius within the next 5 frames, an alarm will be issued.

[0159] S6: Calculate the control parameters of the overhead crane based on the binocular depth of the weight, the horizontal and vertical coordinates of the weight center in the camera coordinate system, and the hook target frame, and control the overhead crane according to the control parameters;

[0160] S61: Combine the horizontal and vertical coordinates of the weight center in the camera coordinate system, the binocular depth of the weight, and the hook target frame to output the overhead crane control parameters. The calculation method is:

[0161] ;

[0162] in, Control parameters for the overhead crane, including the movement of the trolley, carriage, and hook;

[0163] S62: Generate a recommended control instruction based on the calculated overhead crane control parameters. The calculation method is:

[0164] ;

[0165] in, The probability distribution of the recommended control instructions is used to select the recommended control instruction with the highest probability as the recommended control instruction; the recommended control instructions include: trolley to the left, trolley to the right, trolley forward, trolley backward, hook up, hook down, and pause operation.

[0166] For example, an overhead crane is lifting a heavy object. The overhead crane control parameter K is calculated by step S61. Next, we input the overhead crane control parameter K into a Softmax classifier to calculate the probability distribution of each recommended control instruction. , It is a 7-dimensional vector, corresponding to the probabilities of the 7 recommended control instructions:

[0167] ;

[0168] Suppose at some point,

[0169] in, The maximum value is obtained, so “cart forward” is selected as the recommended control instruction and sent to the overhead crane;

[0170] Example 2: The present invention also discloses a crane control system based on machine vision, comprising:

[0171] Image collection and preprocessing module: uses a binocular camera to collect overhead crane image data and preprocesses it to obtain preprocessed overhead crane image data;

[0172] Feature point pair number calculation module: performs heavy object target detection on the pre-processed overhead crane image data, crops the heavy objects after detection, and obtains heavy object image data; performs feature extraction on the heavy object image data and calculates the required number of feature point pairs; performs SIFT feature point matching on the heavy object image data and counts the number of SIFT feature point pairs;

[0173] Binocular matching feature point pair calculation module: Calculates the binocular matching feature point pairs and their descriptors based on the required number of feature point pairs and the number of SIFT feature point pairs, and calculates the binocular depth of the heavy object and the horizontal and vertical coordinates of the center of the heavy object in the camera coordinate system;

[0174] Hook safety range radius function calculation module: performs hook target detection on the pre-processed overhead crane image data to obtain the hook target frame, and then crops the hook image data to calculate the hook lowering distance; performs obstacle target detection on the pre-processed overhead crane image data and extracts obstacle features; combines the hook lowering distance and obstacle features to calculate the hook safety range radius function; and issues an alarm when an obstacle is detected within the hook safety radius.

[0175] Overhead crane control module: Calculates the overhead crane control parameters based on the binocular depth of the weight, the horizontal and vertical coordinates of the weight center in the camera coordinate system, and the hook target frame, and controls the overhead crane according to the overhead crane control parameters.

[0176] It should be noted that the serial numbers of the above-mentioned embodiments of the present invention are for descriptive purposes only and do not represent the advantages or disadvantages of the embodiments. In addition, the terms "including", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method comprising the element.

[0177] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in multiple embodiments of the present invention.

[0178] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for controlling an overhead crane based on machine vision, characterized in that: The following steps are involved: S1: Install the binocular camera, calibrate the binocular camera and connect it to the network; S2: using a binocular camera to collect and preprocess the image data of the overhead crane to obtain the preprocessed image data of the overhead crane; S3: Perform heavy object target detection on the preprocessed overhead crane image data, crop the heavy object after detection, and obtain heavy object image data; perform feature extraction on the heavy object image data, and calculate the number of required feature point pairs; Then perform SIFT feature point matching on the heavy object image data and count the number of SIFT feature point pairs; The S3 step includes: S31: Perform heavy object target detection on the pre-processed overhead crane image data, and output the horizontal position of the heavy object according to the center coordinates of the heavy object. The calculation method is: ； in, are the horizontal and vertical coordinates of the center of the weight in the preprocessed overhead crane image data, is a rectangular heavy object target frame consisting of 4 vertex coordinates; for Network operations, is the pre-processed overhead crane image data; After the heavy object is detected, the pre-processed overhead crane image data is cropped with the rectangular heavy object target frame as the boundary to obtain the heavy object image data, which is recorded as ; S32: Extract features from the heavy object image data to obtain heavy object features, and calculate the number of required feature point pairs based on the heavy object features. The calculation method is: ； in, For heavy object characteristics, For convolutional neural network operations, is the probability distribution of the number of feature point pairs, is the softmax function, Operate for multi-layer perceptron; Pick The category of feature point pairs with the largest probability is taken as the required number of feature point pairs, denoted as ; S33: Perform SIFT feature point matching on the heavy object image data and count the number of SIFT feature point pairs. The calculation method is: ； in, is a SIFT feature point pair, is the number of SIFT feature point pairs, is a SIFT feature point pair descriptor, each SIFT feature point pair corresponds to a SIFT feature point pair descriptor, It is SIFT feature point matching algorithm; S4: If the number of required feature point pairs is greater than the number of SIFT feature point pairs, then the heavy object features are combined to calculate supplementary feature point pairs and their descriptors, and the supplementary feature point pairs and their descriptors are merged with the SIFT feature point pairs and their descriptors to obtain binocular matching feature point pairs and their descriptors; if the number of required feature point pairs is less than the number of SIFT feature point pairs, then the heavy object features are combined to delete the SIFT feature point pairs to obtain binocular matching feature point pairs and their descriptors; based on the binocular matching feature point pairs, the binocular depth of the heavy object and the horizontal and vertical coordinates of the center of the heavy object in the camera coordinate system are calculated; S5: Perform hook target detection on the preprocessed overhead crane image data to obtain a hook target frame, and obtain hook image data after cropping, and calculate the hook lowering distance; perform obstacle target detection on the preprocessed overhead crane image data and extract obstacle features; calculate the hook safety range radius function by combining the hook lowering distance and obstacle features; and issue an alarm when an obstacle is detected within the hook safety radius; S6: Based on the binocular depth of the heavy object, the horizontal and vertical coordinates of the center of the heavy object in the camera coordinate system, and the hook target frame, the overhead crane control parameters are calculated, and the overhead crane is controlled according to the overhead crane control parameters.

2. The method for controlling an overhead crane based on machine vision according to claim 1, characterized in that: The S1 step includes: S11: Install a binocular camera on the trolley in front of the overhead crane trolley. The straight-line distance between the binocular camera and the hook should be no less than 1.5 meters. S12: Use Zhang Zhengyou calibration method to calibrate the binocular camera; S13: Connect the binocular camera to the industrial computer through a gigabit switch and configure the network communication parameters of the camera.

3. The method for controlling an overhead crane based on machine vision according to claim 1, characterized in that: The S2 step includes: S21: capturing image data of the overhead crane in the operating area of ​​the overhead crane in real time through a binocular camera, wherein the image data of the overhead crane includes left image data and right image data, and the left image data and the right image data are respectively acquired by two different cameras of the binocular camera; S22: performing denoising processing on the overhead crane image data by using Gaussian filtering to obtain denoised overhead crane image data; S23: using the Bouguet algorithm to perform stereoscopic correction on the left image data and the right image data in the denoised overhead crane image data to ensure that the two images are on the same plane, thereby obtaining preprocessed overhead crane image data; the preprocessed overhead crane image data includes preprocessed left image data and preprocessed right image data.

4. The method for controlling an overhead crane based on machine vision according to claim 1, characterized in that: The S4 step includes: S41: If the number of required feature point pairs is greater than the number of SIFT feature point pairs, the heavy object features are combined to calculate the supplementary feature point pairs and their descriptors, and the supplementary feature point pairs and their descriptors are merged with the SIFT feature point pairs and their descriptors to obtain the binocular matching feature point pairs and their descriptors. The calculation method is: ； in, To supplement the features of feature point pairs, is the sigmoid function, To supplement the feature point pairs, To supplement the descriptor of feature point pairs, For splicing operation, is the binocular matching feature point pair, Descriptor for binocular matching feature point pairs; S42: If the number of required feature point pairs is less than the number of SIFT feature point pairs, SIFT feature point pairs are deleted in combination with the heavy object features. The calculation method is: ； in, is the feature of the SIFT feature point pair that needs to be deleted. is the probability distribution of SIFT feature point pairs that need to be deleted; Calculate After that, sort the SIFT feature point pairs according to the probability of deletion from large to small, and delete SIFT feature point pairs and their descriptors, the undeleted SIFT feature point pairs and their descriptors are binocular matching feature point pairs and their descriptors, respectively recorded as ; S43: According to the binocular matching feature point pair , calculate the binocular depth of the heavy object and the horizontal and vertical coordinates of the center of the heavy object in the camera coordinate system. The calculation process is: S431: For each pair of binocular matching feature points, calculate the disparity. The calculation method is: ； in, According to i The disparity calculated by the binocular matching feature point pairs, i is the index of binocular matching feature point pairs, For the i The horizontal coordinates of the binocular matching feature point pairs in the preprocessed left image data, For the i The horizontal coordinates of the binocular matching feature point pairs in the preprocessed right image data; S432: Calculate the depth of each pair of binocular matching feature points, and the calculation method is: ； in, For the i The depth of a pair of binocular matching feature points, f is the focal length of the camera, and B is the baseline distance; S433: Calculate the weight of the binocular matching feature point pairs, and calculate the binocular depth of the heavy object and the horizontal and vertical coordinates of the center of the heavy object in the camera coordinate system according to the depth of each binocular matching feature point pair. The calculation method is: ； in, For the i The weight of the binocular matching feature point pairs, is the attention mechanism operation, Binocular depth for heavy objects; are the horizontal and vertical coordinates of the center of the weight in the camera coordinate system respectively.

5. The method for controlling an overhead crane based on machine vision according to claim 4, characterized in that: The S5 step includes: S51: Perform hook target detection on the pre-processed overhead crane image data to obtain a hook target frame. The calculation method is: ； in, The hook target box is a rectangle consisting of 4 vertex coordinates; Then, the pre-processed crane image data is cropped with the hook target frame as the boundary to obtain the hook image data, which is recorded as ; S52: Count the number of pixels of the hook in the hook image data, and calculate the hook lowering distance by combining the number of pixels of the hook and the hook cable scale. The calculation method is: ； in, is the number of pixels of the hook in the hook image data, is the summation statistical function, is the hook lowering distance, It is the scale for the hook and cable; S53: Perform obstacle target detection and extract obstacle features on the pre-processed overhead crane image data. The calculation method is: ； in, is a rectangular obstacle target frame consisting of four vertex coordinates. is the preliminary obstacle feature, For VGGNet network operation, is the obstacle feature, are the horizontal, vertical and vertical coordinates of the obstacle in the camera coordinate system respectively; according to Calculated; S54: Calculate the hook safety range radius function by combining the hook lowering distance and obstacle characteristics. The calculation method is: ； in, is the radius function of the hook safety range; S55: When an obstacle is detected within the safety radius of the hook, an alarm is issued.

6. The method for controlling an overhead crane based on machine vision according to claim 5, characterized in that: The S6 step includes: S61: Combine the horizontal and vertical coordinates of the center of the weight in the camera coordinate system, the binocular depth of the weight, and the hook target frame to output the overhead crane control parameters. The calculation method is: ； in, It is the control parameter of the overhead crane, including the movement of the trolley, carriage and hook; S62: Generate a recommended control instruction according to the calculated overhead crane control parameters. The calculation method is: ； in, The probability distribution of the recommended control instructions is selected as the recommended control instructions; the recommended control instructions include: large vehicle to the left, large vehicle to the right, small vehicle to the front, small vehicle to the back, hook up, hook down and pause operation.

7. A crane control system based on machine vision, characterized in that: include: Image collection and preprocessing module: using binocular cameras to collect overhead crane image data and preprocess them to obtain preprocessed overhead crane image data; Feature point pair quantity calculation module: perform heavy object target detection on the pre-processed overhead crane image data, crop the heavy object after detection, and obtain heavy object image data; perform feature extraction on the heavy object image data and calculate the required number of feature point pairs; perform SIFT feature point matching on the heavy object image data and count the number of SIFT feature point pairs; Binocular matching feature point pair calculation module: According to the required number of feature point pairs and the number of SIFT feature point pairs, the binocular matching feature point pairs and their descriptors are calculated, and the binocular depth of the heavy object and the horizontal and vertical coordinates of the center of the heavy object in the camera coordinate system are calculated; Hook safety range radius function calculation module: perform hook target detection on the pre-processed overhead crane image data to obtain the hook target frame, and obtain the hook image data after cutting, and calculate the hook lowering distance; perform obstacle target detection on the pre-processed overhead crane image data and extract obstacle features; Combine the hook lowering distance and obstacle characteristics to calculate the hook safety range radius function; when an obstacle is detected within the hook safety radius, an alarm is issued; Overhead crane control module: Based on the binocular depth of the heavy object, the horizontal and vertical coordinates of the center of the heavy object in the camera coordinate system, and the hook target frame, the overhead crane control parameters are calculated, and the overhead crane is controlled according to the overhead crane control parameters; To realize a crane control method based on machine vision as described in any one of claims 1-6.

Images