A method and system for automatically lifting workpieces based on machine vision

Through the automated lifting method based on machine vision, the lifting ring recognition model and neural network controller are used to realize automatic movement of the lifting trolley and hook locking, solving the problems of workers' safety risks and low operating efficiency, and achieving efficient automatic lifting of workpieces rise.

CN118387769BActive Publication Date: 2024-11-15WUXI GENERAL CRANE & CARRIER
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410846285.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-27
Publication Date
2024-11-15
Estimated Expiration
2044-06-27

AI Technical Summary

Technical Problem

Workers need to get close to heavy objects and lifting equipment during lifting, which poses high safety risks and relies on manual operations to be time-consuming and labor-intensive.

Method used

Using the machine vision-based workpiece automatic lifting method, the lifting trolley automatically moves to the lifting position by building a lifting ring recognition model and a neural network-based PID controller, and automatically locks the hook and the lifting ring through depth sensors and electromagnets. and lift.

Benefits of technology

Automatic lifting of workpieces is realized, reducing manual participation, reducing safety risks and improving production efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118387769B_ABST
    Figure CN118387769B_ABST
Patent Text Reader

Abstract

The invention provides a method and system for automatically lifting a workpiece based on machine vision, and relates to the technical field of cranes. The method comprises: constructing a lifting ring recognition model based on machine vision; constructing a PID controller based on a neural network for a bridge crane; acquiring a workpiece image; determining the lifting ring position through the lifting ring recognition model; determining the target position of the crane trolley with the lifting trolley being located just above the lifting ring position as a target; acquiring the current position of the crane trolley; calculating the system control force through a PID controller based on a neural network according to the difference between the current position of the crane trolley and the target position of the crane trolley; controlling the lifting trolley to move to just above the lifting ring position; controlling the lowering of a lifting hook, acquiring the relative position between the lifting hook and the lifting ring in real time through a depth sensor arranged on the lifting hook, and turning on an electromagnet to lock the lifting hook and the lifting ring when the distance between the lifting hook and the lifting ring is less than a preset distance; and controlling the lifting hook to rise to lift the workpiece.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of cranes, and in particular to a method and system for automatically lifting a workpiece based on machine vision. Background Art

[0002] Bridge crane is a kind of lifting equipment widely used in factories, warehouses, ports and other places. It can lift and move heavy objects within a certain area, and has the advantages of high efficiency, safety and flexible operation.

[0003] When lifting workpieces with a bridge crane, workers are often required to control it through a remote control. After the lifting trolley carries the hook to the designated location, the worker connects the hook with the lifting ring and then controls the crane to lift the workpiece through the remote control. During the lifting process, the worker needs to be close to the heavy objects and lifting equipment, which poses a high safety risk. At the same time, it is time-consuming and labor-intensive to rely on manual operation. Summary of the invention

[0004] In order to solve the technical problems in the prior art that workers need to be close to heavy objects and lifting equipment during the lifting process, which poses a high safety risk, and relies on manual operation which is time-consuming and labor-intensive, the present invention provides a workpiece automatic lifting method and system based on machine vision.

[0005] The technical solution provided by the embodiment of the present invention is as follows:

[0006] First aspect:

[0007] An embodiment of the present invention provides a method for automatically lifting a workpiece based on machine vision, comprising:

[0008] S1: Build a ring recognition model based on machine vision;

[0009] S2: Construct a neural network based PID controller for overhead crane;

[0010] S3: Acquire workpiece image;

[0011] S4: identifying the lifting ring in the workpiece image by using the lifting ring recognition model to determine the position of the lifting ring;

[0012] S5: According to the lifting ring position, the target position of the crane trolley is determined with the lifting trolley being located directly above the lifting ring position as the target;

[0013] S6: Get the current position of the crane trolley;

[0014] S7: Calculating the system control force through the neural network-based PID controller according to the difference between the current position of the crane trolley and the target position of the crane trolley;

[0015] S8: According to the system control force, control the lifting trolley to move to the position just above the lifting ring;

[0016] S9: Control the hook to be lowered, and obtain the relative position between the hook and the ring in real time through the depth sensor set on the hook. When the distance between the hook and the ring is less than the preset distance, turn on the electromagnet to lock the hook and the ring;

[0017] S10: Control the hook to rise and lift the workpiece.

[0018] Second aspect:

[0019] An embodiment of the present invention provides a workpiece automatic lifting system based on machine vision, comprising:

[0020] processor;

[0021] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method for automatically lifting a workpiece based on machine vision as described in the first aspect is implemented.

[0022] The third aspect:

[0023] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method for automatically lifting a workpiece based on machine vision as described in the first aspect is implemented.

[0024] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0025] In the present invention, the lifting ring in the workpiece image is identified through the lifting ring recognition model to determine the position of the lifting ring, and then the lifting trolley is controlled to move to just above the lifting ring position, the hook is controlled to be lowered, and the relative position between the hook and the lifting ring is acquired in real time through the depth sensor arranged on the hook. When the distance between the hook and the lifting ring is less than the preset distance, the electromagnet is turned on to lock the hook and the lifting ring, and the hook is controlled to rise to lift the workpiece. Automatic lifting of the workpiece can be realized without human intervention, thereby reducing safety risks and improving production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1A schematic diagram of a flow chart of a method for automatically lifting a workpiece based on machine vision provided by an embodiment of the present invention;

[0028] Figure 2 A schematic structural diagram of a workpiece automatic lifting system based on machine vision provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0030] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0031] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0032] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0033] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0034] Reference Manual Attached Figure 1 , showing a flow chart of a method for automatically lifting a workpiece based on machine vision provided in an embodiment of the present invention.

[0035] The embodiment of the present invention provides a method for automatically lifting a workpiece based on machine vision, which can be implemented by an automatic lifting device for a workpiece based on machine vision, and the automatic lifting device for a workpiece based on machine vision can be a terminal or a server. The processing flow of the method for automatically lifting a workpiece based on machine vision may include the following steps:

[0036] S1: Build a ring recognition model based on machine vision.

[0037] S2: Construct a neural network based PID controller for overhead crane.

[0038] In a possible implementation, the neural network-based PID controller includes an input layer, a hidden layer, and an output layer.

[0039] The input layer is used to input real-time displacement deviation.

[0040] Optionally, the displacement deviation is specifically:

[0041] e t =x t -x g

[0042] Where, e represents the displacement deviation, e t represents the displacement deviation at time t, x represents the displacement of the crane trolley, and x t represents the displacement at time t, x g Indicates the target displacement.

[0043] The hidden layer includes proportional neurons, integral neurons and differential neurons, which are used to control the proportional gain coefficient, the integral gain coefficient and the differential gain coefficient respectively.

[0044] The hidden layer is used to determine the real-time proportional gain coefficient, integral gain coefficient and derivative gain coefficient during the crane control process.

[0045] The output layer is used to output system control force.

[0046] Traditional PID controllers have fixed proportional, integral and differential gain coefficients, which usually need to be adjusted through experiments during the design and debugging phase. However, the neural network-based PID controller can automatically adjust these gain coefficients according to the real-time status of the system, thereby adapting to different working conditions and environmental changes, and improving the response speed and stability of the control system.

[0047] Optionally, the output of the ratio neuron is specifically:

[0048] f1(t)=e(t)K p (t)

[0049] Among them, f1(t) represents the output of the proportional neuron at time t, e(t) represents the displacement deviation at time t, and K p (t) represents the proportional gain coefficient at time t.

[0050] Optionally, the output of the integrating neuron is specifically:

[0051] f2(t)=e(t)K i (t)+f2(t-1)

[0052] Among them, f2(t) represents the output of the integrating neuron at time t, Ki (t) represents the integral gain coefficient at time t, and f2(t-1) represents the output of the integrating neuron at time t-1.

[0053] Optionally, the output of the differentiable neuron is specifically:

[0054] f3(t)=e(t)K d (t)+K d (t-1)e(t-1)

[0055] Among them, f3(t) represents the output of the differential neuron at time t, K d (t) represents the differential gain coefficient at time t, K d (t-1) represents the differential gain coefficient at time t-1, and e(t-1) represents the displacement deviation at time t-1.

[0056] Optionally, the system control force output by the output layer is specifically:

[0057] F(t)=w1f1(t)+w2f2(t)+w3f3(t)

[0058] Among them, F(t) represents the system control force output at time t, w1 represents the weight coefficient between the proportional neuron and the output layer, w2 represents the weight coefficient between the integral neuron and the output layer, and w3 represents the weight coefficient between the differential neuron and the output layer.

[0059] Among them, those skilled in the art can set the weight coefficient w1 between the proportional neuron and the output layer, the weight coefficient w2 between the integral neuron and the output layer, and the weight coefficient w3 between the differential neuron and the output layer according to actual conditions, and the present invention does not limit this.

[0060] Specifically, the best weight coefficient can be determined by optimization algorithms such as particle swarm optimization algorithm and genetic algorithm.

[0061] In the present invention, through the neural network-based PID controller, precise control of the bridge crane can be achieved, control parameters can be dynamically adjusted, the control accuracy and robustness of the system can be improved, the workload of manual parameter adjustment can be reduced, it can adapt to complex nonlinear systems, and it has good compatibility and scalability.

[0062] Furthermore, during the control process, the kinematic model of the bridge crane is constructed:

[0063]

[0064] Among them, M represents the mass of the trolley, m represents the load mass, and x represents the displacement of the trolley. represents the acceleration of the crane trolley, L represents the cable length, θ represents the load swing angle, represents the load angular velocity, represents the load angular acceleration, and F represents the system control force.

[0065] The state vector and control variables of the bridge crane are constructed, and the state equation of the bridge crane in the nonlinear state space is determined according to the kinematic model.

[0066] The state vector of the bridge crane is specifically:

[0067] X = [x1, x2, x3, x4]

[0068]

[0069] Among them, X represents the state vector, x1, x2, x3 and x4 are the elements in the state vector, x1 corresponds to the displacement x of the crane, and x2 corresponds to the speed of the crane. x3 corresponds to the load swing angle θ, and x4 corresponds to the load angular velocity

[0070] The specific control variable is: system control force F.

[0071] The state equation of the bridge crane in the nonlinear state space is specifically:

[0072]

[0073] Wherein, a represents the comprehensive quality, a=M+m.

[0074] It should be noted that in the nonlinear state space, the behavior of the system is closer to the actual situation. By constructing the nonlinear state space equation, it can provide a basis for designing nonlinear controllers (such as fuzzy control, sliding mode control, etc.), so that the controller can better cope with the nonlinear characteristics and complex dynamics of the system.

[0075] In the present invention, by constructing the state vector and control variables of the bridge crane and determining its state equation in the nonlinear state space, a powerful tool and foundation can be provided for system control, simulation, analysis and optimization. It can not only improve the control accuracy and response speed of the system, but also enhance the robustness and stability of the system, adapt to various complex working conditions, and finally realize the efficient and safe operation of the bridge crane.

[0076] At zero swing angle, the state equation of the nonlinear state space is linearized to determine the state equation of the bridge crane in the linear state space.

[0077] In a possible implementation, at zero swing angle x3=0, then: sin(x3)=x3,cos(x3)=1.

[0078] The state equation of the bridge crane in the linear state space is specifically:

[0079]

[0080] It should be noted that linearization at zero swing angle can achieve higher control accuracy when the system state is close to this point, which is especially important for the initial startup and light load operation of the crane. The linearized state equation is more easily applied to classical linear control methods, such as PID control, LQR (linear quadratic regulator), etc. These methods are effective in dealing with linear systems, and their design and implementation are relatively mature and simple.

[0081] In the present invention, by linearizing the nonlinear state space equation of the bridge crane at the zero swing angle, the design and implementation of the controller can be simplified, the system analysis can be facilitated, the control accuracy can be improved, the calculation and simulation process can be simplified, and the overall performance and reliability of the system can be improved. This method is particularly suitable for precise control when the system state is close to the linearization point (small angle swing).

[0082] Furthermore, according to the state equation of the linear state space, the disturbance observer of the bridge crane is constructed, and the disturbance term of the system control force is added to the state equation of the linear state space:

[0083]

[0084] Here, d represents the disturbance.

[0085] Construct a disturbance observer for an overhead crane:

[0086]

[0087] in, represents the disturbance observation value, ε1 and ε2 represent auxiliary variables, λ represents the observer gain, Represents the state estimate of x2, x3, and x4.

[0088] Among them, those skilled in the art can set the size of the observer gain λ according to actual conditions, and the present invention does not limit this.

[0089] Specifically, the specific size of the observer gain can be determined by pole placement method, LQR design method, etc.

[0090] Pole placement is a common method that determines the gain of the observer by placing the poles of the observer at the desired position. Specifically, the desired pole position is selected according to the response requirements of the system (usually selected in the left half plane to ensure the stability of the system). The characteristic equation of the observer is constructed using the state space equation of the system. The observer gain is determined by matching the coefficients in the characteristic equation.

[0091] In the present invention, by adding the disturbance term of the system control force into the state equation and constructing a disturbance observer, real-time estimation and compensation of the disturbance can be achieved. This can not only enhance the robustness and control accuracy of the system, but also simplify the design of the controller and improve the response speed and adaptability of the system.

[0092] S3: Acquire the workpiece image.

[0093] S4: Identify the lifting ring in the workpiece image through the lifting ring recognition model to determine the position of the lifting ring.

[0094] In a possible implementation, S4 specifically includes sub-steps S401 to S406:

[0095] S401: transforming the workpiece image into RGB color space, HSV color space and Lab color space.

[0096] Among them, the RGB color space is a color model based on the additive principle, which represents colors by combining different intensities of the three basic colors of red, green, and blue. The HSV color space consists of hue, saturation, and value. It is closer to the way humans perceive colors and is often used in image processing and computer vision. The Lab color space is based on the CIE 1931XYZ color space and consists of L* (brightness), a* (green to red), and b* (blue to yellow). It is a perceptually uniform color space designed to more evenly reflect human visual perception.

[0097] S402: Obtain a color histogram of the workpiece image in each color channel.

[0098] It should be noted that the three color spaces have a total of 9 channels.

[0099] S403: extracting a variety of color features in each color space from the color histogram, the color features including mean, variance, standard deviation, skewness, kurtosis, entropy, energy and mode.

[0100] Among them, the mean is the average value of all values ​​in the data set, reflecting the central tendency of the data. In image processing, the mean can represent the average brightness of the color channel. Variance is an indicator of the degree of dispersion of data distribution, which represents the average of the squared differences between each data point and the mean. The larger the variance, the more dispersed the data is. The standard deviation is the square root of the variance, reflecting the degree of dispersion of the data. The larger the standard deviation, the wider the distribution of the data. Skewness describes the symmetry of the data distribution. Positive skewness indicates that the data is right-skewed, and negative skewness indicates that the data is left-skewed. A skewness of zero indicates that the data is symmetrically distributed. Kurtosis is a measure of the steepness of the data distribution. High kurtosis indicates that the data distribution is concentrated around the mean, and low kurtosis indicates that the data distribution is relatively flat. Entropy is an indicator of the uncertainty or randomness of the data. In image processing, entropy is used to describe the amount of information and complexity of the image. Energy is the sum of the squares of the sum of the pixel values ​​in the image, reflecting the smoothness and structure of the image. High energy indicates that the image is relatively smooth. The mode is the value that appears most frequently in the data set. In image processing, the mode represents the pixel value with the highest frequency, reflecting the common colors of the image.

[0101] S404: Combining multiple color features in each color space into a 1×8 feature vector to obtain an RGB feature vector, an HSV feature vector, and a Lab feature vector.

[0102] S405: Perform feature fusion on the RGB feature vector, the HSV feature vector and the Lab feature vector to obtain a fused feature vector.

[0103] Optionally, S405 specifically includes: performing feature fusion according to the following formula:

[0104] X=λ RGB x RGB +λ HSV x HSV +λ Lab x Lab

[0105] Among them, X represents the fused feature vector, x RGB represents the RGB feature vector, λ RGB represents the fusion coefficient of RGB feature vector, x HSV represents the HSV eigenvector, λ HSV Represents the fusion coefficient of HSV feature vector, x Lab represents the Lab eigenvector, λ Lab Represents the fusion coefficient of Lab feature vector.

[0106] In the present invention, the features from different color spaces are fused to more fully and accurately represent the color characteristics of the image. Different color spaces emphasize different aspects of color, and combining them can capture more information and details. By fusing features from different color spaces, the model can use more color information for classification and recognition, which usually improves the accuracy of classification and recognition. For example, some color patterns may not be obvious in RGB space, but may be more obvious in HSV or Lab space. Fusion of these features helps to improve the robustness of the model.

[0107] S406: Input the fused feature vector of the workpiece image into the U-Net network, identify the lifting ring in the workpiece image, and determine the position of the lifting ring.

[0108] Among them, the U-Net network is a convolutional neural network (CNN) that can effectively use limited image data for efficient segmentation during training and achieve efficient target detection.

[0109] Optionally, S406 specifically includes:

[0110] In the convolution layer of the U-Net network, a convolution operation is performed on the fused feature vector to extract the features of the fused feature vector.

[0111] In the pooling layer of the U-Net network, convolution operations are performed on the extracted features to perform dimensionality reduction.

[0112] In the deconvolution layer of the U-Net network, the extracted features are deconvolved.

[0113] In the normalization layer of the U-Net network, the output of each layer is normalized.

[0114] In the classification layer of the U-Net network, the probability that each pixel in the workpiece image is a ring is determined through the Softmax activation function.

[0115] In the output layer of the U-Net network, pixels with probabilities greater than the preset probability are grouped into a ring area, and the ring position is output.

[0116] In the present invention, by inputting the fused feature vector of the workpiece image into the U-Net network, the powerful image segmentation capability of U-Net can be fully utilized to accurately identify and locate the position of the lifting ring in the workpiece image, thereby improving the efficiency and accuracy of the automated lifting operation.

[0117] In a possible implementation, the training method of the ring recognition model specifically includes:

[0118] Construct the loss function for the ring recognition model.

[0119] Optionally, the loss function of the ring recognition model is specifically:

[0120] L1=μ1L CIOU +μ2L cls +μ3L conf +μ4‖|θ||

[0121] Among them, L1 represents the loss function of the SDD network, L CIOU represents the bounding box loss function, μ1 represents the weight of the bounding box loss function, and L cls represents the cross entropy loss function, μ2 represents the weight of the cross entropy loss function, L conf represents the confidence loss function, μ3 represents the weight of the confidence loss function, θ represents the model parameters, ||θ|| represents the modulo operation on the model parameters, and μ4 represents the weight of the regularization term.

[0122] Among them, the bounding box loss function can minimize the distance between the predicted bounding box and the true bounding box. Ensure that the model can accurately locate the position of the ring, so that the predicted bounding box is as close to the true bounding box as possible, and improve the positioning accuracy. The cross entropy loss function can minimize the difference between the predicted category probability distribution and the true category probability distribution. Ensure that the model can accurately classify the ring and the background, and improve the accuracy of classification. The confidence loss function can minimize the difference between the confidence predicted by the model and the actual confidence. Ensure that the model's confidence in its prediction is accurate, so that high-confidence predictions are more reliable, and low-confidence predictions can be effectively identified and processed. The regularization term can minimize the norm of the model parameters. Prevent model overfitting, promote the sparsity and stability of model parameters, and thus improve the generalization ability of the model on test data.

[0123] In the present invention, the comprehensive use of multiple loss functions can provide more information and gradient signals during the training process, which helps to accelerate model convergence and improve training efficiency. Each loss function provides a different optimization direction, so that the model can find the optimal solution faster.

[0124] With the goal of minimizing the loss function of the lifting ring recognition model, the model parameters of the lifting ring recognition model are optimized by the gradient descent method.

[0125]

[0126] Among them, θ t+1 represents the model parameters at the t+1th iteration, θ t represents the model parameters at the tth iteration, α t represents the learning rate at the tth iteration, τ t represents the friction coefficient of the model parameters at the tth iteration, represents the first-order moment after the correction of the model parameter deviation at the tth iteration, represents the second-order moment of the model parameter deviation corrected at the t-th iteration, and ε represents a hyperparameter to prevent the denominator from being zero.

[0127] By minimizing the loss function of the ring recognition model through the gradient descent method, the accuracy and robustness of the model can be significantly improved, the model's automatic learning ability and generalization ability can be enhanced, while reducing the computational complexity and improving the training efficiency.

[0128] The specific learning rate is:

[0129]

[0130] Among them, α t represents the learning rate at the tth iteration, α0 represents the initial learning rate, g k represents the gradient of the model parameters at the kth iteration, k = 1, 2, …, t, and δ represents the hyperparameter.

[0131] The gradient is:

[0132]

[0133] Among them, g represents the gradient, L represents the loss function, and θ represents the model parameters.

[0134] It should be noted that as the training process progresses, the cumulative sum of squares of the gradient increases and the learning rate gradually decreases. A larger learning rate is used in the early stages of training so that the model can quickly approach the optimal solution. In the later stages of training, reducing the learning rate helps the model fine-tune parameters within a smaller range, avoid oscillations, and improve the final convergence accuracy.

[0135] The friction coefficient is:

[0136]

[0137] Among them, exp represents the exponential function with natural constant as the base, g t-1 represents the gradient of the model parameters at the t+1th iteration, g t Represents the gradient of the model parameters at the tth iteration.

[0138] It should be noted that the introduction of friction coefficient and dynamic adjustment according to gradient changes can effectively improve the stability and adaptability of parameter updates, control the parameter update step size, avoid local optimality and premature convergence, and enhance numerical stability. When the gradient changes drastically, the step size should be reduced to prevent oscillation and instability caused by excessive parameter updates; when the gradient changes gently, the step size should be increased to speed up convergence.

[0139] The first-order moment after deviation correction is specifically:

[0140]

[0141] m t =β1m t-1 +(1-β1)g t

[0142] Among them, m t represents the mean gradient of the model parameters at the tth iteration, represents the t-th power of β1, β1 represents the first attenuation rate, m t-1 Represents the mean gradient of the model parameters at the t-1th iteration.

[0143] The second-order moment after deviation correction is specifically:

[0144]

[0145] v t =β2v t-1 +(1-β2)v t

[0146] Among them, v t represents the gradient variance of the model parameters at the tth iteration, represents the t-th power of β2, β2 represents the second attenuation rate, v t-1 Represents the gradient variance of the model parameters at the tth iteration.

[0147] It should be noted that the first-order moment and second-order moment after deviation correction can provide more accurate and stable gradient estimation in the gradient descent process, dynamically adjust the learning rate, reduce the high deviation in the initial stage, enhance numerical stability, and improve the convergence speed.

[0148] S5: According to the position of the lifting ring, the target position of the crane trolley is determined with the lifting trolley being located directly above the lifting ring.

[0149] S6: Get the current position of the crane trolley.

[0150] S7: Based on the difference between the current position of the crane trolley and the target position of the crane trolley, the system control force is calculated through a neural network-based PID controller.

[0151] Optionally, disturbances are considered in the control process. According to the current crane state, the current disturbance is estimated through a disturbance observer. According to the current disturbance, the current system control force is compensated to determine the final control force:

[0152] F * (t) = F (t) + d (t)

[0153] Among them, F *Indicates the ultimate control force, F * (t) represents the final control force at time t, d represents the disturbance, and d(t) represents the disturbance at time t.

[0154] In the present invention, the current system control force is compensated according to the current disturbance, which can accurately compensate for the error caused by the disturbance, so that the control accuracy of the system is significantly improved.

[0155] S8: According to the system control force, control the lifting trolley to move to the position just above the lifting ring.

[0156] S9: Control the lowering of the hook, and obtain the relative position between the hook and the ring in real time through the depth sensor set on the hook. When the distance between the hook and the ring is less than the preset distance, turn on the electromagnet to lock the hook and the ring.

[0157] Among them, those skilled in the art can set the size of the preset distance according to actual conditions, and the present invention does not limit it.

[0158] Optionally, the depth sensor may be a depth sensor suitable for measuring distance, such as a laser radar (LiDAR), an ultrasonic sensor or an infrared sensor.

[0159] It should be noted that the relative position between the hook and the ring can be obtained in real time through the depth sensor set on the hook, and the electromagnet can be turned on at the right time to lock the hook and the ring. This can not only improve the degree of automation of the operation, but also significantly improve work efficiency and safety.

[0160] S10: Control the hook to rise and lift the workpiece.

[0161] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0162] In the present invention, the lifting ring in the workpiece image is identified through the lifting ring recognition model to determine the position of the lifting ring, and then the lifting trolley is controlled to move to just above the lifting ring position, the hook is controlled to be lowered, and the relative position between the hook and the lifting ring is acquired in real time through the depth sensor arranged on the hook. When the distance between the hook and the lifting ring is less than the preset distance, the electromagnet is turned on to lock the hook and the lifting ring, and the hook is controlled to rise to lift the workpiece. Automatic lifting of the workpiece can be realized without human intervention, thereby reducing safety risks and improving production efficiency.

[0163] Reference Manual Attached Figure 2 , showing a structural schematic diagram of a workpiece automatic lifting system based on machine vision provided by the present invention.

[0164] The present invention further provides a workpiece automatic lifting system 20 based on machine vision, which is applied to the above-mentioned workpiece automatic lifting method based on machine vision, comprising:

[0165] Processor 201;

[0166] The memory 202 stores computer-readable instructions, and when the computer-readable instructions are executed by the processor 201, the method for automatically lifting a workpiece based on machine vision as described in the method embodiment is implemented.

[0167] The machine vision-based workpiece automatic lifting system 20 provided in the present invention can execute the above-mentioned machine vision-based workpiece automatic lifting method and achieve the same or similar technical effects. To avoid repetition, the present invention will not go into details.

[0168] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0169] In the present invention, the lifting ring in the workpiece image is identified through the lifting ring recognition model to determine the position of the lifting ring, and then the lifting trolley is controlled to move to just above the lifting ring position, the hook is controlled to be lowered, and the relative position between the hook and the lifting ring is acquired in real time through the depth sensor arranged on the hook. When the distance between the hook and the lifting ring is less than the preset distance, the electromagnet is turned on to lock the hook and the lifting ring, and the hook is controlled to rise to lift the workpiece. Automatic lifting of the workpiece can be realized without human intervention, thereby reducing safety risks and improving production efficiency.

[0170] It should be understood that the processor in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0171] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0172] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0173] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0174] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0175] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0176] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0177] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0178] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0181] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0182] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for automatically lifting a workpiece based on machine vision as described in the method embodiment is implemented.

[0183] A computer-readable storage medium provided by the present invention can implement the steps and effects of the automatic workpiece lifting method based on machine vision in the above method embodiment. To avoid repetition, the present invention will not go into details.

[0184] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0185] In the present invention, the lifting ring in the workpiece image is identified through the lifting ring recognition model to determine the position of the lifting ring, and then the lifting trolley is controlled to move to just above the lifting ring position, the hook is controlled to be lowered, and the relative position between the hook and the lifting ring is acquired in real time through the depth sensor arranged on the hook. When the distance between the hook and the lifting ring is less than the preset distance, the electromagnet is turned on to lock the hook and the lifting ring, and the hook is controlled to rise to lift the workpiece. Automatic lifting of the workpiece can be realized without human intervention, thereby reducing safety risks and improving production efficiency.

[0186] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0187] There are a few points to note:

[0188] (1) The drawings of the embodiments of the present invention only relate to the structures related to the embodiments of the present invention, and other structures may refer to the general design.

[0189] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present invention, the thickness of the layers or regions is exaggerated or reduced, that is, these drawings are not drawn according to the actual scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element or there may be intermediate elements.

[0190] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to obtain new embodiments.

[0191] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for automatically lifting a workpiece based on machine vision, characterized in that: include: S1: Build a ring recognition model based on machine vision; S2: Construct a neural network based PID controller for overhead crane; S3: Acquire workpiece image; S4: identifying the lifting ring in the workpiece image by using the lifting ring recognition model to determine the position of the lifting ring; S5: According to the lifting ring position, the target position of the crane trolley is determined with the lifting trolley being located directly above the lifting ring position as the target; S6: Get the current position of the crane trolley; S7: Calculating the system control force through the neural network-based PID controller according to the difference between the current position of the crane trolley and the target position of the crane trolley; S8: According to the system control force, control the lifting trolley to move to the position just above the lifting ring; S9: Control the hook to be lowered, and obtain the relative position between the hook and the ring in real time through the depth sensor set on the hook. When the distance between the hook and the ring is less than the preset distance, turn on the electromagnet to lock the hook and the ring; S10: Control the hook to rise and lift the workpiece; Wherein, the S4 specifically includes: S401: transforming the workpiece image into RGB color space, HSV color space and Lab color space; S402: Acquire a color histogram of the workpiece image in each color channel; S403: extracting a plurality of color features in each color space from the color histogram, the color features including mean, variance, standard deviation, skewness, kurtosis, entropy, energy and mode; S404: Combining multiple color features in each color space into a 1×8 feature vector to obtain an RGB feature vector, an HSV feature vector, and a Lab feature vector; S405: performing feature fusion on the RGB feature vector, the HSV feature vector and the Lab feature vector to obtain a fused feature vector; S406: inputting the fused feature vector of the workpiece image into a U-Net network, identifying the lifting ring in the workpiece image, and determining the position of the lifting ring; Wherein, the S405 is specifically: According to the following formula, feature fusion is performed: ; in, X represents the fused feature vector, x RGB represents the RGB feature vector, λ RGB represents the fusion coefficient of RGB feature vector, x HSV represents the HSV feature vector, λ HSV represents the fusion coefficient of HSV feature vector, x Lab represents the Lab eigenvector, λ Lab Represents the fusion coefficient of Lab feature vector; The training method of the lifting ring recognition model specifically includes: Constructing a loss function of the hanging ring recognition model; With the goal of minimizing the loss function of the lifting ring recognition model, the model parameters of the lifting ring recognition model are optimized by the gradient descent method; Among them, the loss function of the ring recognition model is specifically: ; in, L 1 represents the loss function of the SDD network, L CIOU represents the bounding box loss function, μ 1 represents the weight of the bounding box loss function, L cls represents the cross entropy loss function, μ 2 represents the weight of the cross entropy loss function, L conf represents the confidence loss function, μ 3 represents the weight of the confidence loss function, θ represents the model parameters, represents the modulo operation on the model parameters, μ 4 represents the weight of the regularization term.

2. The method for automatically lifting a workpiece based on machine vision according to claim 1, characterized in that: The S406 specifically includes: In the convolution layer of the U-Net network, a convolution operation is performed on the fused feature vector to extract features of the fused feature vector; In the pooling layer of the U-Net network, a convolution operation is performed on the extracted features to perform dimensionality reduction processing; In the deconvolution layer of the U-Net network, a deconvolution operation is performed on the extracted features; In the normalization layer of the U-Net network, the output of each layer is normalized; In the classification layer of the U-Net network, the probability that each pixel in the workpiece image is a hanging ring is determined by a Softmax activation function; In the output layer of the U-Net network, pixels with a probability greater than a preset probability are grouped into a ring area, and the ring position is output.

3. The method for automatically lifting a workpiece based on machine vision according to claim 1, characterized in that: The neural network-based PID controller includes an input layer, a hidden layer, and an output layer; The input layer is used to input real-time displacement deviation; The hidden layer includes proportional neurons, integral neurons and differential neurons, which are used to control the proportional gain coefficient, the integral gain coefficient and the differential gain coefficient respectively; The hidden layer is used to determine the real-time proportional gain coefficient, integral gain coefficient and differential gain coefficient during the crane control process; The output layer is used to output system control force.

4. The method for automatically lifting a workpiece based on machine vision according to claim 3, characterized in that: The displacement deviation is specifically: ; in, e represents the displacement deviation, e t express t The displacement deviation at the moment, x Indicates the displacement of the crane trolley, x t express t The displacement of time, x g Indicates the target displacement.

5. The method for automatically lifting a workpiece based on machine vision according to claim 3, characterized in that: The output of the proportional neuron is specifically: ; in, f 1( t ) represents the ratio neuron t Output at the moment, e ( t )express t The displacement deviation at the moment, K p ( t )express t The proportional gain coefficient at the moment; The output of the integrating neuron is specifically: ; in, f 2( t ) represents the integrating neuron t Output at the moment, K i ( t )express t The integral gain coefficient at time , f 2( t -1) represents an integrating neuron t -1 time output; The output of the differential neuron is specifically: ; in, f 3( t ) represents a differentiable neuron t Output at the moment, K d ( t )express t The differential gain coefficient at time , K d ( t -1) indicates t The differential gain coefficient at -1 time, e ( t -1) indicates t -1 displacement deviation at moment; The system control force output by the output layer is specifically: ; in, F ( t )express t System control power output at all times, w 1 represents the weight coefficient between the proportional neuron and the output layer, w 2 represents the weight coefficient between the integral neuron and the output layer, w 3 represents the weight coefficient between the differential neuron and the output layer.

6. A workpiece automatic lifting system based on machine vision, characterized in that: include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method for automatically lifting a workpiece based on machine vision as described in any one of claims 1 to 4 is implemented.

Citation Information

Patent Citations

  • System and method of identification, inspection and training for material lifting products

    CA2581455A1

  • Double-mobile-robot synchronous control method and system

    CN112180918A

  • Steel ladle position measuring method based on machine vision

    CN113096186A

  • Control method and device of lifting hook equipment, computer equipment and storage medium

    CN117383437A

  • Artificial intelligence-based screw bolt lifting ring damage assessment method and device

    CN118070682A