Workpiece chasing shearing optimization control method and system based on machine vision

Abstract

The invention relates to the technical field of follow-up shearing control, in particular to a workpiece follow-up shearing optimization control method and system based on machine vision, and the method comprises the following steps: synchronously transmitting a workpiece image collected by an industrial camera and a workpiece physical state fed back by an encoder in a workpiece movement and follow-up shearing mechanical structure to a digital twin platform; a visual processing model on the platform generates multi-dimensional state features of the workpiece according to the workpiece image and the physical state of the workpiece; and a shearing control instruction is generated by a workpiece dynamical model and a supplementary prediction model on the digital twin platform according to the multi-dimensional state characteristics of the workpiece, and is synchronously transmitted to a shearing instruction controller. According to the method, the digital twin platform is constructed, data interaction virtual-real synchronization is carried out with a real workpiece machining environment, and control parameter optimization of the follow-up shearing operation is carried out on the digital twin platform according to multi-dimensional state characteristics, so that simulation and testing can be carried out for many times in a virtual environment, and the trial and error cost in a workpiece machining physical environment is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of shearing control, and in particular to a workpiece shearing optimization control method and system based on machine vision. Background Art

[0002] Tracking shearing is a highly efficient metal cutting method that enables precise machining of sheet metal. Therefore, optimizing the accuracy of tracking shearing control helps ensure optimal sheet metal machining. Currently, most tracking shearing controls rely on pre-set, standardized control instructions to control the shearing equipment. However, in actual machining environments, workpiece motion and edge characteristics can hinder optimal cutting performance based on standard control instructions. Consequently, motion errors can occur in the cutting result, requiring compensation.

[0003] Therefore, the standardized shearing control instructions in the prior art lack motion compensation and have poor shearing control effects. Summary of the Invention

[0004] The purpose of the present invention is to provide a workpiece tracking and shearing optimization control method and system based on machine vision to solve the technical problems in the prior art of lack of motion compensation and poor tracking and shearing control effect.

[0005] In order to solve the above technical problems, the present invention specifically provides the following technical solutions: A workpiece tracking and shearing optimization control method based on machine vision includes the following steps: The workpiece motion and shearing mechanical structure, industrial camera, and shearing instruction controller are three-dimensionally modeled, and a visual processing model for extracting the workpiece's multi-dimensional state characteristics is integrated with a workpiece dynamics model and a supplementary prediction model for generating workpiece shearing control instructions. This results in a digital twin platform for controlling workpiece shearing through machine vision. The workpiece image captured by the industrial camera and the workpiece physical state feedback from the encoder in the workpiece motion and shearing mechanical structure are synchronously transmitted to the digital twin platform. The visual processing model on the digital twin platform generates the workpiece's multi-dimensional state characteristics based on the workpiece image and physical state. The workpiece dynamics model and the supplementary prediction model on the digital twin platform generate shear control instructions based on the multi-dimensional state characteristics of the workpiece and synchronously transmit them to the shear instruction controller; The shearing instruction controller controls the movement of the workpiece and the shearing mechanical structure to perform shearing on the workpiece according to the shearing control instruction.

[0006] As a preferred solution of the present invention, the method for constructing the visual processing model includes: On the digital twin platform, Blender is used to simulate the workpiece movement and cutting process, and render and generate workpiece images with feature annotations; Using a convolutional neural network to perform a convolution operation on the workpiece image to obtain local features of the workpiece image; Using a Transformer encoder to perform global modeling on the local features of the workpiece image to obtain a global feature of the workpiece image; The local features of the workpiece image and the global features of the workpiece image are spliced ​​and fused to obtain the fusion features of the workpiece image; The mean square error between the fusion features of the workpiece image and the annotated features of the workpiece image is used as a loss function to train the convolutional neural network and the Transformer structure to obtain the visual processing model.

[0007] As a preferred solution of the present invention, the method for constructing the multi-dimensional state characteristics of the workpiece includes: The workpiece image fusion features output by the visual processing model are combined with the physical state of the workpiece to obtain the multi-dimensional state features of the workpiece.

[0008] As a preferred solution of the present invention, the method for constructing the workpiece dynamics model and the supplementary prediction model includes: The workpiece state space is modeled by workpiece dynamics to obtain a workpiece state space model that describes the change of workpiece state with time and control instruction input. , where 、 The multi-dimensional state characteristics of the workpiece at the t+1th and tth moments, is the control instruction at the tth moment, is the state transition function, is the system noise; Establishing the optimization goal of generating control instructions in the workpiece state space model , where is the shearing error term, is the actual output of the shearing at the t-th moment, is the expected output of the shearing at the tth moment, Q is the weight matrix of the shearing error term, is the control quantity constraint term, R is the weight matrix of the control quantity constraint term, To control the stationary term, , is the control instruction at the t-1th moment, L is the weight matrix of the control stability term, N is the prediction step size, and min is the minimization operator; The control instructions of N predicted steps obtained according to the optimization target are trained in time series through the LSTM network to obtain a prediction model for the control instructions after N predicted steps. , where is the control instruction of N prediction steps obtained based on the optimization target prediction, It is the control instruction of m prediction steps output by the prediction model after N prediction steps; The prediction model As a complementary prediction model to the workpiece dynamics model.

[0009] As a preferred solution of the present invention, the physical state of the workpiece includes the position and movement speed of the workpiece.

[0010] As a preferred solution of the present invention, the control instructions include speed instructions, torque instructions and phase adjustment instructions of the servo motor in the workpiece movement and shearing mechanical structure.

[0011] As a preferred solution of the present invention, the actual output of the tracking shearing and the expected output of the tracking shearing include the position of the cutting point of the workpiece and the synchronization speed between the tracking shearing tool and the workpiece.

[0012] As a preferred embodiment of the present invention, the present invention provides a workpiece tracking shearing optimization control system based on machine vision, which is applied to a workpiece tracking shearing optimization control method based on machine vision. The system includes: The digital twin platform includes a 3D model of the workpiece motion and shearing mechanical structure, industrial camera, and shearing instruction controller. It also integrates a visual processing model for extracting multi-dimensional state features of the workpiece, a workpiece dynamics model for generating shearing control instructions, and a supplementary prediction model for controlling workpiece shearing through machine vision. Industrial cameras, used to capture images of workpieces; The three-dimensional model of the industrial camera and the tracking shearing instruction controller and the workpiece movement have a data interaction channel with the industrial camera and the tracking shearing instruction controller for transmitting workpiece images and tracking shearing control instructions.

[0013] As a preferred solution of the present invention, the visual processing model is composed of a convolutional neural network and a Transformer structure, which is used to generate multi-dimensional state features of the workpiece based on the workpiece image captured by the industrial camera and the workpiece movement and physical state feedback of the encoder in the shearing mechanical structure.

[0014] As a preferred solution of the present invention, the workpiece dynamics model is used to generate control instructions in the workpiece state space model according to the optimization target, and the supplementary prediction model is used to perform supplementary prediction on the workpiece dynamics model to obtain control instructions after N prediction steps.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention constructs a digital twin platform to conduct virtual-real data exchange and synchronization with the actual workpiece processing environment, thereby extracting the multi-dimensional state characteristics of the workpiece on the digital twin platform and optimizing the control parameters of the shearing operation based on the multi-dimensional state characteristics. This helps to perform multiple simulations and tests in a virtual environment, reduce the trial and error costs in the physical environment of workpiece processing, lower the risks, and ensure the stability of workpiece processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other implementation drawings based on the provided drawings without inventive effort.

[0017] Figure 1 A flow chart of a workpiece tracking and shearing optimization control method based on machine vision provided by an embodiment of the present invention; Figure 2 This is a block diagram of a workpiece shearing optimization control system based on machine vision provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0019] like Figure 1 As shown, the present invention provides a workpiece tracking and shearing optimization control method based on machine vision, comprising the following steps: The workpiece motion and shearing mechanical structure, industrial camera, and shearing instruction controller are three-dimensionally modeled, and a visual processing model for extracting the workpiece's multi-dimensional state characteristics is integrated with a workpiece dynamics model and a supplementary prediction model for generating workpiece shearing control instructions. This results in a digital twin platform for controlling workpiece shearing through machine vision. The workpiece image captured by the industrial camera and the workpiece physical state feedback from the encoder in the workpiece motion and shearing mechanical structure are synchronously transmitted to the digital twin platform. The visual processing model on the digital twin platform generates the workpiece's multi-dimensional state characteristics based on the workpiece image and physical state. The workpiece dynamics model and supplementary prediction model on the digital twin platform generate shear control instructions based on the multi-dimensional state characteristics of the workpiece and synchronously transmit them to the shear instruction controller; The shearing instruction controller controls the movement of the workpiece and the shearing mechanical structure to shear the workpiece according to the shearing control instruction.

[0020] The present invention uses digital twin technology to convert the shearing control optimization from the physical environment of the workpiece production line to the virtual environment, synchronously transmits the data in the physical environment of the production line to the virtual environment, and generates instructions for optimization based on the model laid out in the virtual environment. In the virtual environment, multiple simulations and tests can be performed based on the generated control instructions, reducing the trial and error costs in the physical environment of workpiece processing and improving safety and production stability.

[0021] The present invention first uses machine vision (industrial camera) to acquire workpiece images, transmits them to the virtual environment of the digital twin platform for feature extraction, and obtains high-dimensional features of the workpiece from the workpiece image, such as shape, texture, edge key points, etc., and combines the physical state of the workpiece (position, movement speed, etc.) fed back by the encoder to form multi-dimensional state characteristics of the workpiece, which can be used to optimize and solve the shear control instructions of the subsequent workpiece dynamics model.

[0022] After obtaining the multi-dimensional state characteristics of the workpiece, the present invention inputs them into the workpiece dynamics model, and can predict the subsequent N moments in the virtual environment of the digital twin platform. Since the prediction step size of the workpiece dynamics model is large, achieving long-term prediction effects will also increase the computational burden. When the prediction step size is short, although the prediction effect burden is reduced, it cannot achieve long-term prediction effects and has poor adaptability to sudden disturbances. To this end, the present invention trains a supplementary prediction model to predict the shearing control instructions for m moments subsequent to N moments after the workpiece dynamics model prediction is completed, so that the workpiece dynamics model can use a short prediction step size to obtain long-term prediction effects when the prediction step size is large, while reducing the computational burden of long steps and increasing adaptability to sudden disturbances.

[0023] When constructing the workpiece state characteristics, the present invention first utilizes visual features and then combines the physical state to form multidimensional state characteristics, which can provide more feature information for use in the workpiece dynamics model to obtain a more accurate prediction effect in multidimensional information.

[0024] This paper uses a combination of convolutional neural networks and Transformer encoders to construct a visual feature extraction model. The convolutional neural network has locality and extracts local spatial features (such as edges and textures), while the Transformer encoder has globality and uses the self-attention mechanism to capture long-range dependencies (such as the overall structure of an object). Combining global and local feature extraction and fusion improves feature extraction and achieves higher accuracy, as follows: The methods for building a visual processing model include: On the digital twin platform, Blender is used to simulate the workpiece movement and cutting process, and render and generate workpiece images with feature annotations; Use convolutional neural network to perform convolution operation on the workpiece image to obtain the local features of the workpiece image; The Transformer encoder is used to globally model the local features of the workpiece image to obtain the global features of the workpiece image; The local features of the workpiece image and the global features of the workpiece image are spliced ​​and fused to obtain the fusion features of the workpiece image; The mean square error between the fusion features of the workpiece image and the annotated features of the workpiece image is used as the loss function to train the convolutional neural network and Transformer structure to obtain the visual processing model.

[0025] When constructing workpiece state features, the present invention first utilizes visual features and then combines them with physical states to form multidimensional state features. This can provide more feature information for use in the workpiece dynamics model to obtain a more accurate prediction effect from the multidimensional information, as follows: The method for constructing the multi-dimensional state characteristics of the workpiece includes: The workpiece image fusion features output by the visual processing model are combined with the physical state of the workpiece to obtain the multi-dimensional state features of the workpiece.

[0026] After acquiring the multi-dimensional state characteristics of the workpiece, the present invention inputs them into the workpiece dynamics model, and can predict the subsequent N moments in the virtual environment of the digital twin platform. Since the prediction step size of the workpiece dynamics model is large, achieving long-term prediction effects will also increase the computational burden. When the prediction step size is short, the prediction effect burden is reduced, but long-term prediction effects cannot be achieved and the adaptability to sudden disturbances is poor. To this end, the present invention trains a supplementary prediction model to predict the shear control instructions for the subsequent m moments after the workpiece dynamics model prediction is completed. This achieves the goal that the workpiece dynamics model uses a short prediction step size to obtain the long-term prediction effect when the prediction step size is large, while reducing the computational burden of long step sizes and increasing the adaptability to sudden disturbances. The details are as follows: The method for constructing the workpiece dynamics model and the supplementary prediction model includes: The workpiece state space is modeled by workpiece dynamics to obtain a workpiece state space model that describes the change of workpiece state with time and control instruction input. , where 、 The multi-dimensional state characteristics of the workpiece at the t+1th and tth moments, is the control instruction at the tth moment, is the state transition function, is the system noise; State transfer function, describing the dynamic characteristics of the system, such as linear models: , 𝐴, 𝐵 are system matrices. Nonlinear model: The dynamic equations need to be discretized (e.g., considering friction and inertia); Establishing the optimization goal of generating control instructions in the workpiece state space model , where is the shearing error term, is the actual output of the shearing at the t-th moment, is the expected output of the shearing at the tth moment, Q is the weight matrix of the shearing error term, is the control quantity constraint term, R is the weight matrix of the control quantity constraint term, To control the stationary term, , is the control instruction at the t-1th moment, L is the weight matrix of the control stability term, N is the prediction step size, and min is the minimization operator; The optimization objectives include three aspects. The first is the shearing error term, which measures the difference between the actual shearing result and the expected result. Q is a positive definite matrix used to adjust the penalty for the shearing error. The second is the control constraint term. R is used to limit the control variable to avoid motor overload (for example, limiting the maximum torque 𝑇𝑚𝑎𝑥) and prevent high-frequency oscillation (suppressing sudden changes in control commands). The third is the control smoothness term. L is used to smooth control commands, reduce mechanical shock (for example, limiting the acceleration of servo motors), and avoid frequent switching of control commands (improving system stability). Combining these three optimization objectives yields the optimal prediction results.

[0027] Optimization goal N prediction step lengths of control instructions can be predicted. When N is large, the prediction is longer-term, but the computational complexity is high. When N is small, the real-time performance is good, but the adaptability to sudden disturbances is poor. Therefore, the present invention uses the control instructions of N prediction step lengths predicted according to the optimization target to train the LSTM network to obtain a supplementary prediction model. The control instructions at subsequent moments can be predicted based on the temporal regularity of the control instructions of N prediction step lengths predicted according to the optimization target. When the N value is selected to be small, the real-time prediction performance is retained and the computational burden is small. At the same time, the long-term prediction performance is compensated and the adaptability to sudden disturbances is improved according to the supplementary prediction model, as follows: The control instructions of N predicted steps obtained according to the optimization target are trained in time series through the LSTM network to obtain a prediction model for the control instructions after N predicted steps. , where is the control instruction of N prediction steps obtained based on the optimization target prediction, It is the control instruction of m prediction steps output by the prediction model after N prediction steps; The prediction model As a complementary prediction model to the workpiece dynamics model.

[0028] The present invention can obtain a more comprehensive and better-performing control instruction prediction effect through the mutual cooperation between the workpiece dynamics model and the supplementary prediction model, and can obtain a higher shear cutting effect after feedback to the workpiece physical environment.

[0029] The physical state of the workpiece includes the position and movement speed of the workpiece.

[0030] The control instructions include speed instructions, torque instructions and phase adjustment instructions for the servo motor in the workpiece movement and shearing mechanical structure.

[0031] The actual output and expected output of the tracking shear include the cutting point position of the workpiece and the synchronization speed between the tracking shear tool and the workpiece.

[0032] like Figure 2 As shown, the present invention provides a workpiece tracking shearing optimization control system based on machine vision, which is applied to a workpiece tracking shearing optimization control method based on machine vision. The system includes: The digital twin platform includes a 3D model (using tools such as MATLAB Simulink and ANSYS TwinBuilder) of the workpiece motion and shearing mechanical structure (servo motor, encoder), industrial camera, and shearing command controller (PLC / industrial computer). It also integrates a visual processing model for extracting multi-dimensional state features of the workpiece, a workpiece dynamics model for generating shearing control commands, and a supplementary prediction model for controlling shearing through machine vision. Industrial cameras, used to capture images of workpieces; There is a data interaction channel (EtherCAT / PROFINET protocol access device data) between the 3D modeling model of the industrial camera and the tracking shear command controller and the workpiece movement, which is used to transmit workpiece images and tracking shear control instructions.

[0033] The visual processing model consists of a convolutional neural network and a Transformer structure, and is used to generate multi-dimensional state features of the workpiece based on the workpiece images captured by the industrial camera and the workpiece motion and physical state feedback from the encoder in the tracking and shearing mechanical structure.

[0034] Encoder signal synchronization: Subscribe to actual encoder pulses via OPC UA to drive the virtual encoder model. Image data streaming: Use the RTSP protocol to transmit the actual camera image for comparison and calibration with the virtual rendered image.

[0035] The workpiece dynamics model is used to generate control instructions in the workpiece state space model according to the optimization target, and the supplementary prediction model is used to perform supplementary prediction on the workpiece dynamics model to obtain control instructions after N prediction steps.

[0036] The present invention constructs a digital twin platform to conduct virtual-real data exchange and synchronization with the actual workpiece processing environment, thereby extracting the multi-dimensional state characteristics of the workpiece on the digital twin platform and optimizing the control parameters of the shearing operation based on the multi-dimensional state characteristics. This helps to perform multiple simulations and tests in a virtual environment, reduce the trial and error costs in the physical environment of workpiece processing, lower the risks, and ensure the stability of workpiece processing.

[0037] The above embodiments are merely exemplary embodiments of the present application and are not intended to limit the scope of the present application. The scope of protection of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and scope of protection of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the scope of protection of the present application.

Claims

1. A workpiece tracking and shearing optimization control method based on machine vision, characterized in that: The following steps are involved: The workpiece motion and shearing mechanical structure, industrial camera, and shearing instruction controller are 3D modeled. A visual processing model for extracting multi-dimensional state features of the workpiece, a workpiece dynamics model for generating shearing control instructions, and a supplementary prediction model are integrated to create a digital twin platform for controlling workpiece shearing through machine vision. The workpiece image captured by the industrial camera and the workpiece physical state feedback from the encoder in the workpiece motion and tracking shearing mechanism are synchronously transmitted to the digital twin platform. The visual processing model on the digital twin platform generates multi-dimensional state features of the workpiece based on the workpiece image and workpiece physical state. The workpiece dynamics model and the supplementary prediction model on the digital twin platform generate shear control instructions based on the multi-dimensional state characteristics of the workpiece and synchronously transmit them to the shear instruction controller; The shearing instruction controller controls the movement of the workpiece and the shearing mechanical structure to perform shearing on the workpiece according to the shearing control instruction.

2. The workpiece tracking shearing optimization control method based on machine vision according to claim 1 is characterized in that: The method for constructing the visual processing model includes: On the digital twin platform, Blender is used to simulate the workpiece movement and cutting process, and render and generate workpiece images with feature annotations; Using a convolutional neural network to perform a convolution operation on the workpiece image to obtain local features of the workpiece image; Using a Transformer encoder to perform global modeling on the local features of the workpiece image to obtain a global feature of the workpiece image; The local features of the workpiece image and the global features of the workpiece image are spliced ​​and fused to obtain the fusion features of the workpiece image; The mean square error between the fusion features of the workpiece image and the annotated features of the workpiece image is used as a loss function to train the convolutional neural network and the Transformer structure to obtain the visual processing model.

3. The workpiece tracking shearing optimization control method based on machine vision according to claim 2 is characterized in that: The method for constructing the multi-dimensional state feature of the workpiece includes: The workpiece image fusion features output by the visual processing model are combined with the physical state of the workpiece to obtain the multi-dimensional state features of the workpiece.

4. The workpiece tracking shearing optimization control method based on machine vision according to claim 3 is characterized in that: The method for constructing the workpiece dynamics model and the supplementary prediction model includes: The workpiece state space is modeled by workpiece dynamics to obtain a workpiece state space model that describes the change of workpiece state with time and control instruction input. , where 、 The multi-dimensional state characteristics of the workpiece at the t+1th and tth moments, is the control instruction at the tth moment, is the state transition function, is the system noise; Establishing the optimization goal of generating control instructions in the workpiece state space model , where is the shearing error term, is the actual output of the shearing at the t-th moment, is the expected output of the shearing at the tth moment, Q is the weight matrix of the shearing error term, is the control quantity constraint term, R is the weight matrix of the control quantity constraint term, To control the stationary term, , is the control instruction at the t-1th moment, L is the weight matrix of the control stability term, N is the prediction step size, and min is the minimization operator; The control instructions of N predicted steps obtained according to the optimization target are trained in time series through the LSTM network to obtain a prediction model for the control instructions after N predicted steps. , where is the control instruction of N prediction steps obtained based on the optimization target prediction, It is the control instruction of m prediction steps output by the prediction model after N prediction steps; The prediction model As a complementary prediction model to the workpiece dynamics model.

5. The workpiece tracking shearing optimization control method based on machine vision according to claim 1 is characterized in that: The physical state of the workpiece includes the position and movement speed of the workpiece.

6. The workpiece tracking shearing optimization control method based on machine vision according to claim 1 is characterized in that: The control instructions include speed instructions, torque instructions and phase adjustment instructions for the servo motor in the workpiece movement and the shearing mechanical structure.

7. The workpiece tracking shearing optimization control method based on machine vision according to claim 4 is characterized in that: The actual output of the tracking shearing and the expected output of the tracking shearing include the position of the cutting point of the workpiece and the synchronization speed between the tracking shearing tool and the workpiece.

8. A workpiece shearing optimization control system based on machine vision, characterized in that: According to a machine vision-based workpiece tracking and shearing optimization control method according to any one of claims 1 to 7, the system comprises: The digital twin platform includes a 3D model of the workpiece motion and shearing mechanical structure, industrial camera, and shearing instruction controller. It also integrates a visual processing model for extracting multi-dimensional state features of the workpiece, a workpiece dynamics model for generating shearing control instructions, and a supplementary prediction model for controlling workpiece shearing through machine vision. Industrial cameras, used to capture images of workpieces; The three-dimensional model of the industrial camera and the tracking shearing instruction controller and the workpiece movement have a data interaction channel with the industrial camera and the tracking shearing instruction controller for transmitting workpiece images and tracking shearing control instructions.

9. The machine vision-based workpiece tracking and shearing optimization control system according to claim 8, characterized in that: The visual processing model is composed of a convolutional neural network and a Transformer structure, and is used to generate multi-dimensional state features of the workpiece based on the workpiece image captured by an industrial camera and the workpiece physical state feedback from the encoder in the workpiece motion and tracking shearing mechanical structure.

10. The machine vision-based workpiece tracking shearing optimization control system according to claim 8, characterized in that: The workpiece dynamics model is used to generate control instructions in the workpiece state space model according to the optimization target, and the supplementary prediction model is used to perform supplementary prediction on the workpiece dynamics model to obtain control instructions after N prediction steps.

Images