A welding robot vision control system based on deep learning

Abstract

The application provides a welding robot visual control system based on deep learning, comprising: a welding seam track determination module, which is used for building a welding seam area identification model based on a deep learning algorithm and determining a welding seam track; a movement parameter determination module, which is used for generating track movement parameters based on original image features of the welding seam track; a welding parameter determination module, which is used for determining welding parameters based on a groove feature of the welding seam track; a control parameter determination module, which is used for building a welding control parameter determination model based on a deep learning algorithm and determining target welding control parameters; and a welding control optimization module, which is used for continuously learning and optimizing welding control of the welding robot based on time sequence information and space sequence information of a welding position of a latest obtained welding torch head. The deep learning algorithm is combined with visual detection to quickly generate welding control parameters with high precision, and continuous optimization and closed-loop control in the welding process are realized, so that the welding effect is better.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, and particularly relates to a welding robot visual control system based on deep learning. BACKGROUND

[0002] At present, by applying visual detection and robot control technology in the field of industrial welding technology, a welding robot controllable through programming is developed. The existing welding robot controls the welding robot through pre-programming, and the welding robot can weld according to the welding trajectory imported in the pre-programmed control program. In the process of welding effect acceptance or identification of the welding starting point, the end point and the inflection point position, the visual detection technology improves the automation of the welding process and saves the labor cost.

[0003] However, the control programming process of the existing welding robot needs to obtain the weld seam trajectory through manual identification or other visual detection weld seam identification process, and the welding control parameters are manually decided, so that a complete control program can be further generated. The accuracy of the manual collection and decision process is difficult to guarantee, which leads to the control precision of the control program of the welding robot being unable to be guaranteed, and there is no other optimization means in the welding process, which leads to the final automatic welding effect of the machine being poor.

[0004] Therefore, the present application provides a welding robot visual control system based on deep learning. SUMMARY

[0005] The present application provides a welding robot visual control system based on deep learning, which combines deep learning algorithm and visual detection to realize automatic identification of the weld seam trajectory, and based on the related parameters of the obtained weld seam groove characteristics and weld seam trajectory, generates high-precision welding control parameters quickly without manual decision, improves the control precision of the control program of the welding robot, and through continuous learning of the actual welding position in the welding process, realizes continuous optimization and closed-loop control of the welding robot control, so that the final welding effect is better, thereby overcoming the problems in the background technology.

[0006] The present application provides a welding robot visual control system based on deep learning, which combines deep learning algorithm and visual detection to realize automatic identification of the weld seam trajectory, and based on the related parameters of the obtained weld seam groove characteristics and weld seam trajectory, generates high-precision welding control parameters quickly without manual decision, improves the control precision of the control program of the welding robot, and through continuous learning of the actual welding position in the welding process, realizes continuous optimization and closed-loop control of the welding robot control, so that the final welding effect is better, thereby overcoming the problems in the background technology.

[0007] The weld seam trajectory determination module is used for building a weld seam area identification model based on a deep learning algorithm, and determining the weld seam trajectory based on the weld seam area identification model and the selected sorting instruction input by the user;

[0008] The movement parameter determination module is used for determining a three-dimensional coordinate representation of the weld seam trajectory based on the original image features of all selected weld seams in the weld seam trajectory in the workpiece image, and generating trajectory movement parameters based on the three-dimensional coordinate representation of the weld seam trajectory.

[0009] a welding parameter determination module, configured to identify a groove feature of the weld seam trajectory in the weld seam marked image, and determine the welding parameter based on the groove feature and preset experience parameters;

[0010] a control parameter determination module, configured to build a welding control parameter determination model based on a deep learning algorithm, input the welding parameter and the trajectory movement parameter into the welding control parameter determination model, and obtain a target welding control parameter;

[0011] a welding control optimization module, configured to control the welding robot to weld based on the target welding control parameter, and constantly learn based on time sequence information and spatial sequence information of the welding position of the latest obtained welding torch head, optimize the welding control of the welding robot, and obtain a complete welding result.

[0012] Preferably, the weld seam trajectory determination module comprises:

[0013] a first training building sub-module, configured to obtain a weld seam region identification model by deep learning training on a large number of workpiece images in which different weld seam types are marked;

[0014] a weld seam identification and marking sub-module, configured to identify candidate weld seam regions in a two-dimensional image of a workpiece to be welded in real time based on the weld seam region identification model, and highlight mark all the candidate weld seam regions in the two-dimensional image to obtain a weld seam marked image;

[0015] a user selection and sorting sub-module, configured to select and sort all the candidate weld seam regions in the weld seam marked image based on a user selection and sorting instruction to obtain a weld seam trajectory.

[0016] Preferably, the movement parameter determination module comprises:

[0017] a three-dimensional coordinate determination sub-module, configured to determine a three-dimensional coordinate representation of the weld seam trajectory based on original image features of all the selected weld seams in the weld seam trajectory in the workpiece image;

[0018] a movement parameter determination sub-module, configured to determine trajectory parameters of the weld seam trajectory as the trajectory movement parameter based on the three-dimensional coordinate representation of the weld seam trajectory.

[0019] Preferably, the three-dimensional coordinate determination sub-module comprises:

[0020] a region contour identification unit, configured to identify all contour lines in the workpiece image based on a contour identification algorithm, identify straight line contours in all the contour lines, and determine a single surface region in the workpiece image based on all the contour lines;

[0021] The reference plane screening unit is configured to screen a plurality of reference planes from all single-face regions based on all straight line contours in the workpiece image.

[0022] The three-dimensional coordinate determination unit is configured to determine a three-dimensional distance between each contour corner point on the reference plane and a preset three-dimensional origin point based on the distance measuring sensor, and determine a relative azimuth angle of each contour corner point in a preset three-dimensional coordinate system, and determine a three-dimensional coordinate representation of each contour corner point based on the three-dimensional distance and the relative azimuth angle of each contour corner point.

[0023] The conversion relationship determination unit is configured to determine a coordinate conversion relationship based on the three-dimensional coordinate representation and the two-dimensional coordinate representation of the contour corner points of all the reference planes and the pixel parameters of all the reference planes.

[0024] The weld three-dimensional determination unit is configured to determine a two-dimensional coordinate representation of the selected weld based on the original image features of all the selected welds in the workpiece image, and determine a three-dimensional coordinate representation of the weld trajectory based on the coordinate conversion relationship and the two-dimensional coordinate representation of the selected weld.

[0025] Preferably, the reference plane screening unit comprises:

[0026] The reference surface determination subunit is configured to take a single-face region where two non-parallel straight line contours belonging to the same single face as a reference surface.

[0027] The pixel sequence determination subunit is configured to determine a gradual change feature judgment direction based on the extension directions of the two non-parallel straight line contours, and determine a plurality of pixel point sequences in the reference surface where the extension directions of adjacent pixel points are consistent with the gradual change feature judgment direction.

[0028] The gradual change feature judgment unit is configured to judge whether the image features of all the pixel point sequences belonging to the same gradual change feature judgment direction are gradually smooth and consistent in the gradual change direction, and if so, determine that the reference surface is a reference plane, otherwise, determine that the reference surface is not a reference plane.

[0029] Preferably, the welding parameter determination module comprises:

[0030] The groove shape determination sub-module is configured to identify a groove cross-sectional region of the weld trajectory in the weld mark image.

[0031] The coordinate three-dimensional conversion sub-module is configured to determine a two-dimensional coordinate representation of the weld trajectory and a two-dimensional coordinate of the groove cross-sectional region in the weld mark image, and determine a three-dimensional coordinate representation of the groove cross-section based on the coordinate conversion relationship between the two-dimensional coordinate representation and the three-dimensional coordinate representation of the weld trajectory and the two-dimensional coordinate representation of the groove cross-sectional region.

[0032] The welding parameter determination submodule is configured to determine a three-dimensional coordinate representation of a weld shape based on a three-dimensional coordinate representation of the weld trajectory and a three-dimensional coordinate representation of the groove cross section as a groove feature, and determine the welding parameters based on the groove feature and a preset welding parameter determination mode corresponding to the current welding mode in the preset experience parameters.

[0033] Preferably, the control parameter determination module comprises:

[0034] The second training establishment submodule is configured to train a welding control parameter determination model based on a large number of historical welding control instances containing historical welding parameters, historical trajectory movement parameters, and corresponding historical welding control parameters.

[0035] The control parameter determination submodule is configured to input the welding parameters and the trajectory movement parameters into the welding control parameter determination model to obtain the target welding control parameters.

[0036] Preferably, the welding control optimization module comprises:

[0037] The welding dynamic simulation submodule is configured to dynamically simulate the welding process based on the welding parameters and the trajectory movement parameters to obtain a virtual welding trajectory.

[0038] The welding control monitoring submodule is configured to control the welding robot to perform welding based on the target welding control parameters, and acquire a welding torch head monitoring video of the welding robot in real time.

[0039] The actual information acquisition submodule is configured to determine time sequence information and spatial sequence information of the welding position of the welding torch head based on the welding torch head monitoring video.

[0040] The control parameter optimization submodule is configured to continuously optimize the welding control parameter determination model based on the latest obtained time sequence information and spatial sequence information of the welding position of the welding torch head and the virtual welding trajectory to obtain the latest welding optimization control parameters.

[0041] The welding control optimization submodule is configured to continuously update the target welding control parameters of the welding robot based on the latest obtained welding optimization control parameters until the latest obtained welding position of the welding torch head reaches the end point of the virtual welding trajectory to obtain a complete welding result.

[0042] Preferably, the welding dynamic simulation submodule comprises:

[0043] The welding parameter analysis unit is configured to determine welding cross section formation parameters and welding cross section normal movement parameters of a welding fusion zone based on the welding parameters.

[0044] The trajectory movement parameter analysis unit is configured to determine a trajectory movement direction based on the trajectory movement parameters.

[0045] The welding process dynamic simulation unit is used for simulating a dynamic welding process based on welding section formation parameters, welding section normal movement parameters and trajectory movement directions, and obtaining a virtual welding trajectory.

[0046] Preferably, the actual information acquisition sub-module comprises:

[0047] The space sequence information determination unit is used for determining the welding position of the welding torch head based on the welding torch head monitoring video, determining the actual welding trajectory, dividing the actual welding trajectory according to a preset slice interval, and obtaining the space sequence information of the welding position of the welding torch head.

[0048] The time sequence information determination unit is used for determining the coincidence position point of the actual welding trajectory and the virtual welding trajectory, dividing the actual welding trajectory according to the coincidence position point as a division limit, obtaining a plurality of partial trajectory sequences, generating a corresponding time slice based on the formation time period of each partial trajectory in the partial trajectory sequence, and generating the space sequence information of the welding position of the welding torch head based on the time slices of all the partial trajectories in the partial trajectory sequence.

[0049] The present application has the beneficial effects that: by combining the deep learning algorithm with the visual detection, the automatic identification of the weld trajectory is realized, and based on the acquired groove feature of the weld and the related parameters of the weld trajectory, the welding control parameters with high precision are quickly generated without manual decision, the control precision of the control program of the welding robot is improved, and through the continuous learning of the actual welding position in the welding process, the continuous optimization and closed-loop control of the welding robot control are realized, so that the final welding effect is better.

[0050] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application can be realized and achieved by the structure particularly pointed out in the written description, claims, and drawings.

[0051] The technical solutions of the present application will be further described in detail below with the help of the drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, illustrate the embodiments of the present application, and are used to explain the present application, and do not constitute a limitation on the present application. In the drawings:

[0053] Figure 1 It is a function execution schematic diagram of a welding robot visual control system based on deep learning in the embodiment of the present application;

[0054] Figure 2A device schematic diagram for realizing control of a welding robot visual control system based on deep learning in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of an optimization principle of a welding robot visual control system based on deep learning in an embodiment of the present application in a welding process. DETAILED DESCRIPTION

[0056] The preferred embodiments of the present application are described below in conjunction with the accompanying drawings, and it should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0057] Embodiment 1:

[0058] The present application provides a welding robot visual control system based on deep learning, referring to Figure 1 , to 3, comprising:

[0059] A weld seam trajectory determination module is configured to build a weld seam area recognition model based on a deep learning algorithm (the deep learning algorithm is a process of learning image features of a large number of workpiece images with different weld seam types marked in the weld seam area), use a large-scale deep convolutional neural network model to learn image features of a large number of workpiece images with different weld seam types marked in the weld seam area, and generalize to an actual recognition scene to obtain an image recognition model that can accurately recognize the weld seam area in the workpiece image, such as using a convolutional neural network (CNN) for deep learning to obtain the weld seam area recognition model, and determine a weld seam trajectory based on the weld seam area recognition model and a selected sorting instruction input by a user (the selected sorting instruction is an instruction input by the user for screening a weld seam area that needs to be welded by the welding robot from the selected weld seam area recognized based on the weld seam area recognition model, and determining a welding sequence thereof), wherein the weld seam trajectory is a welding trajectory composed of the weld seam area obtained by sorting the weld seam area selected by the user based on the selected sorting instruction; the weld seam trajectory determination module realizes automatic recognition of the weld seam trajectory by combining the deep learning algorithm with visual detection;

[0060] The mobile parameter determining module is configured to determine a three-dimensional coordinate representation of the weld seam track based on original image features of all selected weld seams in the weld seam track in the workpiece image (the original image features of the selected weld seams are original image features displayed by the weld seam regions in the workpiece image, for example, pixel value, chroma value, contrast value, brightness value and other image attribute parameters of all pixel points in the weld seam regions in the workpiece image), and generate track mobile parameters (parameters including track length, moving direction and shape, for example, the weld seam track is a straight line segment with a length of 30 cm or a circular arc segment with a radius of 5 cm and an arc of 1 rad) based on the three-dimensional coordinate representation of the weld seam track.

[0061] The welding parameter determination module is configured to identify a groove feature of the weld seam track in the weld seam mark image (the groove feature refers to a groove geometric shape and a dimension parameter, the geometric shape is for example a single I shape, a single V shape, a double single V shape (also referred to as a K shape), a V shape, a double V shape (also referred to as an X shape), a J shape, a double J shape, a U shape, a double U shape, a horn single V shape, a horn V shape, a crimping butt joint, and an I shape groove, an outer corner V shape groove, an outer corner single V shape groove, an inner corner single V shape groove, an outer corner U shape groove, an outer corner J shape groove, an inner corner J shape groove, an inner corner single horn V shape groove, a horn V shape groove, and an inner corner single horn outer corner end joint in an angle joint, and the dimension parameter is for example a root gap, a land, a groove angle, a groove face angle, a U shape root radius, and a crimping height and a crimping radius of a crimping groove), determine the welding parameter (for arc welding, the welding parameter includes a welding current, an arc voltage, a welding speed, a welding wire (electrode) diameter, a current polarity, a welding wire extension length, a protective gas flow, and the like; for resistance welding, the welding parameter includes an electrode pressure, an electric conduction time and a secondary voltage, an electrode diameter, and the like) based on the groove feature and a preset experience parameter (the preset experience parameter refers to a preset method for determining the welding parameter corresponding to different groove features, for example, a groove angle size causes a position reached by an arc output by a welding head in a groove to be different, a greater groove angle results in a shallower unfused depth, that is, a greater penetration depth, a smaller groove angle results in a deeper unfused depth, that is, a smaller penetration depth, that is, a greater accumulation of filler metal on a surface, and thus the different groove features need to be considered when the arc output by the welding head is determined to ensure a welding effect).

[0062] The control of the welding robot mainly includes two parts: control of a welding process and control of a moving process, although the two are simultaneously performed in the welding process, the determination principles of the control parameters are completely different, and thus the welding process control and the moving process control are analyzed respectively by the above-mentioned moving parameter determination module and the welding parameter determination module, and the control precision of the welding robot is further ensured.

[0063] The control parameter determination module is configured to build a welding control parameter determination model based on a deep learning algorithm (in this step, the correlation between the historical welding parameters, the historical trajectory movement parameters and the corresponding historical welding control parameters in a large number of historical welding control instances containing the historical welding parameters, the historical trajectory movement parameters and the corresponding historical welding control parameters is learned, and then the correlation between the welding parameters, the historical trajectory movement parameters and the corresponding historical welding control parameters under different conditions is determined, i.e., the correlation is generalized to different welding control instances, and then a model capable of determining the target welding control parameters that meet the welding control target corresponding to the welding parameters and the welding control target corresponding to the trajectory movement parameters is obtained, for example, a welding control parameter determination model is built by deep learning using a fully connected layer neural network), and the welding parameters and the trajectory movement parameters are input into the welding control parameter determination model to obtain the target welding control parameters (the target welding parameters are all control parameters used to control the welding robot to start welding, such as welding torch head movement speed, welding torch head output voltage, etc.);

[0064] The above-mentioned movement parameter determination module, welding parameter determination module and control parameter determination module can quickly generate welding control parameters with high precision without manual decision-making, and improve the control precision of the control program of the welding robot.

[0065] The welding control optimization module is configured to control the welding robot to weld based on the target welding control parameters, and constantly learn and optimize the welding control of the welding robot based on the time sequence information (the time sequence information is time change information of the actual movement trajectory of the welding torch head of the welding robot, for example, the time point when the welding torch head reaches the A position point from the starting point is t seconds) and the space sequence information (the space sequence information is space change information of the actual movement trajectory of the welding torch head of the welding robot, for example, the actual position of the welding torch head changes with time) of the welding position of the welding torch head of the welding robot (the target welding control parameters are constantly optimized by using logistic regression to reduce the error between the actual welding trajectory of the welding head and the preset virtual welding trajectory), until a complete welding result is obtained (the complete welding result is a workpiece or a workpiece image obtained after the welding robot is controlled by the welding robot vision control system based on deep learning in the embodiment to weld the workpiece to be welded).

[0066] The welding control optimization module constantly learns the actual welding position during the welding process, realizes continuous optimization and closed-loop control of the welding robot control, and makes the final welding effect better.

[0067] Embodiment 2:

[0068] On the basis of embodiment 1, the weld seam track determination module comprises:

[0069] The first training building sub-module is configured to train a weld seam area recognition model by deep learning on a large number of workpiece images in which different weld seam types are marked, for example, initialize the weights of a convolutional neural network (CNN), input the workpiece images in which different weld seam types are marked into the convolutional neural network, and obtain the weld seam area marked by the current model in the workpiece image through forward propagation of the convolutional layer, the down-sampling layer and the full connection layer, and then return the coordinate deviation between the weld seam area marked by the current model in the workpiece image and the original weld seam area of the workpiece to the convolutional neural network, and sequentially obtain the errors of the full connection layer, the down-sampling layer and the convolutional layer, update the initial weights based on the errors of the layers, and obtain a new convolutional neural network; and input new workpiece images in which different weld seam types are marked into the new convolutional neural network for continuous training until the coordinate error is less than a preset expected value, and the training is ended, and the weld seam area recognition model is obtained.

[0070] The deep learning algorithm is combined with visual detection to train the weld seam area recognition model which can accurately recognize the weld seam area.

[0071] The weld seam recognition and marking sub-module is configured to recognize the candidate weld seam area in the two-dimensional image of the workpiece to be welded (the two-dimensional image contains all the appearance features of the workpiece to be welded) based on the weld seam area recognition model, and mark all the candidate weld seam areas in the two-dimensional image, and obtain the weld seam marking image (the weld seam marking image contains all the candidate weld seam areas marked in highlight, which is convenient for the user to view and select and sort in the subsequent step).

[0072] The user selection and sorting sub-module is configured to select and sort all the candidate weld seam areas in the weld seam marking image based on the selection and sorting instruction of the user, and obtain the weld seam track.

[0073] The weld seam recognition and marking sub-module and the user selection and sorting sub-module are involved in the simple marking of the user, realize the further screening of the automatically recognized weld seam area, and provide the user with more choices.

[0074] Embodiment 3:

[0075] On the basis of embodiment 1, the mobile parameter determination module comprises:

[0076] a three-dimensional coordinate determination sub-module configured to determine a three-dimensional coordinate representation of the weld seam trajectory based on the original image features of all the selected weld seams in the weld seam trajectory in the workpiece image;

[0077] a movement parameter determination sub-module configured to determine trajectory parameters (including parameters of a trajectory length, a movement direction, and a shape) of the weld seam trajectory as the trajectory movement parameters based on the three-dimensional coordinate representation of the weld seam trajectory;

[0078] The step realizes the result based on visual recognition and three-dimensional result, realizes the separate analysis determination of the movement control parameter of the welding robot, and then can guarantee the precision of the subsequent total control process.

[0079] Embodiment 4:

[0080] On the basis of embodiment 3, the three-dimensional coordinate determination sub-module comprises:

[0081] a region contour recognition unit configured to recognize all contour lines in the workpiece image based on a contour recognition algorithm (for example, a Canny algorithm), recognize straight line contours (that is, straight line segments in the contour lines recognized in the foregoing step) in all the contour lines, and determine single surface regions (that is, a single plane or a single curved surface, the single surface region does not contain contour lines, and can be considered as a single connected region obtained after the workpiece image is divided based on the foregoing contour lines) in the workpiece image based on all the contour lines;

[0082] The contour recognition and region division of the workpiece image provide a basis for subsequent determination of reference points used to determine the coordinate conversion relationship;

[0083] a reference plane screening unit configured to screen a plurality of reference planes (the reference plane is a region in the workpiece image where a plane used for subsequent determination of a coordinate conversion relationship between the workpiece image and an actual workpiece is referenced) from all the single surface regions based on all the straight line contours in the workpiece image;

[0084] The depth value gradient feature of the planar region is more stable than the depth value gradient feature of the curved surface region, and therefore, determination of the depth scaling factor based on the planar region as a reference will result in more accurate restoration of the determined two-dimensional to three-dimensional coordinate conversion relationship;

[0085] A three-dimensional coordinate determination unit is configured to determine a three-dimensional distance (the three-dimensional distance is an actual distance between a position point of the contour corner point in the actual space and the preset three-dimensional origin point in the actual three-dimensional space) of each contour corner point on the reference plane and a relative azimuth angle (the relative azimuth angle is an angle of a pointing vector from the preset three-dimensional origin point to the position point of the contour corner point in the actual three-dimensional space in the preset coordinate system, and the relative azimuth angle includes three angles: an angle α between the pointing vector and a horizontal coordinate axis of the preset coordinate system, an angle β between the pointing vector and a vertical coordinate axis of the preset coordinate system, and an angle γ between the pointing vector and a depth coordinate axis of the preset coordinate system) of each contour corner point based on a three-dimensional distance x and the relative azimuth angle of each contour corner point, and determine a three-dimensional coordinate representation (the three-dimensional coordinate representation of the contour corner point is (x*cosα, x*cosβ, x*cosγ)) of each contour corner point;

[0086] The three-dimensional coordinate representation of the position point in the actual three-dimensional space is accurately determined by measuring the distance and the azimuth angle.

[0087] A conversion relationship determination unit is configured to determine a coordinate conversion relationship (specific steps include: determining a vertical coordinate value of each contour corner point based on the three-dimensional coordinate representation of the contour corner point, taking a ratio of the vertical coordinate value of the contour corner point and a depth value in the two-dimensional image as a depth scaling factor of the corresponding contour corner point, taking an average of the final depth scaling factors of all contour corner points in the reference plane as a final depth scaling factor of the corresponding reference plane, taking an average of the final depth scaling factors of all reference planes adjacent to the single-face region except the reference plane as a final scaling factor of the corresponding single-face region, and determining the coordinate conversion relationship of all single-face regions based on the final scaling factors of all single-face regions and original parameters of the camera, i.e.: wherein x, y and z are horizontal coordinate values, vertical coordinate values and depth coordinate values in the three-dimensional coordinate representation respectively, u and v are horizontal coordinate values and vertical coordinate values in the two-dimensional coordinate representation respectively, d is a depth value in the two-dimensional image, s is a depth scaling factor of the depth map of the corresponding reference plane, f x ,f y are focal lengths of the camera on the x and y axes, and cx and cy are aperture centers (i.e. horizontal and vertical offsets of the workpiece image origin relative to the imaging point of the optical center) of the camera.

[0088] The depth scaling factor of the profile angle point of the reference plane determines the scaling factor of the reference plane, which can not only accurately determine the final scaling factor of the reference plane, but also ensure the accuracy of the final scaling factor of other single-face regions determined based on the final scaling factor of the adjacent reference plane, because the profile angle point is located at the junction of the reference plane and other single-face regions;

[0089] The weld three-dimensional determination unit is configured to determine a two-dimensional coordinate representation of the selected weld based on original image features of all the selected welds in the workpiece image (i.e., determine a two-dimensional coordinate representation of each point of the selected weld in the workpiece image in the original image features of all the selected welds in the workpiece image), and determine a three-dimensional coordinate representation of the weld trajectory based on the coordinate conversion relationship and the two-dimensional coordinate representation of the selected weld (obtain the three-dimensional coordinate representation of the weld trajectory by substituting the two-dimensional coordinate representation of the selected weld into the coordinate conversion relationship).

[0090] Based on the determined two-dimensional to three-dimensional coordinate conversion relationship, the conversion from two-dimensional coordinates to three-dimensional coordinates is completed. Compared with the method of directly measuring three-dimensional distance and relative azimuth to determine the three-dimensional coordinate representation, the amount of data to be detected is much smaller, and therefore the determination efficiency is higher.

[0091] Embodiment 5:

[0092] On the basis of embodiment 4, the reference plane screening unit comprises:

[0093] The reference surface determination subunit is configured to take a single-face region where two non-parallel straight line profiles belong to the same single face as a reference surface (the reference surface is a single-face region that contains at least two non-parallel straight line profiles in all single-face regions).

[0094] The above steps achieve the preliminary screening of the single-face region, and complete the preliminary screening of the reference plane from the perspective of the profile features.

[0095] The pixel sequence determination subunit is configured to determine a gradual change feature judgment direction based on the extension direction of the two non-parallel straight line profiles (the extension direction is represented by an angle), determine a plurality of pixel point sequences in which the extension direction of adjacent pixel points (the extension direction of adjacent pixel points is a line parallel to the line corresponding to the gradual change feature judgment direction) and the gradual change feature judgment direction are consistent in the reference surface (the pixel point sequence is a sequence formed by a column of pixel points in the reference surface, and the column of pixel points satisfies the condition that the extension direction of adjacent pixel points and the gradual change feature judgment direction are consistent).

[0096] The pixel value gradual change feature of the pixel point sequence in the direction is judged by using the gradual change feature of the reference surface in the direction consistent with the extending direction of the straight line contour, so as to realize dataization of the region gradual change feature.

[0097] The gradual change feature judging unit is configured to judge whether the image features of all the pixel point sequences belonging to the same gradual change feature judging direction are gradually changed and smooth and consistent in the gradual change direction (i.e., whether the pixel difference values of all the adjacent pixel points in all the pixel point sequences belonging to the same gradual change feature judging direction are equal or approximately equal), and if yes, it is determined that the reference surface is the reference plane, otherwise, it is determined that the reference surface is not the reference plane.

[0098] On the basis of the foregoing screening of the reference surface, further screening of the reference plane is completed from the perspective of the image region features based on the gradual change features of the image region in the reference surface, so that the finally screened single-surface region is an image region in which a plane actually exists in the three-dimensional space.

[0099] Embodiment 6

[0100] On the basis of the embodiment 1, the welding parameter determining module comprises:

[0101] The bevel shape determining submodule is configured to identify a bevel cross-section region of the weld seam track in the weld seam marking image (the bevel cross-section region is a cross-section of a gap body to be welded in the workpiece to be welded, which is perpendicular to the extending direction of the weld seam track).

[0102] The coordinate three-dimensional conversion submodule is configured to determine a two-dimensional coordinate representation of the weld seam track and a two-dimensional coordinate of the bevel cross-section region in the weld seam marking image, and determine a three-dimensional coordinate representation of the bevel cross-section based on a coordinate conversion relationship between the two-dimensional coordinate representation and the three-dimensional coordinate representation of the weld seam track and the two-dimensional coordinate representation of the bevel cross-section region (i.e., the two-dimensional coordinate representation of the bevel cross-section region is substituted into the coordinate conversion relationship between the two-dimensional coordinate representation and the three-dimensional coordinate representation of the weld seam track to obtain the three-dimensional coordinate representation of the bevel cross-section).

[0103] The welding parameter determination submodule is configured to determine a three-dimensional coordinate representation of a weld shape body based on the three-dimensional coordinate representation of the weld seam track and the three-dimensional coordinate representation of the groove cross section, as a groove feature (therefore, the weld seam track can have a visual dead angle in the two-dimensional image, which results in the three-dimensional coordinate representation of the weld seam track not covering the three-dimensional coordinates of all points on the surface of the weld shape body, and the three-dimensional coordinate representation of the groove cross section area can be used to restore the three-dimensional coordinates of the weld seam track, thereby obtaining the three-dimensional coordinates of all points on the surface of the weld shape body), determine the welding parameters based on the groove feature and a preset experience parameter corresponding to the current welding method (for example, resistance welding or arc welding), and determine the welding parameters based on the welding parameter determination mode corresponding to the current welding method (different welding methods correspond to different welding parameters, and therefore, the welding parameter determination mode corresponding to the current welding method is different, which can be imported in advance according to artificial experience, or can be determined by training a large number of historical welding parameter instances).

[0104] The above steps achieve perfect determination of the three-dimensional coordinates of all points on the surface of the weld shape body, ensure the data integrity of the groove feature, and further improve the accuracy of the finally determined welding parameters.

[0105] Embodiment 7:

[0106] On the basis of embodiment 1, the control parameter determination module comprises:

[0107] The second training building submodule is configured to train a welding control parameter determination model based on a large number of historical welding control instances containing historical welding parameters (the historical welding parameters are the welding parameters contained in the historical welding control instance), historical track movement parameters (the historical track movement parameters are the historical track movement parameters contained in the historical welding control instance), and corresponding historical welding control parameters (the historical welding control parameters are the target welding control parameters contained in the historical welding control instance), and the historical welding control parameters are determined by artificial experience based on the historical welding parameters and the historical track movement parameters.

[0108] The control parameter determination submodule is configured to input the welding parameters and the track movement parameters into the welding control parameter determination model to obtain the target welding control parameters.

[0109] Based on the functions performed by the above-mentioned second training building submodule and the control parameter determination submodule, the welding control parameter determination model for determining the target welding control parameters is built based on deep learning, and the target welding control parameters required for the current welding control are further determined based on the welding control parameter determination model.

[0110] Embodiment 8:

[0111] On the basis of Embodiment 1, the welding control optimization module, referring to Figure 3 , comprises:

[0112] a welding dynamic simulation submodule for dynamically simulating a welding process based on the welding parameters and the trajectory movement parameters to obtain a virtual welding trajectory (the virtual welding trajectory is a welding trajectory formed in an ideal state when the simulated welding robot is directly controlled according to the welding parameters and the trajectory movement parameters) ;

[0113] By simulating the virtual welding trajectory in the ideal state, a basis for deviation calculation is provided for the subsequent process of determining the welding control parameter determination model and optimizing the welding control parameters, that is, an intuitive reference is provided for the subsequent optimization process;

[0114] a welding control monitoring submodule for controlling the welding robot to perform welding based on the target welding control parameters and acquiring a welding torch head monitoring video of the welding robot in real time (the welding torch head monitoring video is a monitoring video acquired in real time and containing a region where a welding torch head of the welding robot and a contact position of the welding torch head with a workpiece are located) ;

[0115] an actual information acquisition submodule for determining time sequence information and spatial sequence information of the welding position of the welding torch head based on the welding torch head monitoring video;

[0116] This step is equivalent to numerizing the application result of the output of the welding control parameter determination model and representing the output result from two dimensions of time and space, which facilitates subsequent adjustment and optimization from the two angles of time and space, and the adjustment and optimization effect is better than that of directly adjusting and optimizing based on the dynamic trajectory;

[0117] a control parameter optimization submodule for continuously optimizing the welding control parameter determination model based on the latest obtained time sequence information and spatial sequence information of the welding position of the welding torch head and the virtual welding trajectory to obtain the latest welding optimization control parameters (referring to the logic regression closed-loop control process of Figure 3 , continuously updating and optimizing the target welding control parameters based on the continuously obtained welding optimization control parameters, wherein the welding optimization control parameters are the deviation between the time sequence information and the spatial sequence information of the welding position and the corresponding time sequence information and spatial sequence information of the virtual welding trajectory, the weights between layers in the welding control parameter determination model are adjusted, and then the welding control parameter determination model is optimized, and the welding optimization control parameters are obtained based on the optimized welding control parameter determination model) ;

[0118] The step optimizes and adjusts the welding control parameter determination model and the welding control parameter based on the output results of the two dimensions of time and space, so that the adjustment optimization effect is better, and the deviation between the actual welding trajectory formed by the actual position of the welding torch head and the virtual welding trajectory becomes smaller at a faster speed.

[0119] The welding control optimization submodule is used to continuously update the target welding control parameter of the welding robot based on the latest obtained welding optimization control parameter (i.e., replacing the target welding control parameter with the latest welding optimization control parameter), until the welding position of the welding torch head reaches the end point of the virtual welding trajectory, and a complete welding result is obtained.

[0120] Embodiment 9:

[0121] On the basis of embodiment 8, the welding dynamic simulation submodule includes:

[0122] The welding parameter analysis unit is used to determine the welding cross-section forming parameter (i.e., the geometric parameter of the cross section of the ideal state welding shape body) and the welding cross-section normal moving parameter (i.e., the extension speed parameter in the direction perpendicular to the cross section of the ideal state welding shape body) of the welding fusion zone (the ideal state welding shape body formed when the welding robot is directly controlled according to the welding parameter and the trajectory moving parameter) based on the welding parameter;

[0123] The trajectory moving parameter analysis unit is used to determine the trajectory moving direction based on the trajectory moving parameter;

[0124] The welding process dynamic simulation unit is used to simulate the dynamic welding process (i.e., the dynamic extension process of the ideal state welding shape body based on the geometric parameter of the cross section of the ideal state welding shape body, the extension speed parameter in the direction perpendicular to the cross section, and the trajectory moving direction) based on the welding cross-section forming parameter and the welding cross-section normal moving parameter and the trajectory moving direction, and obtain the virtual welding trajectory;

[0125] The dynamic simulation of the ideal state welding process formed when the welding robot is directly controlled according to the welding parameter and the trajectory moving parameter is completed.

[0126] Embodiment 10:

[0127] On the basis of embodiment 9, the actual information acquisition submodule includes:

[0128] The spatial sequence information determination unit is configured to determine the actual welding trajectory based on the welding position of the welding torch (i.e., the welding position is determined based on the pre-trained recognition model in the last frame of the welding torch monitoring video, and the three-dimensional coordinates of the welding position are obtained by converting the welding position to three-dimensional coordinates), fit the three-dimensional coordinates of the welding position determined in sequence according to time to obtain the actual welding trajectory (i.e., the trajectory formed by the welding torch and the points on the workpiece surface), divide the actual welding trajectory according to a preset slicing interval (a preset length interval) to obtain the spatial sequence information of the welding position of the welding torch (i.e., the coordinates of each part of the trajectory obtained by division represent a spatial information in the spatial sequence information).

[0129] The time sequence information determination unit is configured to determine the coincident position points (i.e., the position points coincident in the actual welding trajectory and the virtual welding trajectory) of the actual welding trajectory and the virtual welding trajectory, divide the actual welding trajectory according to the coincident position points as the division limit to obtain a plurality of part trajectory sequences (i.e., a sequence containing a plurality of part trajectories), generate corresponding time slices based on the formation time period (i.e., the time period between the time when the welding torch passes the starting point of the part trajectory and the time when the welding torch passes the ending point of the part trajectory) of each part trajectory in the part trajectory sequence, and generate the spatial sequence information of the welding position of the welding torch based on the time slices of all part trajectories in the part trajectory sequence.

[0130] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A deep learning-based welding robot vision control system, characterized by, The method comprises the following steps: a weld seam trajectory determination module is used to build a weld seam area recognition model based on a deep learning algorithm, and to determine a weld seam trajectory based on the weld seam area recognition model and a selected sorting instruction input by a user; a movement parameter determination module is used to determine a three-dimensional coordinate representation of the weld seam trajectory based on original image features of all selected weld seams in the weld seam trajectory in a workpiece image, and to generate trajectory movement parameters based on the three-dimensional coordinate representation of the weld seam trajectory, comprising: a three-dimensional coordinate determination sub-module is used to determine a three-dimensional coordinate representation of the weld seam trajectory based on original image features of all selected weld seams in the weld seam trajectory in the workpiece image; a movement parameter determination sub-module is used to determine trajectory parameters of the weld seam trajectory as the trajectory movement parameters based on the three-dimensional coordinate representation of the weld seam trajectory; wherein the three-dimensional coordinate determination sub-module comprises: a region contour recognition unit is used to recognize all contour lines in the workpiece image based on a contour recognition algorithm, to recognize straight line contours in all the contour lines, and to determine single-face regions in the workpiece image based on all the contour lines; a reference plane screening unit is used to screen a plurality of reference planes in all the single-face regions based on all the straight line contours in the workpiece image; a three-dimensional coordinate determination unit is used to determine three-dimensional distances of each contour corner point of the reference planes from a preset three-dimensional origin point based on a distance measuring sensor, and to determine relative azimuth angles of each contour corner point in a preset three-dimensional coordinate system, to determine a three-dimensional coordinate representation of each contour corner point based on the three-dimensional distance and the relative azimuth angle of each contour corner point; a conversion relationship determination unit is used to determine a coordinate conversion relationship based on the three-dimensional coordinate representation and the two-dimensional coordinate representation of the contour corner points of all the reference planes and pixel parameters of all the reference planes; a weld seam three-dimensional determination unit is used to determine a two-dimensional coordinate representation of the selected weld seam based on original image features of all the selected weld seams in the workpiece image, and to determine a three-dimensional coordinate representation of the weld seam trajectory based on the coordinate conversion relationship and the two-dimensional coordinate representation of the selected weld seam; a welding parameter determination module is used to recognize a groove feature of the weld seam trajectory in a weld seam marking image, and to determine welding parameters based on the groove feature and preset experience parameters; a control parameter determination module is used to build a welding control parameter determination model based on a deep learning algorithm, and to input the welding parameters and the trajectory movement parameters into the welding control parameter determination model to obtain target welding control parameters; a welding control optimization module is used to control a welding robot to perform welding based on the target welding control parameters, and to constantly learn and optimize welding control of the welding robot based on time sequence information and spatial sequence information of a latest obtained welding position of a welding torch head until a complete welding result is obtained; wherein the welding control optimization module comprises: a welding dynamic simulation sub-module is used to dynamically simulate a welding process based on the welding parameters and the trajectory movement parameters to obtain a virtual welding trajectory; a welding control monitoring sub-module is used to control the welding robot to perform welding based on the target welding control parameters, and to acquire a welding torch head monitoring video of the welding robot in real time. The actual information acquisition submodule is configured to determine time sequence information and spatial sequence information of the welding position of the welding torch based on the welding torch monitoring video. The control parameter optimization submodule is configured to continuously optimize the welding control parameter determination model based on the latest obtained time sequence information and spatial sequence information of the welding position of the welding torch and the virtual welding trajectory, and obtain the latest welding optimization control parameter. The welding control optimization submodule is configured to continuously update the target welding control parameter of the welding robot based on the latest obtained welding optimization control parameter, until the latest obtained welding position of the welding torch reaches the end point of the virtual welding trajectory, and obtain a complete welding result. The welding dynamic simulation submodule includes: The welding parameter analysis unit is configured to determine welding cross-section forming parameters and welding cross-section normal movement parameters of the welding fusion zone based on the welding parameters. The trajectory movement parameter analysis unit is configured to determine a trajectory movement direction based on the trajectory movement parameters. The welding process dynamic simulation unit is configured to simulate a dynamic welding process based on the welding cross-section forming parameters, the welding cross-section normal movement parameters, and the trajectory movement direction, and obtain the virtual welding trajectory. The actual information acquisition submodule includes: The spatial sequence information determination unit is configured to determine the welding position of the welding torch based on the welding torch monitoring video, determine an actual welding trajectory, divide the actual welding trajectory according to a preset slicing interval, and obtain spatial sequence information of the welding position of the welding torch. The time sequence information determination unit is configured to determine a coincidence position point of the actual welding trajectory and the virtual welding trajectory, divide the actual welding trajectory according to the coincidence position point as a division limit, obtain a plurality of partial trajectory sequences, generate a corresponding time slice based on a formation time period of each partial trajectory in the partial trajectory sequences, and generate the spatial sequence information of the welding position of the welding torch based on the time slices of all the partial trajectories in the partial trajectory sequences.

2. The deep learning-based welding robot vision control system of claim 1, wherein, The weld trajectory determination module includes: The first training building submodule is configured to obtain a weld region recognition model by performing deep learning training on a large number of workpiece images in which different weld types are marked. The weld recognition and marking submodule is configured to recognize candidate weld regions in a two-dimensional image of a workpiece to be welded in real time based on the weld region recognition model, and highlight mark all the candidate weld regions in the two-dimensional image to obtain a weld mark image. The user selection and sorting submodule is configured to select and sort all the candidate weld regions in the weld mark image based on a user selection and sorting instruction, and obtain a weld trajectory.

3. The deep learning-based welding robot vision control system of claim 1, wherein, The reference plane screening unit includes: The reference surface determination subunit is configured to regard a single surface region in which two non-parallel straight line contours belonging to the same single surface are located as a reference surface. The pixel sequence determination subunit is configured to determine a gradual change feature judgment direction based on extension directions of the two non-parallel straight line contours, and determine a plurality of pixel point sequences in which adjacent pixel points have the same extension direction as the gradual change feature judgment direction in the reference surface. The gradual change feature judging unit is configured to judge whether image features of all pixel point sequences belonging to the same gradual change feature judging direction are gradually smooth and consistent in gradual change direction, and if yes, the reference surface is determined as the reference plane, otherwise, the reference surface is determined as not the reference plane.

4. The deep learning-based welding robot vision control system of claim 1, wherein, The welding parameter determination module comprises: The groove shape determination submodule is configured to identify a groove cross-section region of the weld seam track in the weld seam mark image; The coordinate three-dimensional conversion submodule is configured to determine a two-dimensional coordinate representation of the weld seam track and a two-dimensional coordinate of the groove cross-section region in the weld seam mark image, and determine a three-dimensional coordinate representation of the groove cross-section based on a coordinate conversion relationship between the two-dimensional coordinate representation and the three-dimensional coordinate representation of the weld seam track and the two-dimensional coordinate representation of the groove cross-section region; The welding parameter determination submodule is configured to determine a three-dimensional coordinate representation of the weld seam shape body as the groove feature based on the three-dimensional coordinate representation of the weld seam track and the three-dimensional coordinate representation of the groove cross-section, and determine the welding parameter based on the groove feature and a welding parameter determination mode corresponding to the current welding mode in the preset experience parameter.

5. The deep learning-based welding robot vision control system of claim 1, wherein, The control parameter determination module comprises: The second training building submodule is configured to train the welding control parameter determination model based on a large number of historical welding control instances containing historical welding parameters, historical track movement parameters and corresponding historical welding control parameters; The control parameter determination submodule is configured to input the welding parameter and the track movement parameter into the welding control parameter determination model to obtain the target welding control parameter.

Images