Industrial robot work method, device, electronic equipment and storage medium

Abstract

The application provides an industrial robot operation method, device, electronic equipment and storage medium. The industrial robot operation method comprises: obtaining an operation target image, and obtaining three-dimensional point cloud data corresponding to an operation target based on the operation target image; obtaining operation target pose information based on the three-dimensional point cloud data; inputting the operation target pose information into an operation path recognition model to obtain an operation path, and obtaining path planning information based on the operation path; wherein the operation path recognition model is obtained by performing transfer learning on a three-dimensional U-Net network model of a cloud server; obtaining operation instructions based on the operation path and operation process target parameters, and controlling the industrial robot to operate based on the operation instructions. The industrial robot operation method provided by the application can improve the autonomous operation capability of the operation robot and improve the efficiency of setting parameters.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to an operating method, device, electronic equipment and storage medium of an industrial robot. Background Art

[0002] Industrial robots integrate modern manufacturing technology, new material technology and information control technology. They are representative products of intelligent manufacturing and an important indicator of a country's scientific and technological innovation and manufacturing level.

[0003] Industrial robots are widely used in the automotive, shipbuilding, construction, and aerospace industries. Currently, the main programming methods for industrial robots include teach-and-play and offline programming. These two programming methods once played a significant role in traditional batch manufacturing, but they suffer from two drawbacks that have kept them increasingly out of step with the demands of modern manufacturing: one program can only be used for one workpiece, and programming and debugging are time-consuming, failing to meet the demands of small-batch, high-variety, and flexible manufacturing. Robot programming also requires programmers to understand both the operating process and the robot's operation, a demanding skill set. Furthermore, industrial robots typically operate in harsh and demanding environments, making it difficult to recruit suitable programmers. Consequently, existing industrial robot systems, despite the highly flexible and customized operation model, still suffer from weak autonomous operation capabilities and inefficient parameter setting, failing to meet the demands for efficient and autonomous operation. Summary of the Invention

[0004] The present invention provides an operating method, device, electronic equipment and storage medium for an industrial robot, which are used to solve the problems in the prior art such as weak autonomous operating capability of industrial robots, low efficiency in setting parameters, and inability to meet the needs of efficient autonomous operation. The present invention can improve the autonomous operating capability of the operating robot and improve the efficiency of setting parameters.

[0005] The present invention provides an operating method of an industrial robot, comprising:

[0006] Acquire an operation target image, and obtain three-dimensional point cloud data corresponding to the operation target based on the operation target image;

[0007] Based on the three-dimensional point cloud data, obtaining the work target posture information;

[0008] Inputting the work target pose information into a work path recognition model to obtain a work path, and obtaining path planning information based on the work path; wherein the work path recognition model is obtained by performing transfer learning on the three-dimensional U-Net network model of the cloud server;

[0009] Based on the operation path and the operation process target parameters, an operation instruction is obtained, and the industrial robot is controlled to perform the operation based on the operation instruction.

[0010] According to the industrial robot operation method provided by the present invention, obtaining the operation target posture information based on the three-dimensional point cloud data includes:

[0011] Inputting the three-dimensional point cloud data into a point cloud classification network model to obtain a classification result of the three-dimensional point cloud data; wherein the point cloud classification network model is trained on the cloud server;

[0012] Based on the classification result, selecting a target two-dimensional image template from a preset two-dimensional image template library, and selecting a three-dimensional point cloud template from a preset three-dimensional point cloud template library;

[0013] Matching the target two-dimensional image template with the three-dimensional point cloud data, and adjusting the orientation of the target three-dimensional point cloud template based on the matching result;

[0014] Based on the adjusted target 3D point cloud template, the work target posture information is obtained.

[0015] According to the industrial robot operation method provided by the present invention, obtaining three-dimensional point cloud data corresponding to the operation target based on the operation target image includes:

[0016] Gray coding and phase shift coding are performed on the target image to obtain a coding pattern;

[0017] Projecting the coded pattern onto a work target by the industrial robot and acquiring an image of the work target taken again by an industrial camera;

[0018] Gray decoding and phase shift decoding are performed on the re-photographed work target image to obtain a decoding pattern;

[0019] The three-dimensional point cloud data is obtained based on the decoded pattern and the stereo calibration result of the industrial camera.

[0020] According to the operation method of the industrial robot provided by the present invention, obtaining path planning information based on the operation path includes:

[0021] Performing curve fitting on the characteristic points of the operation path to obtain an operation path curve;

[0022] Performing a first-order derivative of the work path curve to obtain a direction vector, and obtaining a normal vector of the work path based on the direction vector and an approach vector of the work path;

[0023] The path planning information is obtained based on the direction vector, the approach vector and the normal vector.

[0024] The operating method of the industrial robot provided by the present invention further includes:

[0025] After the industrial robot starts working, acquiring a depth image and a structured light image of the working target;

[0026] Obtaining a first histogram of oriented gradients feature based on the depth image, and obtaining a second histogram of oriented gradients feature and a color feature based on the structured light image;

[0027] Inputting the first oriented gradient histogram feature, the second oriented gradient histogram feature, and the color feature into a correlation filter to obtain three response graphs; wherein the correlation filter is trained on the cloud server;

[0028] Based on the three response diagrams, the characteristic size and characteristic position of the work target are obtained, and based on the characteristic size and characteristic position of the work target, it is determined that when there is an obstructing workpiece, tracking is stopped, and when it is determined that there is no obstructing workpiece, the normal tracking state is maintained.

[0029] The operating method of the industrial robot provided by the present invention further includes:

[0030] An artificial potential field of the camera field of view is established, and when it is determined based on the artificial potential field value of the camera field of view that the camera is about to leave the field of view, the camera position is adjusted so that the feature points at the edge of the camera field of view move toward the center of the field of view.

[0031] The present invention also provides an operating device of an industrial robot, comprising:

[0032] A point cloud acquisition module is used to acquire an image of a work target and obtain three-dimensional point cloud data corresponding to the work target based on the image of the work target;

[0033] A posture extraction module is used to obtain the posture information of the work target based on the three-dimensional point cloud data;

[0034] a path planning module, configured to input the work target pose information into a work path recognition model to obtain a work path, and obtain path planning information based on the work path; wherein the work path recognition model is obtained by performing transfer learning on the three-dimensional U-Net network model of the cloud server;

[0035] The instruction generation module is used to obtain an operation instruction based on the operation path and the operation process target parameters, and control the industrial robot to perform the operation based on the operation instruction.

[0036] The present invention also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described industrial robot operation methods are implemented.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described industrial robot operation methods.

[0038] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of any of the above-mentioned industrial robot operation methods.

[0039] The industrial robot operation method, device, electronic device and storage medium provided by the present invention can be used in an edge server. First, an operation target image is obtained, and based on the operation target image, three-dimensional point cloud data is obtained. An operation path recognition model is obtained by performing transfer learning on the three-dimensional U-Net network model of the cloud server. The operation path recognition model is used to process the three-dimensional point cloud data to obtain an operation path, and further obtain path planning information. Then, based on the path planning information and the operation process target parameters, operation instructions are obtained to control the operation robot to perform the operation.

[0040] By collaborating with cloud servers and edge servers, the autonomous operation of industrial robots can be controlled, giving industrial robots powerful autonomous operation capabilities. This can improve the autonomous operation capabilities of operating robots, increase the efficiency of setting parameters, reduce the workload of robot operators, and alleviate the real problem of labor shortage. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 This is one of the flow charts of the operating method of the industrial robot provided by the present invention;

[0043] Figure 2 This is the second flow chart of the operating method of the industrial robot provided by the present invention;

[0044] Figure 3 This is a structural diagram of the cloud-edge-device collaboration module system architecture provided by the present invention;

[0045] Figure 4This is a schematic diagram of the continuous learning algorithm for industrial robots provided by the present invention;

[0046] Figure 5 This is a flow chart of the method for identifying the pose of an operating target based on transfer learning and hybrid vision provided by the present invention;

[0047] Figure 6 This is a flow chart of a method for three-dimensional reconstruction of an operating target provided by the present invention;

[0048] Figure 7 is a flow chart of the operation path extraction method provided by the present invention;

[0049] Figure 8 It is a schematic diagram generated by the operation program provided by the present invention;

[0050] Figure 9 This is a principle block diagram of the operation path feature point extraction provided by the present invention;

[0051] Figure 10 This is a schematic diagram of the field of view constraint for operation path tracking provided by the present invention;

[0052] Figure 11 This is a block diagram of the principle of feedback adjustment for an industrial robot provided by the present invention;

[0053] Figure 12 It is a structural schematic diagram of the operating device of the industrial robot provided by the present invention;

[0054] Figure 13 It is a structural schematic diagram of the electronic device provided by the present invention.

[0055] Reference numerals:

[0056] 1010: Bracket; 1020: Working tools; 1030: Camera;

[0057] 1040: Rotation axis; 1050: Working path; 1200: Working device;

[0058] 1210: Point cloud acquisition module; 1220: Pose extraction module; 1230: Path planning module;

[0059] 1240: Instruction generation module. DETAILED DESCRIPTION

[0060] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0061] The following combination Figures 1-13 The invention describes an industrial robot operation method, device, electronic device and storage medium.

[0062] The present invention provides an industrial robot operation method, which is applied to an edge server; Figure 1 As shown, the present invention provides an operating method of an industrial robot including:

[0063] Step 110: Acquire an operation target image, and obtain three-dimensional point cloud data corresponding to the operation target based on the operation target image.

[0064] It can be understood that the acquired operation target image is a structured light image corresponding to the operation target, that is, an operation target image acquired based on structured light.

[0065] The operation target is the target object that needs to be operated. The operation target can be an operation workpiece, for example, a welding workpiece corresponding to a welding robot, or a palletizing workpiece corresponding to a palletizing robot, or a spraying workpiece corresponding to a spraying robot.

[0066] Step 120: Obtain the work target posture information based on the three-dimensional point cloud data.

[0067] It can be understood that the work target posture information includes the position and posture information of the work target.

[0068] Step 130: Input the work target posture information into the work path recognition model to obtain the work path, and obtain path planning information based on the work path; wherein the work path recognition model is obtained by performing transfer learning on the three-dimensional U-Net network model of the cloud server, using the preset work target posture information as a sample and the work path corresponding to the preset work target posture information as a sample label.

[0069] It can be understood that the three-dimensional U-Net network model is trained on a cloud server using preset work target posture information as samples and the work path corresponding to the preset work target posture information as sample labels.

[0070] Step 140: Obtain an operation instruction based on the operation path and the operation process target parameters, and control the industrial robot to perform the operation based on the operation instruction.

[0071] It is understandable that after obtaining the operation path, the automatic generation of operation instructions begins. The operation instructions can be a programming program. After the operation instructions are generated, the operation instructions are sent to the operation robot to control the operation robot to perform the operation.

[0072] Generation of target process parameters: This system integrates expert experience, including relevant rules, models, and knowledge bases, to establish an expert process database. Based on this database, the extracted process path is automatically segmented and numbered, and process parameters such as welding voltage and current, wire feed speed, dry stick length, gun trajectory, width, frequency, abrasive belt speed, feed rate, grinding torque, grinding depth, grinding angle, spray flow rate, molding air, cup speed, assembly trajectory, assembly force, and process speed are automatically set for each segment.

[0073] Programming program generation for the operation: The industrial robot trajectory planning module plans the robot trajectory according to the operation path and operation process parameters, generates a series of robot joint coordinate values, and the program instruction module generates operation instructions, including operation mode initialization instructions, operation initial point guidance instructions, motion and control instructions, return to HOME point motion instructions, etc.

[0074] Since the robot programming languages ​​and communication interfaces currently used by various robot manufacturers are incompatible with each other, in order to facilitate application and promotion, this embodiment uses the national standard "GB / T 32197-2015 Robot Controller Open Communication Specification" (hereinafter referred to as GB / T 32197-2015) promulgated and implemented in 2016 as the basis for generating instruction codes.

[0075] GB / T 32197-2015 defines a set of communication interface protocols for obtaining the basic properties of industrial robots and controlling their operating modes. Readable basic robot properties include joint position, velocity, acceleration, and DI / AI status values; controllable operating modes include servo motion mode, servo cycle, DO / AO output, etc. All of these properties and operating mode variables are associated with one or a set of communication objects with a "dictionary index." Furthermore, GB / T 32197-2015 provides an expansion mechanism that allows equipment manufacturers to provide unique functionality. These features of GB / T 32197-2015 meet the requirements of this invention for industrial robot program generation.

[0076] In some embodiments, Figure 2 FIG. 1 is another flow chart of the operation method of the industrial robot provided by the present invention, as shown in FIG. Figure 2As shown, the method includes: a cloud-edge-end collaboration module, a continuous learning and evolution module, a work target three-dimensional reconstruction module, a work target posture recognition module, a work path extraction module, an work program automatic generation module, a workpiece feature point extraction module, a work path tracking module, etc.

[0077] The cloud-edge-end collaboration module combines cloud servers, edge servers, and robot terminals to form an autonomous operation system for industrial robots. The continuous learning and evolution module uses a knowledge accumulation-based method to continuously improve the cognitive and decision-making capabilities of the autonomous operation system. The operation target three-dimensional reconstruction module uses deep learning and coded structured light to reconstruct the operation target in three dimensions. The operation target posture recognition module uses transfer learning and hybrid vision to recognize the position and posture of the operation target. The operation path extraction module uses deep learning technology to segment and extract operation path points.

[0078] The automatic operation program generation module integrates expert experience to generate robot operation programs containing operation instructions. The workpiece feature point extraction module uses the correlation filtering method of multi-feature fusion to extract the feature points of the workpiece. The operation path tracking module realizes robust tracking of the industrial robot's operation path through artificial potential field method and fuzzy control.

[0079] Among them, according to the autonomous operation requirements of industrial robots, this embodiment builds a cloud-edge-end collaborative module system architecture as follows Figure 3 As shown in the figure, the current computing power of robots is limited. Wirelessly extended cloud computing capabilities are needed to provide the computing power required by robots. The neural network of the industrial robot system is constructed through wireless 5G communication networks and Ethernet networks, connecting the robot itself, various sensors, edge processors, and cloud servers.

[0080] The cloud server has system management capabilities such as data storage, model management, and process monitoring. It can also conduct deep learning training, adapt to a variety of neural network algorithm libraries, and possess intelligent analysis and decision-making capabilities.

[0081] Edge servers, also known as edge computing devices, are the output end of edge devices. They have high real-time and scalability. They also have strong floating-point computing capabilities to optimize the deployment of deep learning models trained on cloud servers. Therefore, edge servers have functions such as data organization, intelligent voice, intelligent vision, intelligent motion, process optimization, security and privacy.

[0082] In order to enable industrial robots to complete autonomous programming and autonomous operation, a variety of sensing devices are configured for industrial robot terminals to realize the robot terminals' environmental perception capabilities such as sound, two-dimensional images, three-dimensional point clouds, text, torque, etc.

[0083] At the same time, the robot terminal receives motion operation instructions from the edge server and applies the operation process, such as completing the adjustment of welding voltage and current, gun swing trajectory, width, and frequency during robot welding, completing the adjustment of grinding belt speed, feed speed, grinding angle, and grinding depth during robot grinding, completing the adjustment of spray flow, molding air, and rotation speed during robot spraying, completing the modification of assembly trajectory, assembly force, and motion speed during robot assembly, and completing the adjustment of handling speed, handling posture, and handling trajectory during robot handling.

[0084] Through cloud-edge-end integration, the advantages of cloud computing, cloud storage and other infrastructure shared services can be effectively leveraged. Compared with independent robot bodies, robots connected to the cloud have disruptive capabilities.

[0085] Information and knowledge sharing among groups: A cloud server can control many robots. The cloud server can collect the visual, voice and environmental information of all connected robots. The data information after intelligent processing by the cloud brain can be used by all connected robots.

[0086] Using cloud servers, we can aggregate information collected by each robot for training and updating, continuously improving the robot's autonomous operation capabilities. Through cloud servers, robots no longer work independently; multiple robots can work together, enabling flexible production capabilities across multiple robot systems.

[0087] With the evolution of manufacturing models, the tasks generated in flexible and customized work models are constantly changing, resulting in complex and diverse tasks. However, existing industrial robot learning strategies primarily focus on data-driven, single-task learning strategies, failing to consider the generalization and scalability of single-task learning models. This significantly limits the perception and decision-making capabilities of industrial robots, making them unable to effectively handle the complex and diverse tasks in flexible and customized work models, posing certain challenges to the autonomous operation of industrial robots.

[0088] To address the above issues, in order to improve the generalization ability of deep learning models in different work tasks, cloud servers are used to help learn the knowledge required for new tasks based on the accumulated knowledge of previous tasks. At the same time, the performance of deep learning models in known tasks is maximized. Through knowledge accumulation and updating, the common representation performance of deep learning models in different work tasks is continuously improved, thereby continuously enhancing the cognitive and decision-making capabilities of industrial robots in complex working environments.

[0089] Figure 4 This is a schematic diagram of the continuous learning algorithm for industrial robots proposed in this invention. The algorithm model is mainly divided into three parts: based on the shared task parameter θ sA base network with parameters θ for the previous task o A network with parameters θ for the new task n network.

[0090] The basic network uses a pre-trained neural network model (such as MobileNet, ResNet, Inception, etc.) to initialize the basic network weights. The parameter is θ o and θ n The network structure is designed according to the different task types under the actual operation objectives. By fine-tuning the parameters θ s , and train the parameters θ n Make the model achieve the best performance on the new task and realize the transfer and update of knowledge. At the same time, fine-tune the parameter θ o , so that its model (θ s ,θ o ) Keep the performance on the previous task as unchanged as possible to achieve knowledge accumulation.

[0091] By sharing parameters, this mechanism can learn the constraints and relationships between different tasks, improving the model's robustness and generalization performance while effectively avoiding the problem of overly large network models. Unlike transfer learning models, this learning mechanism not only involves the transfer of knowledge between tasks but also considers the continuous updating and retention of knowledge. It can be used to adapt to different work objectives and task types, providing an innovative solution for autonomous decision-making in industrial robots in complex environments.

[0092] To achieve autonomous operation for industrial robots, accurate identification of target objects is crucial. Industrial production environments are complex and volatile, characterized by varying lighting conditions and complex backgrounds. These complexities, combined with the simple textures and low contrast of target objects, significantly impact target identification. To ensure accurate target identification, this paper proposes a method for 3D target reconstruction based on deep learning and coded structured light, as well as a method for target pose recognition based on transfer learning and hybrid vision.

[0093] In some embodiments, obtaining the work target pose information based on the three-dimensional point cloud data includes:

[0094] Input the 3D point cloud data into the point cloud classification network model to obtain the classification results of the 3D point cloud data; wherein, the point cloud classification network model is trained on the cloud server based on the PointNet network, using the preset 3D point cloud data as samples and the classification results corresponding to the preset 3D point cloud data as sample labels, and then sent to the edge server;

[0095] Based on the classification results, a target two-dimensional image template is selected from a preset two-dimensional image template library, and a three-dimensional point cloud template is selected from a preset three-dimensional point cloud template library;

[0096] Matching the target 2D image template with the 3D point cloud data, and adjusting the orientation of the target 3D point cloud template based on the matching result;

[0097] Based on the adjusted target 3D point cloud template, the work target posture information is obtained.

[0098] It should be noted that the target two-dimensional image template and the target three-dimensional point cloud template are templates corresponding to the categories of the three-dimensional point cloud data.

[0099] It is understandable that after training the point cloud classification network model, the cloud server obtains a migration module based on the trained point cloud classification network model and sends the migration module to the edge server.

[0100] After obtaining the three-dimensional point cloud data corresponding to the work target in the work target, in order to complete the work path planning of the industrial robot, it is necessary to perform posture recognition on the work target, that is, the work target.

[0101] The work target posture recognition mainly includes two tasks. The first task is to classify the type of work target and determine which category the work target belongs to; the second task is to obtain the three-dimensional position and posture information of the work target relative to the robot base coordinate system, that is, the posture information.

[0102] The traditional method to solve the problem of workpiece pose recognition is based on workpiece feature matching. For different recognition tasks and scenarios, manual selection or design of appropriate workpiece features is required to obtain accurate recognition results. Therefore, traditional methods are highly dependent on humans and lack autonomy and adaptability.

[0103] Deep learning methods have been used in recent years to identify object poses because they can automatically discover and extract features. However, these methods require extensive offline training with large amounts of data to achieve high recognition rates, resulting in weak generalization capabilities. Consequently, when the task objectives change, existing methods require extensive retraining to ensure accurate and stable performance.

[0104] To solve this problem, the present invention proposes a method for object pose recognition based on transfer learning and hybrid vision. The pose recognition process is as follows: Figure 5As shown in the figure, the PointNet network structure is improved by using transfer learning technology, so that the neural network can be quickly deployed to the recognition task of unknown work targets to improve the flexibility and accuracy of pose recognition; and the two-dimensional image information and three-dimensional point cloud information obtained by coded structured light are fused to complete the work target pose estimation, and high-precision pose estimation results are obtained by combining coarse matching and fine matching.

[0105] Extracting 3D point cloud information includes the following steps:

[0106] Operation target point cloud recognition based on transfer learning: The network structure design process of transfer learning in this embodiment is as follows: first, the PointNet network is trained on the ModelNet40 dataset of the cloud server, and the network weights and network structure of the first half are extracted from the trained PointNet network model and remain unchanged. This part is called the migration module.

[0107] Because the transfer module is directly extracted from the trained PointNet network model, it can extract stable global features from the workpiece point cloud data. These features are then fed into a newly constructed multi-layer classification network to classify the new data. When training on a new dataset, care must be taken to ensure that the parameters within the transfer module are not involved in weight updates. Therefore, the transfer module can be considered a complex nonlinear function. Since the training process only requires learning the parameters of the multi-layer classification network, the number of parameters required is greatly reduced. This results in a shorter training time for the transfer learning network and a faster learning capability.

[0108] Pose estimation by integrating 2D image information and 3D point cloud information: First, a template library containing 2D image templates and 3D point cloud templates is established in advance for all workpieces to be inspected. The template library of 2D image templates and 3D point cloud templates can be stored on a cloud server or an edge server.

[0109] Then, a scale-invariant feature transform (i.e., SIFT) is used on the two-dimensional image for rough matching. Usually, the feature matching accuracy of the scale-invariant feature transform is not high. Therefore, the present invention proposes a multi-template matching strategy, which can select the most appropriate template to match the detected workpiece.

[0110] Subsequently, the 3D point cloud template is rotated and translated based on the matching results of the 2D image. The point cloud on the 3D point cloud template is transformed to a pose that is roughly the same as the target point cloud (i.e., the collected 3D point cloud data). This pose is used as the initial pose for the iterative closest point (ICP) matching algorithm, and the iterative closest point matching algorithm is executed to accurately match the 3D point cloud. The pose of the target relative to the reference template is solved by optimizing the Euclidean distance error of the corresponding points of the two sets of point clouds. The Euclidean distance matching error is defined as follows:

[0111]

[0112] Where x i is the 3D point in the target point cloud, c i is a 3D point in the template point cloud, R is the relative rotation matrix, t is the relative translation vector, and N is the total number of matching points. This algorithm uses 2D image matching to find a good initial pose for 3D point cloud matching, significantly improving the convergence speed of the iterative closest point matching algorithm and effectively avoiding the problem of iterative closest point matching algorithm converging to a local optimum. It is a fast and highly accurate method.

[0113] In some embodiments, obtaining three-dimensional point cloud data corresponding to the work target based on the work target image includes:

[0114] Gray coding and phase shift coding are performed on the target image to obtain a coding pattern;

[0115] The coded pattern is projected onto the work target by the industrial robot, and the image of the work target is captured again by the industrial camera;

[0116] Gray decoding and phase shift decoding are performed on the re-photographed work target image to obtain a decoding pattern;

[0117] Based on the decoded pattern and the stereo calibration results of the industrial camera, three-dimensional point cloud data is obtained.

[0118] Furthermore, Gray decoding and phase-shift decoding are performed on the re-photographed work target image. Gray decoding is first performed on the re-photographed work target image to obtain an image to be phase-shift decoded.

[0119] The three-step phase shift method is used to obtain the wrapped phase corresponding to two different frequency phase shift codes, and the wrapped phase is input into the phase shift decoding network model. The phase shift decoding network model is sent to the edge server after training on the cloud server.

[0120] The phase-shift decoding network model includes convolutional layers, pooling layers, residual layers, feature fusion layers, upsampling layers, etc. After training, the phase-shift decoding network model can output the fringe order corresponding to the phase of the high-frequency coding pattern package.

[0121] Based on the fringe order corresponding to the wrapped phase, the image to be phase-shifted and decoded is phase-unwrapped to obtain three-dimensional point cloud data.

[0122] It should be noted that the method of obtaining three-dimensional point cloud data through the above encoding and decoding is also the coded structured light method. This method directly projects a full-width coded image and has the characteristics of high measurement efficiency and good robustness.

[0123] Phase-shift encoding uses multiple sinusoidal grating patterns with fixed phase differences to achieve 3D measurements. This method offers significant advantages in phase resolution accuracy, enabling full-resolution results. However, phase-shift encoding requires sampling gratings of different frequencies to perform phase unwrapping to obtain absolute phase values. Existing multi-frequency heterodyne phase unwrapping methods struggle to obtain accurate absolute phase values ​​due to factors such as noise.

[0124] Based on existing research work, this embodiment proposes a method for fast and accurate 3D reconstruction of working targets based on deep learning and coded structured light. The reconstruction process is as follows: Figure 6 As shown, it mainly includes stereo calibration, coding pattern generation, pattern acquisition, pattern decoding, 3D point cloud computing, etc.

[0125] In order to improve the anti-interference ability of phase-shift coding and obtain more accurate absolute phase values, the present invention uses a deep convolutional neural network to perform phase unwrapping. First, a three-step phase shift method is used to obtain the wrapped phases corresponding to two phase-shift codes with different frequencies.

[0126] Then, the wrapped phase is used as input and sent to the constructed convolutional neural network model, which includes convolutional layers, pooling layers, residual layers, feature fusion layers, upsampling layers, etc. After training, the network can output the fringe order corresponding to the wrapped phase of the high-frequency coding pattern, realize fast and accurate phase unfolding of the coding pattern, and improve the accuracy and robustness of phase-shift pattern decoding in industrial production environments.

[0127] Finally, combined with the stereo calibration results, three-dimensional reconstruction of the complex working environment is achieved, and three-dimensional point cloud data of the robot's working target is obtained.

[0128] In some embodiments, obtaining path planning information based on the operation path includes:

[0129] Perform curve fitting on the characteristic points of the operation path to obtain the operation path curve;

[0130] Taking the first-order derivative of the work path curve to obtain the direction vector of the work path, and based on the direction vector and the approach vector of the work path, obtaining the normal vector of the work path;

[0131] Based on the direction vector, approach vector and normal vector, path planning information is obtained.

[0132] Furthermore, curve fitting is performed on the characteristic points of the work path. This involves inputting the characteristic points of the work path into the three-dimensional position model of the work path to obtain the work path curve. The three-dimensional position model of the work path is trained on the cloud server and sent to the edge server.

[0133] It is understandable that after obtaining the three-dimensional point cloud information of the work target, it is necessary to extract the work path point information from the three-dimensional point cloud information of the work target through a point cloud segmentation method.

[0134] like Figure 7 As shown, the present invention proposes a method for extracting a job path based on deep learning, which consists of the following steps:

[0135] Data enhancement: The 3D point cloud data of the work target belongs to a small sample data set. In order to overcome the problem that small sample training data sets easily lead to overfitting of deep neural network models, the 3D point cloud data set of the work target needs to be enhanced.

[0136] This embodiment adopts but is not limited to point cloud data processing methods such as scale change, affine transformation, point cloud normalization, and adding Gaussian noise to achieve operation target point cloud data enhancement, laying the foundation for the training of three-dimensional convolutional neural network.

[0137] Point cloud segmentation: Based on training data enhancement, this embodiment uses a three-dimensional U-Net convolutional neural network to implement operation path extraction.

[0138] The 3D U-Net model is created based on the 2D U-Net and also consists of an encoding part and a decoding part. The encoding part is used to analyze the input 3D point cloud information and perform feature extraction and analysis, while the corresponding decoding part is to generate a segmented block image.

[0139] The encoding part of the network structure includes a cubic convolution layer, a batch normalization layer, and a cubic pooling layer. The decoding part of the network structure includes a cubic upsampling layer, a feature fusion layer, a cubic convolution layer, a batch normalization layer, etc. At the same time, in order to improve segmentation accuracy, a residual block is added to the 3D U-Net network, and an end-to-end point cloud segmentation solution is constructed to achieve accurate segmentation of the work path points in the 3D point cloud data of the work target.

[0140] After the segmentation of the operation path points is completed, the automatic generation of the operation program begins. This embodiment proposes the steps of "operation path planning → operation process parameter generation → operation program generation" to realize the automatic generation of the operation program. The specific technical route is as follows Figure 8 As shown:

[0141] Path Planning: After extracting the path, it's necessary to build a position and posture model of the path to complete the robot's three-dimensional path planning. To ensure smooth and continuous operation and avoid vibration and impact, this embodiment uses a curve fitting algorithm to process discrete path feature points, build an accurate three-dimensional position model of the path, and ensure the continuity and smoothness of the three-dimensional path.

[0142] Curve fitting algorithms include but are not limited to NURBS curves, cubic B-spline curves, Bezier curves, etc. NURBS curve functions have the characteristics of high smoothness. This embodiment uses NURBS curves to establish a position model of a three-dimensional operation path to ensure fitting accuracy and smoothness.

[0143]

[0144] Where B i，3 is the cubic B-spline basis function, W i is the weight factor, D i is the control vertex, and u represents the node of the fitting curve. i，3 The calculation formula is:

[0145]

[0146] Where u k (k=0, 1, 2, ..., n+4) are nodes.

[0147] When an industrial robot is working, the posture of the work target will also have a significant impact on the work quality. Therefore, in this embodiment, a discrete work path sliding coordinate system posture model is established. First, the direction vector of the work path is obtained by calculating the first-order derivative of the NURBS curve:

[0148]

[0149] Where i, j, and k are the unit vectors of the x-axis, y-axis, and z-axis in the robot base coordinate system, respectively.

[0150] Then, get the approach vector of the job path:

[0151]

[0152] The symbol · represents the dot product operator of two vectors, and the vector b(i) is the unit vector of the laser stripe bisector at the i-th sampling point of the spatial curve trajectory.

[0153] Finally, get the normal vector of the working path:

[0154] n(i)=o(i)×a(i)

[0155] At this point, the position model and posture model of the work path are established. After determining the approach vector and normal vector of the work path at a certain moment, the position of the work path at the next moment can be determined, and then the work path planning information can be obtained.

[0156] In some embodiments, the operating method of the industrial robot further includes:

[0157] After the industrial robot starts working, it obtains the depth image and structured light image of the working target;

[0158] Obtaining a first histogram of oriented gradients feature (i.e., HOG feature) based on the depth image, and obtaining a second histogram of oriented gradients feature and a color feature (i.e., CN feature) based on the structured light image;

[0159] Inputting the first directional gradient histogram feature, the second directional gradient histogram feature, and the color feature into the correlation filter to obtain three response maps; wherein the correlation filter is trained on the cloud server and sent to the edge server;

[0160] Based on the three response graphs, the characteristic size and characteristic position of the work target are obtained. Based on the characteristic size and characteristic position of the work target, it is determined that when there is an obstructed workpiece, tracking is stopped, and when it is determined that there is no obstructed workpiece, the normal tracking state is maintained.

[0161] It should be noted that during the operation of industrial robots, strong arc light, splashing, dust, smoke and other noises may be generated, which may interfere with the collected images. Traditional image processing methods based on geometric features are difficult to accurately and robustly extract the feature points of the operation path in a strong noise environment.

[0162] In the field of computer vision target tracking, the target tracking algorithm based on correlation filtering has the characteristics of fast speed and good accuracy and has been widely used. Therefore, this embodiment introduces the correlation filtering algorithm into the extraction of operation path feature points. The correlation filter corresponding to the basic correlation filtering algorithm is given a sample pair (x i ,y i ), solve the N-dimensional filter coefficient w, where x i is the training sample, y i is a Gaussian label. During the training phase, the filter coefficient w is obtained by solving the following optimization problem:

[0163]

[0164] Where the first term represents the difference between the response map and the corresponding Gaussian label, and the second term is the regularization term.

[0165] In the online tracking stage, given the trained parameter w, the filter response map can be calculated based on the input of the current frame, and the position of the maximum response value is the current target position estimate.

[0166] The histogram of oriented gradients is a commonly used feature in correlation filter tracking algorithms. However, tracking algorithms that only utilize a single feature have difficulty processing images that are subject to interference from multiple factors during operation. Because the structured light vision system used in this embodiment can additionally obtain depth information, this embodiment uses multi-feature fusion to enhance the robustness of the correlation filter tracking algorithm.

[0167] This embodiment fuses the directional gradient histogram features and color features of the structured light image and the directional gradient histogram features of the depth image to obtain three different corresponding response images y c1 、y c2 and y c3 .

[0168] In actual operations, a single feature map is often affected by noise, so each feature map should be filtered before fusion. This embodiment adopts a strategy of filtering three feature maps against each other, replacing the traditional method of using Gaussian filters. To obtain the fusion result of the three filtered response maps, the Kullback-Leibler divergence (i.e., KL divergence) is introduced to measure the relationship between each response map distribution and the optimal response map distribution after fusion.

[0169] In addition, considering that traditional correlation filtering algorithms such as MOSSE (i.e., least squares error filter) and KCF (i.e., kernelized correlation filter) use a fixed window size for target tracking, they cannot cope with changes in the feature size of the workpiece during the operation.

[0170] This embodiment uses depth information to guide the change of window size, automatically adapting to changes in workpiece feature size and enhancing the robustness of the algorithm. At the same time, strong arc light will inevitably block the workpiece features during the operation. Therefore, this embodiment adds an occlusion detection link to the operation path tracking process. If the mean change of the depth map exceeds a certain threshold, it is considered that occlusion has occurred. During the occlusion process, the occluder is segmented, and the candidate area of ​​the previous frame image and the occluder area are used to obtain the candidate area of ​​the current frame.

[0171] When the overlap area between the candidate area and the occlusion area is less than a certain threshold, the occlusion is considered to be over and the normal tracking state is restored. The principle diagram of the operation path feature point extraction based on the multi-feature correlation filter tracking algorithm is as follows: Figure 9 As shown in the figure, the algorithm mainly includes a multi-feature correlation filter tracking module, a scale processing module and an occlusion perception processing module. It has scale change capability and occlusion perception processing capability, and can realize accurate and robust extraction of workpiece feature points.

[0172] In some embodiments, the operation method of the industrial robot further includes:

[0173] An artificial potential field of the camera's field of view is established, and when it is determined that the camera is about to leave the field of view based on the artificial potential field value of the camera's field of view, the camera position is adjusted so that the feature points at the edge of the camera's field of view move toward the center of the field of view.

[0174] The complex space operation path tracking control system proposed by the present invention consists of four parts: visual feedback, position adjustment based on field of view constraints, fuzzy controller and control object. Figure 10 As shown, the working tool 1020 is fixedly connected to the bracket 1010 , and the camera 1030 is fixed to the bracket 1010 via a rotating shaft 1040 , and the camera 1030 collects on-site images of the working path 1050 .

[0175] The visual feedback part is responsible for real-time collection of workpiece feature information and extraction of feature points of the operation path. The position adjustment part based on field of view constraints can ensure that the feature points are always in the camera's field of view during trajectory tracking. The fuzzy controller uses the trajectory tracking deviation and its derivative to calculate the adjustment amount of each axis. The control object is the TCP position of the industrial robot (i.e., the terminal control position).

[0176] Because the future direction of complex work paths is uncertain, controllers designed using traditional modeling methods struggle to achieve smooth and precise control. In contrast, skilled workers can rely on their experience to perform the corresponding operations effectively.

[0177] Therefore, this embodiment employs a fuzzy controller for path tracking control, overcoming modeling difficulties while also incorporating workers' actual operational experience into the controller. The fuzzy controller's inputs are the tracking error and error rate of the target feature point, and its output is the adjustment value at the end of the industrial robot. Fuzzification is performed using appropriate membership functions, and a rule base is established based on workers' operational experience. After the fuzzy controller calculates the adjustment value at the end of the industrial robot arm, the adjustment value for each joint of the robot is calculated using the robot's inverse kinematics.

[0178]

[0179] Where P and W are the translation vector and rotation vector of each joint of the robot, e and de are the tracking error and the rate of change of the error, Fuzzy is the fuzzy control function, and T -1 is the homogeneous transformation of the robot's inverse kinematics.

[0180] Due to the irregularity of complex workpieces and the problem of visual advance detection, workpiece feature points may be out of the camera's field of view at locations with large curvature changes. Figure 11As shown, the position point of the TCP of the industrial robot is P1, the feature point of the work target is P2, the field of view of the camera is Φ, and the distance between P2 and the projection point of the camera optical axis is d.

[0181] Since there is a large corner between P1 and P2, d will exceed the camera's field of view, resulting in failure of trajectory tracking. This embodiment proposes an artificial potential field method to establish camera field of view constraints and integrates them into the controller.

[0182] The artificial potential field of the camera field of view is expressed as:

[0183]

[0184] Where U Ф is the artificial potential field, λ is the gain coefficient, d p is the minimum distance from the workpiece feature point to the edge of the image, d o is the distance threshold.

[0185] After establishing the artificial potential field of the camera's field of view, in addition to adjusting the TCP position for deviation correction during the trajectory tracking process, the artificial potential field value also needs to be calculated in real time. When it is determined that the camera is about to leave the field of view, the rotation axis R at the end of the robot can be corrected so that the feature point at the edge of the camera's field of view moves toward the center of the field of view. The correction amount of the rotation axis R can be expressed as Among them, k R is the gain coefficient.

[0186] Finally, it should be noted that the industrial robot operation method provided by this invention can cover common industrial robot applications such as welding, spraying, gluing, grinding, palletizing, and handling. A certain number of modules of this invention can be combined according to actual operational requirements. Depending on the task, welding and grinding robot systems may require path tracking and therefore require all of the aforementioned modules. Systems such as assembly robots, palletizing robots, and spraying robots do not require path tracking and therefore do not require the workpiece feature point extraction module or the path tracking module.

[0187] In summary, the operating method of the industrial robot provided by the present invention obtains an operating target image, and based on the operating target image, obtains three-dimensional point cloud data corresponding to the operating target; based on the three-dimensional point cloud data, obtains operating target posture information; inputs the operating target posture information into the operating path recognition model to obtain the operating path, and obtains path planning information based on the operating path; wherein, the operating path recognition model is obtained by transfer learning the three-dimensional U-Net network model of the cloud server with the preset operating target posture information as a sample and the operating path corresponding to the preset operating target posture information as a sample label; based on the operating path and the operating process target parameters, an operating instruction is obtained, and the industrial robot is controlled to perform the operation based on the operating instruction.

[0188] The operation method of the industrial robot provided by the present invention can be used in an edge server. First, an operation target image is obtained, and three-dimensional point cloud data is obtained based on the operation target image. An operation path recognition model is obtained by performing transfer learning on the three-dimensional U-Net network model of the cloud server. The operation path recognition model is used to process the three-dimensional point cloud data to obtain the operation path, and further obtain path planning information. Then, based on the path planning information and the operation process target parameters, operation instructions are obtained to control the operation robot to perform the operation.

[0189] By collaborating with cloud servers and edge servers, the autonomous operation of industrial robots can be controlled, giving industrial robots powerful autonomous operation capabilities, reducing the workload of robot operators, and alleviating the real problem of labor shortage.

[0190] In addition, the operating method provided by the present invention will greatly improve the operating efficiency and intelligence level of industrial robots, greatly enhance the productivity of industrial robots, and bring great economic benefits to industries such as robot welding, robot handling, robot grinding, robot gluing, robot palletizing, and robot spraying, thereby promoting the rapid development of the manufacturing industry.

[0191] The system architecture based on cloud-edge-end collaboration gives industrial robots powerful autonomous and continuous learning capabilities. Through cloud computing, 5G communication and deep learning technology, the cloud brain of industrial robots continuously learns from data, continuously enhances its own capabilities, and applies the learned and upgraded capabilities in actual operations, greatly improving the adaptability and robustness of industrial robots.

[0192] The operating device of the industrial robot provided by the present invention is described below. The operating device of the industrial robot described below and the operating method of the industrial robot described above can be referred to in correspondence with each other.

[0193] The industrial robot operation device provided by the present invention is applied to edge servers, such as Figure 12As shown, the working device 1200 of the industrial robot includes: a point cloud acquisition module 1210, a posture extraction module 1220, a path planning module 1230 and an instruction generation module 1240.

[0194] The point cloud acquisition module 1210 is used to acquire an image of a work target and obtain three-dimensional point cloud data corresponding to the work target based on the image of the work target.

[0195] The posture extraction module 1220 is used to obtain the posture information of the work target based on the three-dimensional point cloud data.

[0196] The path planning module 1230 is used to input the work target posture information into the work path recognition model to obtain the work path, and obtain path planning information based on the work path; wherein, the work path recognition model is obtained by transfer learning the three-dimensional U-Net network model of the cloud server with the preset work target posture information as a sample and the work path corresponding to the preset work target posture information as a sample label.

[0197] The instruction generation module 1240 is used to obtain operation instructions based on the operation path and operation process target parameters, and control the industrial robot to perform the operation based on the operation instructions.

[0198] In some embodiments, the pose extraction module 1220 includes: a point cloud classification unit, a template selection unit, a template matching unit and a pose extraction unit.

[0199] The point cloud classification unit is used to input three-dimensional point cloud data into the point cloud classification network model to obtain the classification results of the three-dimensional point cloud data; wherein, the point cloud classification network model is a cloud server that uses preset three-dimensional point cloud data as samples and the classification results corresponding to the preset three-dimensional point cloud data as sample labels, which is obtained based on PointNet network training and sent to the edge server.

[0200] The template selection unit is used to select a target two-dimensional image template from a preset two-dimensional image template library and a three-dimensional point cloud template from a preset three-dimensional point cloud template library based on the classification result.

[0201] The template matching unit is used to match the target two-dimensional image template with the three-dimensional point cloud data, and adjust the orientation of the target three-dimensional point cloud template based on the matching result.

[0202] The posture extraction unit is used to obtain the posture information of the operation target based on the target three-dimensional point cloud template after adjusting the orientation.

[0203] In some embodiments, the point cloud acquisition module 1210 includes: an encoding unit, a projection unit, a decoding unit, and a point cloud acquisition unit.

[0204] The encoding unit is used to perform Gray encoding and phase shift encoding on the target image to obtain a coding pattern.

[0205] The projection unit is used to project the coded pattern onto the work target through the industrial robot and obtain the work target image taken again by the industrial camera.

[0206] The decoding unit is used to perform Gray decoding and phase shift decoding on the re-photographed work target image to obtain a decoding pattern.

[0207] The point cloud acquisition unit is used to obtain three-dimensional point cloud data based on the decoded pattern and the stereo calibration result of the industrial camera.

[0208] In some embodiments, the path planning module 1230 includes: a fitting unit, a vector calculation unit, and a path planning unit.

[0209] The fitting unit is used to perform curve fitting on the characteristic points of the operation path to obtain the operation path curve.

[0210] The vector calculation unit is used to perform a first-order derivative of the work path curve to obtain a direction vector of the work path, and to obtain a normal vector of the work path based on the direction vector and the approach vector of the work path.

[0211] The path planning unit is used to obtain path planning information based on the direction vector, the approach vector and the normal vector.

[0212] In some embodiments, the working device 1200 of the industrial robot further includes: an image acquisition module, a feature extraction module, a feature processing module, and a tracking and judgment module.

[0213] The image acquisition module is used to obtain the depth image and structured light image of the working target after the industrial robot starts working.

[0214] The feature extraction module is used to obtain a first directional gradient histogram feature based on the depth image, and to obtain a second directional gradient histogram feature and a color feature based on the structured light image.

[0215] The feature processing module is used to input the first directional gradient histogram feature, the second directional gradient histogram feature and the color feature into the correlation filter to obtain three response maps; wherein, the correlation filter is trained on the cloud server and sent to the edge server.

[0216] The tracking judgment module is used to obtain the characteristic size and characteristic position of the work target based on the three response graphs, and based on the characteristic size and characteristic position of the work target, stop tracking when it is determined that there is an obstructed workpiece, and maintain normal tracking status when it is determined that there is no obstructed workpiece.

[0217] In some embodiments, the operating device 1200 of the industrial robot further includes: a field of view adjustment module.

[0218] The field of view adjustment module is used to establish an artificial potential field of the camera field of view. When it is determined that the camera is about to leave the field of view based on the artificial potential field value of the camera field of view, the camera position is adjusted so that the feature points at the edge of the camera field of view move toward the center of the field of view.

[0219] The electronic device, computer program product, and storage medium provided by the present invention are described below. The electronic device, computer program product, and storage medium described below can be referenced in correspondence with the operating method of the industrial robot described above.

[0220] Figure 13 An example of a physical structure diagram of an electronic device is shown below. Figure 13 As shown, the electronic device may include: a processor 1310, a communication interface 1320, a memory 1330, and a communication bus 1340, wherein the processor 1310, the communication interface 1320, and the memory 1330 communicate with each other via the communication bus 1340. The processor 1310 may call the logic instructions in the memory 1330 to execute the operation method of the industrial robot, which includes:

[0221] Step 110: Acquire an operation target image, and obtain three-dimensional point cloud data corresponding to the operation target based on the operation target image;

[0222] Step 120: Obtaining the work target posture information based on the three-dimensional point cloud data;

[0223] Step 130: Input the work target posture information into the work path recognition model to obtain the work path, and obtain path planning information based on the work path; wherein the work path recognition model is obtained by performing transfer learning on the three-dimensional U-Net network model of the cloud server using the preset work target posture information as a sample and the work path corresponding to the preset work target posture information as a sample label;

[0224] Step 140: Obtain an operation instruction based on the operation path and the operation process target parameters, and control the industrial robot to perform the operation based on the operation instruction.

[0225] In addition, the logic instructions in the above-mentioned memory 1330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0226] On the other hand, the present invention further provides a computer program product, comprising a computer program. The computer program may be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the industrial robot operation method provided by each of the above methods, the method comprising:

[0227] Step 110: Acquire an operation target image, and obtain three-dimensional point cloud data corresponding to the operation target based on the operation target image;

[0228] Step 120: Obtaining the work target posture information based on the three-dimensional point cloud data;

[0229] Step 130: Input the work target posture information into the work path recognition model to obtain the work path, and obtain path planning information based on the work path; wherein the work path recognition model is obtained by performing transfer learning on the three-dimensional U-Net network model of the cloud server using the preset work target posture information as a sample and the work path corresponding to the preset work target posture information as a sample label;

[0230] Step 140: Obtain an operation instruction based on the operation path and the operation process target parameters, and control the industrial robot to perform the operation based on the operation instruction.

[0231] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the operating method of the industrial robot provided by the above methods, the method comprising:

[0232] Step 110: Acquire an operation target image, and obtain three-dimensional point cloud data corresponding to the operation target based on the operation target image;

[0233] Step 120: Obtaining the work target posture information based on the three-dimensional point cloud data;

[0234] Step 130: Input the work target posture information into the work path recognition model to obtain the work path, and obtain path planning information based on the work path; wherein the work path recognition model is obtained by performing transfer learning on the three-dimensional U-Net network model of the cloud server using the preset work target posture information as a sample and the work path corresponding to the preset work target posture information as a sample label;

[0235] Step 140: Obtain an operation instruction based on the operation path and the operation process target parameters, and control the industrial robot to perform the operation based on the operation instruction.

[0236] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0237] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0238] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An industrial robot operation method, characterized in that: include: Acquire an operation target image, and obtain three-dimensional point cloud data corresponding to the operation target based on the operation target image; Based on the three-dimensional point cloud data, obtaining the work target posture information; Inputting the work target pose information into a work path recognition model to obtain a work path, and obtaining path planning information based on the work path; wherein the work path recognition model is obtained by performing transfer learning on a three-dimensional U-Net network model of a cloud server; Obtaining an operation instruction based on the operation path and the operation process target parameters, and controlling the industrial robot to perform the operation based on the operation instruction; The method further comprises: After the industrial robot starts working, acquiring a depth image and a structured light image of the working target; Obtaining a first histogram of oriented gradients feature based on the depth image, and obtaining a second histogram of oriented gradients feature and a color feature based on the structured light image; Inputting the first oriented gradient histogram feature, the second oriented gradient histogram feature, and the color feature into a correlation filter to obtain three response graphs; wherein the correlation filter is trained on the cloud server; Based on the three response diagrams, the characteristic size and characteristic position of the work target are obtained, and based on the characteristic size and characteristic position of the work target, it is determined that when there is an obstructing workpiece, tracking is stopped, and when it is determined that there is no obstructing workpiece, the normal tracking state is maintained.

2. The industrial robot operation method according to claim 1, characterized in that: The step of obtaining the work target posture information based on the three-dimensional point cloud data includes: Inputting the three-dimensional point cloud data into a point cloud classification network model to obtain a classification result of the three-dimensional point cloud data; wherein the point cloud classification network model is trained on the cloud server; Based on the classification result, selecting a target two-dimensional image template from a preset two-dimensional image template library, and selecting a target three-dimensional point cloud template from a preset three-dimensional point cloud template library; Matching the target two-dimensional image template with the three-dimensional point cloud data, and adjusting the orientation of the target three-dimensional point cloud template based on the matching result; Based on the adjusted target 3D point cloud template, the work target posture information is obtained.

3. The operating method of the industrial robot according to claim 1, characterized in that: The step of obtaining three-dimensional point cloud data corresponding to the operation target based on the operation target image includes: Gray coding and phase shift coding are performed on the target image to obtain a coding pattern; Projecting the coded pattern onto a work target by the industrial robot and acquiring an image of the work target taken again by an industrial camera; Gray decoding and phase shift decoding are performed on the re-photographed work target image to obtain a decoding pattern; The three-dimensional point cloud data is obtained based on the decoded pattern and the stereo calibration result of the industrial camera.

4. The operating method of the industrial robot according to claim 1, characterized in that: The obtaining of path planning information based on the operation path includes: Performing curve fitting on the characteristic points of the operation path to obtain an operation path curve; Taking a first-order derivative of the work path curve to obtain a work path direction vector, and obtaining a normal vector of the work path based on the direction vector and an approach vector of the work path; The path planning information is obtained based on the direction vector, the approach vector and the normal vector.

5. The industrial robot operation method according to any one of claims 1 to 4, characterized in that: Also includes: An artificial potential field of the camera field of view is established, and when it is determined based on the artificial potential field value of the camera field of view that the camera is about to leave the field of view, the camera position is adjusted so that the feature points at the edge of the camera field of view move toward the center of the field of view.

6. An industrial robot operating device, characterized in that: include: A point cloud acquisition module is used to acquire an image of a work target and obtain three-dimensional point cloud data corresponding to the work target based on the image of the work target; A posture extraction module is used to obtain the posture information of the work target based on the three-dimensional point cloud data; A path planning module, configured to input the work target pose information into a work path recognition model to obtain a work path, and to obtain path planning information based on the work path; wherein the work path recognition model is obtained by transfer learning of a three-dimensional U-Net network model of a cloud server; An instruction generation module is used to obtain an operation instruction based on the operation path and the operation process target parameters, and control the industrial robot to perform the operation based on the operation instruction; The instruction generation module is further used to: After the industrial robot starts working, acquiring a depth image and a structured light image of the working target; Obtaining a first histogram of oriented gradients feature based on the depth image, and obtaining a second histogram of oriented gradients feature and a color feature based on the structured light image; Inputting the first oriented gradient histogram feature, the second oriented gradient histogram feature, and the color feature into a correlation filter to obtain three response graphs; wherein the correlation filter is trained on the cloud server; Based on the three response diagrams, the characteristic size and characteristic position of the work target are obtained, and based on the characteristic size and characteristic position of the work target, it is determined that when there is an obstructing workpiece, tracking is stopped, and when it is determined that there is no obstructing workpiece, the normal tracking state is maintained.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the operating method of the industrial robot according to any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the operating method of the industrial robot according to any one of claims 1 to 5 are implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the operating method of the industrial robot according to any one of claims 1 to 5 are implemented.

Images