Collaborative robot guidance and positioning method, system, intelligent device and storage medium

Abstract

The present application relates to the field of robotics technology, and specifically to a collaborative robot guidance and positioning method, system, intelligent device and storage medium, which aims to solve the technical problem of how to flexibly, efficiently and conveniently realize the automated operation of the production line. To this end, the present application controls the collaborative robot module to move to a preset photo-taking position, performs image acquisition on the workstation to be operated, and obtains a first image. Based on the trained target detection model, the pixel coordinates of the workstation to be operated are obtained according to the first image. The world coordinates of the workstation to be operated are obtained according to the pixel coordinates, the relative posture between the camera and the collaborative robot, and the workstation height of the workstation to be operated. Controlling the collaborative robot to complete the preset operation according to the world coordinates can improve the accuracy and repeatability of the operation, speed up the production speed, reduce the repeated configuration process and adjustment time, and effectively improve the flexibility and adaptability of the production process.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a collaborative robot guidance and positioning method, system, intelligent device, and storage medium. Background Art

[0002] In modern manufacturing, improving production efficiency and operational precision is a continuous pursuit. Traditional production lines rely on manual operations to complete tasks such as assembly and tightening, which is not only inefficient but also often fails to meet high standards in terms of accuracy and repeatability.

[0003] Although existing automation technologies such as machine vision systems and automated robots are used to improve efficiency, these systems often lack sufficient flexibility and adaptability. Current automation solutions, such as fixed-program robots or basic vision guidance systems, are mostly designed for specific tasks and fixed operating environments. These systems often require cumbersome reconfiguration and lengthy adjustments when faced with the rapid change of production lines or the adaptation needs of diversified products, resulting in reduced production efficiency. In addition, hand-eye coordination technology has poor versatility between different robots or workstations, and requires special calibration and programming for each new task, which not only increases the complexity of operation, but also limits the flexible deployment capabilities of the production line. In addition, existing systems often face synchronization and compatibility issues when integrating different technologies (such as robotic systems, vision systems and PLC control systems).

[0004] Accordingly, the art needs a new collaborative robot guidance and positioning solution to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned defects, the present application is proposed to solve or at least partially solve the technical problem of how to flexibly, efficiently and conveniently realize the automated operation of the production line.

[0006] In a first aspect, a collaborative robot guidance and positioning method is provided, the method comprising:

[0007] The method is applied to an intelligent device, wherein the intelligent device is respectively communicatively connected with a collaborative robot module and a production line control module, wherein the collaborative robot module includes a camera and a collaborative robot;

[0008] The method comprises:

[0009] Based on the demand instruction of the production line control module, a motion instruction is sent to the collaborative robot, the collaborative robot module is controlled to move to a preset photographing position, and the camera is controlled to capture an image of the workstation to be operated corresponding to the photographing position to obtain a first image;

[0010] Based on a preset trained object detection model, obtaining pixel coordinates of the workstation to be operated according to the first image;

[0011] Obtaining the world coordinates of the workstation to be operated according to the pixel coordinates, the pre-stored relative position between the camera and the collaborative robot, and the workstation height of the workstation to be operated;

[0012] A motion instruction is sent to the collaborative robot module according to the world coordinates to control the collaborative robot to move to the workstation to be operated to complete the preset operation.

[0013] In one technical solution of the collaborative robot guidance and positioning method, the method includes training the target detection model according to the following steps:

[0014] Obtain image data samples and construct training and validation datasets;

[0015] Training the target detection model according to the training data set to obtain a pre-trained target detection model;

[0016] The pre-trained target detection model is verified according to the verification data set to obtain the trained target detection model.

[0017] In one technical solution of the collaborative robot guidance and positioning method, verifying the pre-trained target detection model based on the verification data set includes:

[0018] Verifying the pre-trained target detection model based on the verification data set to obtain a verification accuracy indicator;

[0019] Determine whether the verification accuracy index meets the preset requirements;

[0020] If satisfied, the pre-trained target detection model is saved as the trained target detection model;

[0021] If not, the parameters of the target detection model are adjusted, and the step of "training the target detection model according to the training data set" is performed.

[0022] In one technical solution of the collaborative robot guidance and positioning method, the method further includes obtaining the relative position between the camera and the collaborative robot according to the following steps:

[0023] Sending a motion instruction to the collaborative robot to control the collaborative robot to move to a preset position;

[0024] Controlling the camera to capture a preset number of second images of the preset positions;

[0025] The relative position between the camera and the collaborative robot is obtained according to the preset number of second images.

[0026] In one technical solution of the collaborative robot guidance and positioning method, there are multiple preset positions;

[0027] The controlling the camera to collect a preset number of second images at the preset positions includes:

[0028] controlling the camera to capture a second image at the current position, and determining whether the second image reaches a preset number;

[0029] If yes, execute “obtaining the relative position between the camera and the collaborative robot according to the preset number of second images” and control the collaborative robot to move to the initial position;

[0030] If not, the collaborative robot is controlled to move to the next preset position, and the step of "controlling the camera to capture a second image at the current position, and determining whether the second image reaches a preset number" is executed.

[0031] In one technical solution of the collaborative robot guidance and positioning method, the method further includes obtaining the station height of the station to be operated according to the following steps:

[0032] Sending a motion instruction to the collaborative robot to control the collaborative robot to move to a preset photographing position and obtaining the current posture of the collaborative robot;

[0033] Controlling the camera to capture an image of the calibration plate corresponding to the photographing position to obtain a third image;

[0034] Acquire a relative position between the calibration plate and the camera according to the third image;

[0035] The station height of the station to be operated is obtained according to the relative posture between the calibration plate and the camera and the relative posture between the camera and the collaborative robot.

[0036] In one technical solution of the collaborative robot guidance and positioning method, before sending a motion instruction to the collaborative robot, the method further includes:

[0037] Connecting the camera and starting a first thread for controlling the camera;

[0038] Connect the collaborative robot, start the second thread for controlling the collaborative robot, and establish a communication connection with the collaborative robot module.

[0039] In a second aspect, an intelligent device is provided, which includes at least one processor; and a memory communicatively connected to the at least one processor; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method described in any one of the technical solutions of the above-mentioned collaborative robot guidance and positioning method is implemented.

[0040] In a third aspect, a computer-readable storage medium is provided, which stores a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the method described in any one of the technical solutions of the above-mentioned collaborative robot guidance and positioning method.

[0041] In a fourth aspect, a collaborative robot guidance and positioning system is provided, which includes: a collaborative robot module, a production line control module and the intelligent device described in the above-mentioned intelligent device technical solution; the collaborative robot module and the production line control module are respectively communicated with the intelligent device.

[0042] The above one or more technical solutions of this application have at least one or more of the following beneficial effects:

[0043] In implementing the technical solution of the collaborative robot guidance and positioning method provided by this application, this application sends motion instructions to the collaborative robot according to the demand instructions of the production line control module, controls the collaborative robot module to move to a preset photo position, and controls the camera to capture images of the workstation to be operated corresponding to the photo position to obtain a first image. Based on the trained target detection model, the pixel coordinates of the workstation to be operated are obtained according to the first image. According to the pixel coordinates and the pre-stored relative position between the camera and the collaborative robot, and the workstation height of the workstation to be operated, the world coordinates of the workstation to be operated are obtained. According to the world coordinates, motion instructions are sent to the collaborative robot module so that the collaborative robot moves to the workstation to be operated to complete the preset operation. Through the above configuration, this application realizes accurate target detection based on the target detection model to obtain the pixel coordinates of the workstation to be operated, and automatically corrects the pixel coordinates according to the relative position between the camera and the collaborative robot and the workstation height of the workstation to be operated, obtains the world coordinates of the workstation to be operated, and controls the collaborative robot to perform the preset operation according to the world coordinates, which can improve the accuracy and repeatability of the operation and speed up the production speed. The collaborative robot guidance and positioning method of this application can effectively adapt to needs in situations such as production line changes, reducing repeated configuration processes and adjustment time, further improving production efficiency. Furthermore, since it does not require specialized calibration processes and programming for different products or production tasks, it effectively improves the flexibility and adaptability of the production process and can be applied to a variety of different industrial production scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The disclosure of this application will become more easily understood with reference to the accompanying drawings. Those skilled in the art will readily appreciate that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Among them:

[0045] Figure 1 This is a flow chart of the main steps of a collaborative robot guidance and positioning method according to an embodiment of the present application;

[0046] Figure 2 1 is a schematic diagram of the main components of a collaborative robot guidance and positioning system according to an embodiment of the present application;

[0047] Figure 3 This is a flowchart of the main steps for training an object detection model according to an implementation of an embodiment of the present application;

[0048] Figure 4 This is a flow chart of the main steps of obtaining the relative posture between a camera and a collaborative robot according to one embodiment of the present application;

[0049] Figure 5 This is a flow chart of the main steps for obtaining the height of a workstation to be operated according to one embodiment of the present application;

[0050] Figure 6 1 is a flow chart of the main steps of a collaborative robot guidance and positioning method according to an embodiment of the present application;

[0051] Figure 7 It is a schematic diagram of the main structure of a smart device according to an embodiment of the present application.

[0052] Reference numerals:

[0053] 11: Memory; 12: Processor. DETAILED DESCRIPTION

[0054] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the scope of protection of the present application.

[0055] In the description of this application, "module" and "processor" may include hardware, software, or a combination of both. A module may include hardware circuitry, various suitable sensors, communication ports, and memory. It may also include software components, such as program code, or a combination of software and hardware. A processor may be a central processing unit (CPU), a microprocessor, an image processor, a digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor may be implemented in software, hardware, or a combination of both. Computer-readable storage media include any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" refers to all possible combinations of A and B, such as only A, only B, or both A and B. The terms "at least one of A or B" or "at least one of A and B" have similar meanings to "A and / or B" and may include only A, only B, or both A and B. The singular forms "a" and "the" may also include the plural forms.

[0056] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, and based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0057] The user personal information processed by this application will vary depending on the specific product / service scenario and must be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. This application will treat the user's personal information and its processing with a high degree of diligence.

[0058] This application attaches great importance to the security of user personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0059] See attached Figure 1 , Figure 1 This is a flow chart of the main steps of the collaborative robot guidance and positioning method according to an embodiment of the present application. Figure 1 As shown, the collaborative robot guidance and positioning method in the embodiment of the present application is applied to an intelligent device, which is respectively connected to a collaborative robot module and a production line control module. The collaborative robot module includes a camera and a collaborative robot. The collaborative robot guidance and positioning method mainly includes the following steps S101 to S104.

[0060] Step S101: Based on the demand instructions of the production line control module, a motion instruction is sent to the collaborative robot, the collaborative robot module is controlled to move to a preset photographing position, and the camera is controlled to capture an image of the workstation to be operated corresponding to the photographing position to obtain a first image.

[0061] In this embodiment, motion instructions can be sent to the collaborative robot according to the demand instructions of the production line control model, so that the collaborative robot module moves to a preset photographing position, captures images of the operating station, and obtains a first image.

[0062] In one embodiment, the production line control module may be a module based on a Programmable Logic Controller (PLC).

[0063] In one embodiment, before sending motion instructions to the collaborative robot, the smart device can be controlled to establish a communication connection with the camera and start controlling the first thread of the camera; and the smart device can be controlled to establish a communication connection with the collaborative robot module and start the second thread of controlling the collaborative robot.

[0064] In one embodiment, before sending motion instructions to the collaborative robot, a communication connection between the intelligent device and the production line control module may be established.

[0065] Step S102: Based on a preset trained target detection model, the pixel coordinates of the workstation to be operated are obtained according to the first image.

[0066] In this embodiment, the target detection model can be trained to obtain a trained target detection model, and the trained target detection model can be used to detect the first image to obtain the pixel coordinates of the workstation to be operated, wherein the pixel coordinates are coordinates in the camera coordinate system.

[0067] In one embodiment, the target detection model can be a model obtained by combining an attention mechanism module and a depth-wise separable convolutional network.

[0068] Step S103: Obtain the world coordinates of the workstation to be operated according to the pixel coordinates, the pre-stored relative posture between the camera and the collaborative robot, and the workstation height of the workstation to be operated.

[0069] In this embodiment, coordinate transformation can be performed based on the pixel coordinates and the pre-stored relative position between the camera and the collaborative robot and the height of the workstation to be operated, thereby obtaining the world coordinates of the workstation to be operated. Wherein, the world coordinates are coordinates in the world coordinate system.

[0070] In one embodiment, the pixel coordinates may be transformed by combining the camera intrinsic parameters, the relative position between the camera and the collaborative robot, and the height of the workstation to be operated to obtain the world coordinates.

[0071] In one embodiment, the intelligent device may be a host computer, which is a computer that can send control commands. The trained object detection model, the relative position between the camera and the collaborative robot, and the height of the workstation to be operated can be stored in the host computer.

[0072] Step S104: Send motion instructions to the collaborative robot module according to the world coordinates to control the collaborative robot to move to the operating station to complete the preset operation.

[0073] In this embodiment, motion instructions can be sent to the collaborative robot based on the world coordinates to control the collaborative robot to move to the workstation to be operated to complete the preset operation.

[0074] In one embodiment, the intelligent device can communicate with the collaborative robot module and the production line control module respectively through the TCP (Transmission Control Protocol) communication protocol. The collaborative robot module is used to control the precise movement of the collaborative robot; the production line control module is used to provide real-time production line signals to generate demand instructions; the intelligent device is responsible for triggering the camera to acquire images, running the target detection model, processing and transmitting signals, and integrating the logic of the collaborative robot guidance positioning process, thereby achieving seamless connection between the intelligent device, the collaborative robot module and the production line control module, which not only ensures the synchronization and accuracy of the operation process, but also significantly improves flexibility and adaptability. When the collaborative robot module or the production line control module changes, communication can be achieved by modifying the corresponding network connection address. There is no need to modify the host computer configuration, which can reduce debugging costs.

[0075] In one embodiment, the preset operation may be a bolt tightening operation.

[0076] Based on the method described in steps S101 to S104 above, the embodiment of the present application sends motion instructions to the collaborative robot according to the demand instructions of the production line control module, controls the collaborative robot module to move to a preset photo position, and controls the camera to capture images of the workstation to be operated corresponding to the photo position to obtain a first image. Based on the trained target detection model, the pixel coordinates of the workstation to be operated are obtained according to the first image. According to the pixel coordinates and the pre-stored relative position between the camera and the collaborative robot, and the workstation height of the workstation to be operated, the world coordinates of the workstation to be operated are obtained. According to the world coordinates, motion instructions are sent to the collaborative robot module so that the collaborative robot moves to the workstation to be operated and completes the preset operation. Through the above configuration, the embodiment of the present application realizes accurate target detection based on the target detection model to obtain the pixel coordinates of the workstation to be operated, and automatically corrects the pixel coordinates according to the relative position between the camera and the collaborative robot and the workstation height of the workstation to be operated, obtains the world coordinates of the workstation to be operated, and controls the collaborative robot to perform the preset operation according to the world coordinates, which can improve the accuracy and repeatability of the operation and speed up production. The collaborative robot guidance and positioning method of the present application can also effectively adapt to production line changes, reducing reconfiguration and adjustment time, further improving production efficiency. Furthermore, since it eliminates the need for specialized calibration and programming for different products or production tasks, it effectively increases the flexibility and adaptability of the production process, enabling its application in a variety of different industrial production scenarios.

[0077] The following further describes the training process of the target detection model, the specific process of obtaining the relative posture between the camera and the collaborative robot, and the specific process of obtaining the height of the workstation to be operated.

[0078] In one implementation of the embodiment of the present application, the target detection model may be trained according to the following steps S201 to S203:

[0079] Step S201: Obtain image data samples and construct a training data set and a verification data set.

[0080] In this embodiment, a certain number of image data samples may be obtained, and the image data samples may be labeled. The labeled image data samples may be separated to construct a training data set and a verification data set, respectively.

[0081] Step S202: Train the target detection model according to the training data set to obtain a pre-trained target detection model.

[0082] In this embodiment, the target detection model can be trained according to the training data set to obtain a trained target detection model.

[0083] In one embodiment, the training process can use the pytorch platform to train the target detection model using GPU accelerated training.

[0084] Step S203: Verify the pre-trained target detection model based on the verification data set to obtain a trained target detection model.

[0085] In this embodiment, step S203 may further include the following steps S2031 to S2034:

[0086] Step S2031: Validate the pre-trained target detection model based on the validation data set to obtain a validation accuracy indicator.

[0087] Step S2032: Determine whether the verification accuracy index meets the preset requirements; if so, execute step S2033; if not, execute step S2034.

[0088] Step S2033: Save the pre-trained target detection model as a trained target detection model.

[0089] Step S2034: After adjusting the parameters of the target detection model, execute step S202.

[0090] In this embodiment, the pre-trained target detection model can be verified based on the validation data set to determine the validation accuracy index of the target detection model. If the target is met (i.e., the preset requirements are met), the pre-trained target detection model can be saved as the trained target detection model. If the target is not met, the parameters of the target detection model can be adjusted, such as adjusting the learning rate, optimizer, etc. of the target detection model, and then the target detection model can be trained.

[0091] In one embodiment, mAP5.0 can be used as a validation accuracy metric. mAP5.0 refers to the mean average precision at IoU 0.5. IoU refers to the intersection over union (IoU), and 0.5 is the evaluation window size.

[0092] In one embodiment, the saved model format of the trained target detection model can be ONNX (Open Neural Network Exchange, an open source framework for representing deep learning models), which can enable the deployment of the trained target detection model on different platforms, ensuring the flexibility of the collaborative robot's guided positioning process.

[0093] In one embodiment, please refer to the attached Figure 3 , Figure 3FIG. 1 is a flow chart of the main steps of training a target detection model according to an embodiment of the present application. Figure 3 As shown, the target detection model can be trained according to the following steps S301 to S307:

[0094] Step S301: Collect image data samples.

[0095] Step S302: Dataset (including training dataset and validation dataset) is generated.

[0096] Step S303: Target detection model training and verification.

[0097] Step S304: Accuracy test.

[0098] Step S305: Determine whether the verification accuracy index meets the preset requirements; if so, execute step S306; if not, execute step S307.

[0099] Step S306: Save the model as a trained target detection model and then end.

[0100] Step S307: After adjusting the model parameters, execute step S303.

[0101] In one implementation of the embodiment of the present application, the relative position between the camera and the collaborative robot can be obtained according to the following steps S401 to S403.

[0102] Step S401: Send a motion instruction to the collaborative robot to control the collaborative robot to move to a preset position.

[0103] In this embodiment, the collaborative robot can be controlled to move to a preset position.

[0104] Step S402: controlling the camera to capture a preset number of second images at preset positions.

[0105] In this embodiment, there may be multiple preset positions for the second image, and there may be multiple second images. Step S402 may further include the following steps S4021 to S4022:

[0106] Step S4021: Control the camera to capture a second image at the current position, and determine whether the number of second images reaches a preset number; if so, execute step S403; if not, execute step S4022.

[0107] Step S4022: After controlling the collaborative robot to move to the next preset position, execute step S4021.

[0108] Step S403: Acquire the relative position between the camera and the collaborative robot based on a preset number of second images.

[0109] In this embodiment, a second image of the collaborative robot in different postures can be automatically captured, and the posture solution is performed based on the second image to obtain the camera internal parameters and the relative posture between the camera and the collaborative robot (i.e., the hand-eye relationship). The process of obtaining the relative posture does not require human intervention, and the precise correspondence between the camera and the collaborative robot can be obtained, which significantly reduces the operating cost. At the same time, due to the high efficiency of the process of obtaining the relative posture, it can be implemented in different collaborative robot modules, thereby ensuring that the collaborative robot modules can be quickly adapted to new production tasks or different workstations, which can improve the flexibility and response speed of the production line.

[0110] In one embodiment, after a preset number of second images are captured, the collaborative robot may be controlled to move to an initial position.

[0111] In one embodiment, please refer to the attached Figure 4 , Figure 4 FIG. 1 is a flow chart of the main steps of obtaining the relative posture between the camera and the collaborative robot according to an embodiment of the present application. Figure 4 As shown, the relative position between the camera and the collaborative robot can be obtained according to the following steps S501 to S513:

[0112] Step S501: Connect the camera and execute steps S502 and S503

[0113] Step S502: After starting the first thread for controlling the camera, step S509 is executed through the first thread.

[0114] Step S503: Connect to the collaborative robot and execute steps S504 and S505.

[0115] Step S504: After starting the second thread for controlling the collaborative robot, execute step S506.

[0116] Step S505: After establishing a communication connection between the robot server and the robot client through TCP communication, step S507 is executed through the second thread.

[0117] Step S506: Sending motion instructions to the robot through the second thread.

[0118] Step S507: Receive location information.

[0119] Step S508: Determine whether the collaborative robot has moved to the specified position; if so, execute step S509; if not, execute step S513.

[0120] Step S509: Acquire a second image.

[0121] Step S510: Determine whether the number of second image acquisitions reaches a preset number; if so, execute step S511; if not, execute step S506.

[0122] Step S511: Calculate the relative pose between the camera and the collaborative robot based on the second image.

[0123] Step S512: Save the result.

[0124] Step S513: Alarm check.

[0125] In one embodiment of the application, the height of the workstation to be operated can be obtained according to the following steps S601 to S604:

[0126] Step S601: Send a motion instruction to the collaborative robot to control the collaborative robot to move to a preset photo taking position and obtain the current posture of the collaborative robot.

[0127] Step S602: Control the camera to capture an image of the calibration plate corresponding to the photographing position to obtain a third image.

[0128] Step S603: Obtain the relative pose between the calibration plate and the camera according to the third image.

[0129] Step S604: Obtain the height of the workstation to be operated based on the relative posture between the calibration plate and the camera and the relative posture between the camera and the collaborative robot.

[0130] In this embodiment, a calibration plate can be placed on the workstation to be operated to measure its height. The calibration plate is photographed with a camera and then solved using an image processing algorithm to determine the relative pose between the calibration plate and the camera. The height of the workstation to be operated is then calculated based on the relative pose between the camera and the collaborative robot. This can be combined with an image processing algorithm to accurately measure the height of the workstation to be operated, optimizing the collaborative robot's operational accuracy and efficiency. This not only reduces hardware costs but also improves the applicability and flexibility of the collaborative robot's guided positioning process.

[0131] In one embodiment, please refer to the attached Figure 5 , Figure 5 This is a flow chart of the main steps of obtaining the height of the workstation to be operated according to an embodiment of the present application. Figure 5 As shown, the height of the workstation to be operated can be obtained through the following steps S701 to S705:

[0132] Step S701: The collaborative robot moves to a preset photo taking position.

[0133] Step S702: Acquire a third image.

[0134] Step S703: Calculate the camera external parameters, that is, the position and posture of the calibration plate in the camera coordinate system in the current state.

[0135] Step S704: Solve the transformation relationship, that is, use the relative posture between the camera and the collaborative robot and the posture of the calibration plate in the camera coordinate system to obtain the transformation relationship.

[0136] Step S705: coordinate transformation, that is, calculating the station height of the station to be operated according to the transformation relationship, thereby achieving 3D positioning of the station to be operated.

[0137] In one embodiment, please refer to the attached Figure 6 , Figure 6 FIG. 1 is a flow chart of the main steps of a collaborative robot guidance and positioning method according to an embodiment of the present application. Figure 6 As shown, the collaborative robot is used to perform a bolt tightening operation, and the collaborative robot guidance and positioning method may include the following steps S801 to S813:

[0138] Step S801: Connect the camera and execute steps S802 and S803.

[0139] Step S802: triggering and starting the first thread for controlling the camera, and executing step S809.

[0140] Step S803: Connect the collaborative robot and execute steps S804 and S805.

[0141] Step S804: Trigger and start the second thread for controlling the collaborative robot, and execute steps S808 and S812.

[0142] Step S805: Connect the production line control module and execute steps S806 and S807.

[0143] Step S806: After triggering and starting the production line control module communication thread, execute step S807.

[0144] Step S807: Receive work instructions.

[0145] Step S808: The collaborative robot moves to a preset photo-taking position.

[0146] Step S809: Acquire a first image.

[0147] Step S810: Target detection model inference to obtain the pixel coordinates of the bolt.

[0148] Step S811: Coordinate transformation to obtain world coordinates. That is, coordinate transformation is performed based on the camera intrinsic parameters, the relative position between the camera and the collaborative robot, and the height of the workstation where the bolt is to be operated, to obtain the world coordinates of the bolt.

[0149] Step S812: Control the collaborative robot to move to the workstation to be operated according to the world coordinates.

[0150] Step S813: Complete the tightening operation.

[0151] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of the present application, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application, and therefore will also fall within the scope of protection of this application.

[0152] It will be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.

[0153] Another aspect of the present application provides a computer-readable storage medium.

[0154] In an embodiment of a computer-readable storage medium according to the present application, the computer-readable storage medium can be configured to store a program for executing the collaborative robot guidance and positioning method of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned collaborative robot guidance and positioning method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-transitory computer-readable storage medium.

[0155] Another aspect of the present application provides a smart device.

[0156] In an embodiment of a smart device according to the present application, the smart device may include at least one processor; and a memory in communication with the at least one processor; wherein the memory stores a computer program, and when the computer program is executed by the at least one processor, the method described in any of the above embodiments is implemented. Figure 7 , Figure 7 exemplarily shows that the memory 11 and the processor 12 are communicatively connected via a bus.

[0157] The electronic device described in this application may be, but is not limited to, a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, a vehicle-mounted device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), an augmented reality (AR) or virtual reality (VR) device, etc., and the embodiments of this application are not limited to this.

[0158] Furthermore, the present application also provides a collaborative robot guidance and positioning system.

[0159] In an embodiment of a collaborative robot guidance and positioning system of the present application, the collaborative robot guidance and positioning system may include a collaborative robot module, a production line control module, and the smart device in the above-mentioned smart device embodiment. The collaborative robot module and the production line control module are respectively connected to the smart device for communication.

[0160] In one embodiment, the smart device may be a host computer. Figure 2 ,like Figure 2 As shown, the collaborative robot guidance and positioning system may include a collaborative robot module, a host computer and a production line control module. The collaborative robot module and the production line control module are respectively connected to the host computer via the TCP communication protocol. The collaborative robot module may include a collaborative robot and a camera. The host computer includes a camera trigger unit, an identification and positioning unit, an integrated communication unit and a trajectory planning unit. Through the above units, target detection, automatic hand-eye calibration (i.e., obtaining the relative posture between the camera and the collaborative robot), 3D positioning (i.e., obtaining the height of the workstation to be operated), integrated communication and other functions can be realized, thereby ultimately realizing the preset operation of the workstation to be operated (e.g., tightening operation).

[0161] Thus far, the technical solution of the present application has been described in conjunction with an embodiment shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present application.

Claims

1. A collaborative robot guidance and positioning method, characterized in that: The method is applied to an intelligent device, wherein the intelligent device is respectively communicatively connected with a collaborative robot module and a production line control module, wherein the collaborative robot module includes a camera and a collaborative robot; The method comprises: Based on the demand instruction of the production line control module, a motion instruction is sent to the collaborative robot, the collaborative robot module is controlled to move to a preset photographing position, and the camera is controlled to capture an image of the workstation to be operated corresponding to the photographing position to obtain a first image; Based on a preset trained object detection model, obtaining pixel coordinates of the workstation to be operated according to the first image; Obtaining the world coordinates of the workstation to be operated according to the pixel coordinates, the pre-stored relative posture between the camera and the collaborative robot, and the workstation height of the workstation to be operated; A motion instruction is sent to the collaborative robot module according to the world coordinates to control the collaborative robot to move to the workstation to be operated to complete the preset operation.

2. The collaborative robot guidance and positioning method according to claim 1, characterized in that: The method comprises training the object detection model according to the following steps: Obtain image data samples and construct training and validation datasets; Training the target detection model according to the training data set to obtain a pre-trained target detection model; The pre-trained target detection model is verified according to the verification data set to obtain the trained target detection model.

3. The collaborative robot guidance and positioning method according to claim 2, characterized in that: The verifying the pre-trained object detection model according to the verification data set includes: Verifying the pre-trained target detection model based on the verification data set to obtain a verification accuracy indicator; Determine whether the verification accuracy index meets the preset requirements; If satisfied, the pre-trained target detection model is saved as the trained target detection model; If not, the parameters of the target detection model are adjusted, and the step of "training the target detection model according to the training data set" is performed.

4. The collaborative robot guidance and positioning method according to claim 1, characterized in that: The method further includes obtaining a relative pose between the camera and the collaborative robot according to the following steps: Sending a motion instruction to the collaborative robot to control the collaborative robot to move to a preset position; Controlling the camera to capture a preset number of second images of the preset positions; The relative position between the camera and the collaborative robot is obtained according to the preset number of second images.

5. The collaborative robot guidance and positioning method according to claim 4, characterized in that: There are multiple preset positions; The controlling the camera to collect a preset number of second images at the preset positions includes: controlling the camera to capture a second image at the current position, and determining whether the second image reaches a preset number; If yes, execute “obtaining the relative position between the camera and the collaborative robot according to the preset number of second images” and control the collaborative robot to move to the initial position; If not, the collaborative robot is controlled to move to the next preset position, and the step of "controlling the camera to capture a second image at the current position, and determining whether the second image reaches a preset number" is executed.

6. The collaborative robot guidance and positioning method according to claim 1, characterized in that: The method further comprises obtaining the station height of the station to be operated according to the following steps: Sending a motion instruction to the collaborative robot to control the collaborative robot to move to a preset photographing position and obtaining the current posture of the collaborative robot; Controlling the camera to capture an image of the calibration plate corresponding to the photographing position to obtain a third image; Acquire a relative position between the calibration plate and the camera according to the third image; The station height of the station to be operated is obtained according to the relative posture between the calibration plate and the camera and the relative posture between the camera and the collaborative robot.

7. The collaborative robot guidance and positioning method according to any one of claims 1 to 6, characterized in that: Before sending the motion instruction to the collaborative robot, the method further includes: Connecting the camera and starting a first thread for controlling the camera; Connect the collaborative robot, start the second thread for controlling the collaborative robot, and establish a communication connection with the collaborative robot module.

8. A smart device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the collaborative robot guidance and positioning method according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the collaborative robot guidance and positioning method according to any one of claims 1 to 7.

10. A collaborative robot guidance and positioning system, characterized in that: The system includes: a collaborative robot module, a production line control module and the intelligent device according to claim 8; the collaborative robot module and the production line control module are respectively communicatively connected to the intelligent device.

Images