Collaborative robot guiding and positioning method and system, intelligent equipment and storage medium

Abstract

The invention relates to the technical field of robots, in particular to a collaborative robot guiding and positioning method and system, intelligent equipment and a storage medium, and aims to solve the technical problem of how to flexibly, efficiently and conveniently realize automatic operation of a production line. In order to achieve the purpose, a collaborative robot module is controlled to move to a preset photographing position, image acquisition is carried out on a to-be-operated station, and a first image is obtained. And based on a trained target detection model, obtaining pixel coordinates of the to-be-operated station according to the first image. And according to the pixel coordinates, the relative pose between the camera and the collaborative robot and the station height of the to-be-operated station, the world coordinates of the to-be-operated station are obtained. The collaborative robot is controlled to complete the preset operation according to the world coordinates, the operation precision and repeatability can be improved, the production speed is increased, the repeated configuration process and adjustment time are reduced, and the flexibility and adaptability of the production process are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a collaborative robot guidance and positioning method, system, intelligent device and storage medium. Background Art

[0002] In modern manufacturing, improving production efficiency and operating accuracy is a continuous pursuit. Traditional production lines rely on manual operations to complete tasks such as assembly and tightening, which is not only inefficient, but also often difficult to meet high standards in terms of accuracy and repeatability.

[0003] Although existing automation technologies such as machine vision systems and automated robots are used to improve efficiency, these systems often lack sufficient flexibility and adaptability. Current automation solutions, such as fixed-program robots or basic vision-guided systems, are mostly designed for specific tasks and fixed operating environments. These systems often require cumbersome reconfiguration and long adjustments when faced with the rapid change of production lines or the adaptation needs of diversified products, resulting in reduced production efficiency. In addition, hand-eye coordination technology has poor versatility among different robots or workstations, and requires special calibration and programming for each new task, which not only increases the complexity of operation, but also limits the flexible deployment capabilities of production lines. In addition, existing systems often face synchronization and compatibility issues when integrating different technologies (such as robotic systems, vision systems and PLC control systems).

[0004] Accordingly, the art needs a new collaborative robot guidance and positioning solution to solve the above problems. Summary of the invention

[0005] In order to overcome the above-mentioned defects, the present application is proposed to solve or at least partially solve the technical problem of how to flexibly, efficiently and conveniently realize the automated operation of the production line.

[0006] In a first aspect, a collaborative robot guidance and positioning method is provided, the method comprising:

[0007] The method is applied to an intelligent device, wherein the intelligent device is respectively connected to a collaborative robot module and a production line control module, wherein the collaborative robot module includes a camera and a collaborative robot;

[0008] The method comprises:

[0009] Based on the demand instruction of the production line control module, a motion instruction is sent to the collaborative robot, the collaborative robot module is controlled to move to a preset photographing position, and the camera is controlled to collect an image of the workstation to be operated corresponding to the photographing position to obtain a first image;

[0010] Based on a preset trained target detection model, the pixel coordinates of the workstation to be operated are acquired according to the first image;

[0011] Acquire the world coordinates of the workstation to be operated according to the pixel coordinates, the pre-stored relative position between the camera and the collaborative robot, and the workstation height of the workstation to be operated;

[0012] Send motion instructions to the collaborative robot module according to the world coordinates to control the collaborative robot to move to the waiting operation station to complete the preset operation.

[0013] In a technical solution of the collaborative robot guidance and positioning method, the method includes training the target detection model according to the following steps:

[0014] Obtain image data samples and construct training and verification data sets;

[0015] Training the target detection model according to the training data set to obtain a pre-trained target detection model;

[0016] The pre-trained target detection model is verified according to the verification data set to obtain the trained target detection model.

[0017] In a technical solution of the collaborative robot guidance and positioning method, the pre-trained target detection model is verified according to the verification data set, including:

[0018] According to the verification data set, the pre-trained target detection model is verified to obtain a verification accuracy index;

[0019] Determine whether the verification accuracy index meets the preset requirements;

[0020] If satisfied, the pre-trained target detection model is saved as a trained target detection model;

[0021] If not, the parameters of the target detection model are adjusted, and the step of "training the target detection model according to the training data set" is performed.

[0022] In a technical solution of the collaborative robot guidance and positioning method, the method further includes obtaining the relative position between the camera and the collaborative robot according to the following steps:

[0023] Sending a motion instruction to the collaborative robot to control the collaborative robot to move to a preset position;

[0024] Controlling the camera to collect a preset number of second images of the preset positions;

[0025] According to the preset number of second images, the relative position between the camera and the collaborative robot is acquired.

[0026] In a technical solution of the above collaborative robot guidance and positioning method, there are multiple preset positions;

[0027] The controlling the camera to collect a preset number of second images of the preset positions includes:

[0028] Controlling the camera to capture a second image at the current position, and determining whether the second image reaches a preset number;

[0029] If yes, then execute “obtaining the relative position between the camera and the collaborative robot according to the preset number of second images”, and control the collaborative robot to move to the initial position;

[0030] If not, the collaborative robot is controlled to move to the next preset position, and the step of "controlling the camera to capture a second image at the current position, and determining whether the second image reaches a preset number" is executed.

[0031] In a technical solution of the collaborative robot guidance and positioning method, the method further includes obtaining the station height of the station to be operated according to the following steps:

[0032] Sending a motion instruction to the collaborative robot to control the collaborative robot to move to a preset photographing position, and obtaining the current position and posture of the collaborative robot;

[0033] Controlling the camera to collect an image of the calibration plate corresponding to the photographing position to obtain a third image;

[0034] According to the third image, obtaining a relative posture between the calibration plate and the camera;

[0035] The station height of the station to be operated is obtained according to the relative posture between the calibration plate and the camera and the relative posture between the camera and the collaborative robot.

[0036] In a technical solution of the collaborative robot guidance and positioning method, before sending a motion instruction to the collaborative robot, the method further includes:

[0037] Connecting the camera and starting a first thread for controlling the camera;

[0038] Connect the collaborative robot, start a second thread for controlling the collaborative robot, and establish a communication connection with the collaborative robot module.

[0039] In a second aspect, an intelligent device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method described in any one of the technical solutions of the above-mentioned collaborative robot guidance and positioning method is implemented.

[0040] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, wherein the program codes are suitable for being loaded and run by a processor to execute the method described in any one of the technical solutions of the above-mentioned collaborative robot guidance and positioning method.

[0041] In a fourth aspect, a collaborative robot guidance and positioning system is provided, the system comprising: a collaborative robot module, a production line control module and the intelligent device described in the above-mentioned intelligent device technical solution; the collaborative robot module and the production line control module are respectively communicated with the intelligent device.

[0042] The above one or more technical solutions of the present application have at least one or more of the following beneficial effects:

[0043] In the implementation of the technical solution of the collaborative robot guidance and positioning method provided by the present application, the present application sends motion instructions to the collaborative robot according to the demand instructions of the production line control module, controls the collaborative robot module to move to the preset photo position, and controls the camera to collect images of the workstation to be operated corresponding to the photo position to obtain a first image. Based on the trained target detection model, the pixel coordinates of the workstation to be operated are obtained according to the first image. According to the pixel coordinates and the relative posture between the pre-stored camera and the collaborative robot, the station height of the workstation to be operated, the world coordinates of the workstation to be operated are obtained. According to the world coordinates, motion instructions are sent to the collaborative robot module so that the collaborative robot moves to the workstation to be operated to complete the preset operation. Through the above configuration method, the present application realizes accurate target detection based on the target detection model to obtain the pixel coordinates of the workstation to be operated, and automatically corrects the pixel coordinates according to the relative posture between the camera and the collaborative robot and the station height of the workstation to be operated, obtains the world coordinates of the workstation to be operated, and controls the collaborative robot to perform preset operations according to the world coordinates, which can improve the accuracy and repeatability of the operation and speed up the production speed. At the same time, the collaborative robot guidance and positioning method of the present application can effectively adapt to the needs in the case of production line replacement, reduce repeated configuration processes and adjustment time, and further improve production efficiency. In addition, since there is no need to conduct special calibration processes and program writing for different products or production tasks, it also effectively improves the flexibility and adaptability of the production process and can be applied to a variety of different industrial production scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The disclosure of the present application will become easier to understand with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present application. Among them:

[0045] Figure 1 It is a schematic flow chart of the main steps of a collaborative robot guidance and positioning method according to an embodiment of the present application;

[0046] Figure 2 It is a schematic diagram of the main components of a collaborative robot guidance and positioning system according to an implementation of an embodiment of the present application;

[0047] Figure 3 It is a schematic diagram of the main steps of training a target detection model according to an implementation of an embodiment of the present application;

[0048] Figure 4 It is a schematic flow chart of the main steps of obtaining the relative posture between a camera and a collaborative robot according to an implementation of an embodiment of the present application;

[0049] Figure 5 It is a schematic flow chart of the main steps of obtaining the station height of the station to be operated according to an implementation of an embodiment of the present application;

[0050] Figure 6 It is a schematic flow chart of the main steps of a collaborative robot guidance and positioning method according to an implementation of an embodiment of the present application;

[0051] Figure 7 It is a schematic diagram of the main structure of a smart device according to an embodiment of the present application.

[0052] Reference numerals:

[0053] 11: memory; 12: processor. DETAILED DESCRIPTION

[0054] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0055] In the description of the present application, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, memory, and may also include software parts, such as program code, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware or a combination of the two. Computer-readable storage media include any suitable medium that can store program code, such as a disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B or A and B. The term "at least one A or B" or "at least one of A and B" has a similar meaning to "A and / or B" and may include only A, only B or A and B. The singular terms "one" and "the" may also include plural forms.

[0056] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0057] The user personal information processed by this application will vary depending on the specific product / service scenario, and shall be based on the specific scenario in which the user uses the product / service, and may involve the user's account information, device information, driving information, vehicle information or other related information. This application will treat the user's personal information and its processing with a high degree of diligence.

[0058] This application attaches great importance to the security of user personal information and has taken reasonable and feasible security protection measures that meet industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0059] See attached Figure 1 , Figure 1 FIG. 1 is a schematic flow chart of the main steps of the collaborative robot guidance and positioning method according to an embodiment of the present application. Figure 1 As shown, the collaborative robot guidance and positioning method in the embodiment of the present application is applied to an intelligent device, and the intelligent device is respectively connected to the collaborative robot module and the production line control module, and the collaborative robot module includes a camera and a collaborative robot. The collaborative robot guidance and positioning method mainly includes the following steps S101 to S104.

[0060] Step S101: Based on the demand instruction of the production line control module, a motion instruction is sent to the collaborative robot, the collaborative robot module is controlled to move to a preset photographing position, and the camera is controlled to collect an image of the workstation to be operated corresponding to the photographing position to obtain a first image.

[0061] In this embodiment, motion instructions can be sent to the collaborative robot according to the demand instructions of the production line control model, so that the collaborative robot module moves to a preset photographing position, captures images of the operating station, and obtains a first image.

[0062] In one embodiment, the production line control module may be a module based on a PLC (Programmable Logic Controller).

[0063] In one embodiment, before sending motion instructions to the collaborative robot, the smart device can be controlled to establish a communication connection with the camera and start controlling the first thread of the camera; and the smart device can be controlled to establish a communication connection with the collaborative robot module and start a second thread of controlling the collaborative robot.

[0064] In one embodiment, before sending motion instructions to the collaborative robot, a communication connection between the intelligent device and the production line control module may also be established.

[0065] Step S102: Based on the preset trained target detection model, the pixel coordinates of the workstation to be operated are obtained according to the first image.

[0066] In this embodiment, the target detection model can be trained to obtain a trained target detection model, and the trained target detection model is used to detect the first image to obtain the pixel coordinates of the workstation to be operated, wherein the pixel coordinates are coordinates in the camera coordinate system.

[0067] In one implementation, the target detection model may be a model based on a combination of an attention mechanism module and a deep separable convolutional network.

[0068] Step S103: Obtain the world coordinates of the workstation to be operated according to the pixel coordinates, the pre-stored relative posture between the camera and the collaborative robot, and the workstation height of the workstation to be operated.

[0069] In this embodiment, coordinate transformation can be performed according to the pixel coordinates and the pre-stored relative position between the camera and the collaborative robot and the station height of the station to be operated, so as to obtain the world coordinates of the station to be operated, wherein the world coordinates are coordinates in the world coordinate system.

[0070] In one implementation, the pixel coordinates may be transformed by combining the camera intrinsic parameters, the relative position between the camera and the collaborative robot, and the height of the workstation to be operated to obtain the world coordinates.

[0071] In one implementation, the intelligent device may be a host computer, which refers to a computer that can send control instructions. The trained target detection model, the relative position between the camera and the collaborative robot, and the height of the workstation to be operated may be stored in the host computer.

[0072] Step S104: Send motion instructions to the collaborative robot module according to the world coordinates to control the collaborative robot to move to the operating station to complete the preset operation.

[0073] In this embodiment, motion instructions can be sent to the collaborative robot according to the world coordinates to control the collaborative robot to move to the operating station to complete the preset operation.

[0074] In one embodiment, the intelligent device can communicate with the collaborative robot module and the production line control module respectively through the TCP (Transmission Control Protocol) communication protocol. The collaborative robot module is used to control the precise movement of the collaborative robot; the production line control module is used to provide real-time production line signals to generate demand instructions; the intelligent device is responsible for triggering the camera to acquire images, running the target detection model, processing and transmitting signals, and integrating the logic of the collaborative robot's guided positioning process, thereby achieving seamless connection between the intelligent device, the collaborative robot module and the production line control module, which not only ensures the synchronization and accuracy of the operation process, but also significantly improves flexibility and adaptability. When the collaborative robot module or the production line control module changes, communication can be achieved by modifying the corresponding network connection address, without modifying the host computer configuration, which can reduce debugging costs.

[0075] In one embodiment, the preset operation may be a bolt tightening operation.

[0076] Based on the method described in steps S101 to S104 above, the embodiment of the present application sends motion instructions to the collaborative robot according to the demand instructions of the production line control module, controls the collaborative robot module to move to the preset photo position, and controls the camera to collect images of the workstation to be operated corresponding to the photo position to obtain a first image. Based on the trained target detection model, the pixel coordinates of the workstation to be operated are obtained according to the first image. According to the pixel coordinates and the relative position and posture between the pre-stored camera and the collaborative robot, the workstation height of the workstation to be operated, the world coordinates of the workstation to be operated are obtained. According to the world coordinates, a motion instruction is sent to the collaborative robot module so that the collaborative robot moves to the workstation to be operated to complete the preset operation. Through the above configuration, the embodiment of the present application realizes accurate target detection based on the target detection model to obtain the pixel coordinates of the workstation to be operated, and automatically corrects the pixel coordinates according to the relative position and posture between the camera and the collaborative robot and the workstation height of the workstation to be operated, obtains the world coordinates of the workstation to be operated, and controls the collaborative robot to perform preset operations according to the world coordinates, which can improve the accuracy and repeatability of the operation and speed up the production speed. At the same time, the collaborative robot guidance and positioning method of the embodiment of the present application can effectively adapt to the needs in the case of production line replacement, reduce repeated configuration processes and adjustment time, and further improve production efficiency. In addition, since there is no need to perform special calibration processes and program writing for different products or production tasks, it also effectively improves the flexibility and adaptability of the production process and can be applied to a variety of different industrial production scenarios.

[0077] The following further describes the training process of the target detection model, the specific process of obtaining the relative posture between the camera and the collaborative robot, and the specific process of obtaining the height of the workstation to be operated.

[0078] In one implementation of the embodiment of the present application, the target detection model may be trained according to the following steps S201 to S203:

[0079] Step S201: Obtain image data samples and construct a training data set and a verification data set.

[0080] In this embodiment, a certain number of image data samples may be obtained, and the image data samples may be labeled, and the labeled image data samples may be separated to construct a training data set and a verification data set, respectively.

[0081] Step S202: Train the target detection model according to the training data set to obtain a pre-trained target detection model.

[0082] In this embodiment, the target detection model can be trained according to the training data set to obtain a trained target detection model.

[0083] In one implementation, the training process may use the pytorch platform to train the target detection model using GPU accelerated training.

[0084] Step S203: verify the pre-trained target detection model according to the verification data set to obtain a trained target detection model.

[0085] In this embodiment, step S203 may further include the following steps S2031 to S2034:

[0086] Step S2031: Verify the pre-trained target detection model based on the verification data set to obtain a verification accuracy index.

[0087] Step S2032: Determine whether the verification accuracy index meets the preset requirements; if so, execute step S2033; if not, execute step S2034.

[0088] Step S2033: Save the pre-trained target detection model as a trained target detection model.

[0089] Step S2034: After adjusting the parameters of the target detection model, execute step S202.

[0090] In this embodiment, the pre-trained target detection model can be verified based on the verification data set to determine the verification accuracy index of the target detection model. If it meets the standard (i.e., meets the preset requirements), the pre-trained target detection model can be saved as a trained target detection model. If it does not meet the standard, the parameters of the target detection model can be adjusted, such as adjusting the learning rate, optimizer, etc. of the target detection model, and then the target detection model is trained.

[0091] In one implementation, mAP5.0 can be used as a verification accuracy indicator. Here, mAP5.0 refers to the mean average precision at IoU 0.5. IoU refers to the intersection over union ratio, and 0.5 is the evaluation window size.

[0092] In one embodiment, the saved model format of the trained target detection model can be ONNX (Open Neural Network Exchange, an open source framework for representing deep learning models), which can realize the deployment of the trained target detection model on different platforms, and ensure the flexibility of the collaborative robot guided positioning process.

[0093] In one embodiment, please refer to the attached Figure 3 , Figure 3FIG. 1 is a flow chart of the main steps of training a target detection model according to an implementation of an embodiment of the present application. Figure 3 As shown, the target detection model can be trained according to the following steps S301 to S307:

[0094] Step S301: Collect image data samples.

[0095] Step S302: Dataset (including training data set and verification data set) is generated.

[0096] Step S303: Target detection model training and verification.

[0097] Step S304: Accuracy test.

[0098] Step S305: Determine whether the verification accuracy index meets the preset requirements; if so, execute step S306; if not, execute step S307.

[0099] Step S306: After saving the model as a trained target detection model, the process ends.

[0100] Step S307: After adjusting the model parameters, execute step S303.

[0101] In one implementation of the embodiment of the present application, the relative position between the camera and the collaborative robot can be acquired according to the following steps S401 to S403.

[0102] Step S401: Send a motion instruction to the collaborative robot to control the collaborative robot to move to a preset position.

[0103] In this embodiment, the collaborative robot can be controlled to move to a preset position.

[0104] Step S402: Control the camera to capture a preset number of second images at preset positions.

[0105] In this embodiment, there may be multiple preset positions of the second image, there may be multiple second images, and step S402 may further include the following steps S4021 to S4022:

[0106] Step S4021: Control the camera to capture the second image at the current position, and determine whether the second image reaches a preset number; if so, execute step S403; if not, execute step S4022.

[0107] Step S4022: After controlling the collaborative robot to move to the next preset position, execute step S4021.

[0108] Step S403: Acquire the relative position and posture between the camera and the collaborative robot according to a preset number of second images.

[0109] In this embodiment, the second image of the collaborative robot in different postures can be automatically collected, and the posture solution is performed based on the second image to obtain the camera internal parameters and the relative posture between the camera and the collaborative robot (i.e., the hand-eye relationship). The process of obtaining the relative posture does not require human participation, and the accurate correspondence between the camera and the collaborative robot can be obtained, which significantly reduces the operating cost. At the same time, due to the high efficiency of the process of obtaining the relative posture, it can be implemented in different collaborative robot modules, thereby ensuring that the collaborative robot module can be quickly adapted to new production tasks or different workstations, and can improve the flexibility and response speed of the production line.

[0110] In one embodiment, after a preset number of second images are captured, the collaborative robot may be controlled to move to an initial position.

[0111] In one embodiment, please refer to the attached Figure 4 , Figure 4 FIG. 1 is a flow chart of the main steps of obtaining the relative position between the camera and the collaborative robot according to an implementation of an embodiment of the present application. Figure 4 As shown, the relative position between the camera and the collaborative robot can be obtained according to the following steps S501 to S513:

[0112] Step S501: Connect the camera and execute steps S502 and S503

[0113] Step S502: After starting the first thread for controlling the camera, step S509 is executed through the first thread.

[0114] Step S503: Connect to the collaborative robot and execute steps S504 and S505.

[0115] Step S504: After starting the second thread for controlling the collaborative robot, execute step S506.

[0116] Step S505: After establishing a communication connection between the robot server and the robot client through TCP communication, step S507 is executed through the second thread.

[0117] Step S506: Sending motion instructions to the robot through the second thread.

[0118] Step S507: Receive location information.

[0119] Step S508: Determine whether the collaborative robot has moved to the specified position; if so, execute step S509; if not, execute step S513.

[0120] Step S509: Acquire a second image.

[0121] Step S510: Determine whether the number of second image acquisitions reaches a preset number; if so, execute step S511; if not, execute step S506.

[0122] Step S511: Calculate the relative position between the camera and the collaborative robot according to the second image.

[0123] Step S512: Save the result.

[0124] Step S513: Alarm check.

[0125] In one implementation of the application example, the station height of the station to be operated can be obtained according to the following steps S601 to S604:

[0126] Step S601: Send a motion instruction to the collaborative robot to control the collaborative robot to move to a preset photo taking position and obtain the current posture of the collaborative robot.

[0127] Step S602: Control the camera to capture an image of the calibration plate corresponding to the photographing position to obtain a third image.

[0128] Step S603: Obtain the relative position and posture between the calibration plate and the camera according to the third image.

[0129] Step S604: Acquire the height of the workstation to be operated according to the relative posture between the calibration plate and the camera and the relative posture between the camera and the collaborative robot.

[0130] In this embodiment, in order to measure the station height of the station to be operated, a calibration plate can be placed on the station to be operated, and the calibration plate can be photographed by a camera and solved by an image processing algorithm to determine the relative position between the calibration plate and the camera, and further solve the station height of the station to be operated according to the relative position between the camera and the collaborative robot. The image processing algorithm can be combined to achieve accurate measurement of the station height of the station to be operated, which can optimize the operating accuracy and efficiency of the collaborative robot, not only reducing the hardware cost, but also improving the applicability and flexibility of the collaborative robot's guided positioning process.

[0131] In one embodiment, please refer to the attached Figure 5 , Figure 5 FIG. 1 is a flow chart of the main steps of obtaining the height of the workstation to be operated according to an implementation of an embodiment of the present application. Figure 5 As shown, the station height of the station to be operated can be obtained through the following steps S701 to S705:

[0132] Step S701: The collaborative robot moves to a preset photo taking position.

[0133] Step S702: Acquire a third image.

[0134] Step S703: Calculate the camera external parameters, that is, the position and posture of the calibration plate in the camera coordinate system in the current state.

[0135] Step S704: Solve the transformation relationship, that is, use the relative posture between the camera and the collaborative robot and the posture of the calibration plate in the camera coordinate system to obtain the transformation relationship.

[0136] Step S705: coordinate transformation, that is, calculating the station height of the station to be operated according to the transformation relationship, so as to achieve 3D positioning of the station to be operated.

[0137] In one embodiment, please refer to the attached Figure 6 , Figure 6 FIG. 1 is a schematic flow chart of the main steps of a collaborative robot guidance and positioning method according to an implementation of an embodiment of the present application. Figure 6 As shown, the collaborative robot is used to perform a bolt tightening operation, and the collaborative robot guidance positioning method may include the following steps S801 to S813:

[0138] Step S801: Connect the camera, and execute steps S802 and S803.

[0139] Step S802: triggering and starting the first thread for controlling the camera, and executing step S809.

[0140] Step S803: Connect to the collaborative robot and execute steps S804 and S805.

[0141] Step S804: Trigger and start the second thread for controlling the collaborative robot, and execute steps S808 and S812.

[0142] Step S805: Connect the production line control module and execute steps S806 and S807.

[0143] Step S806: After triggering and starting the production line control module communication thread, execute step S807.

[0144] Step S807: Receive work instructions.

[0145] Step S808: The collaborative robot moves to a preset photo-taking position.

[0146] Step S809: Acquire a first image.

[0147] Step S810: Target detection model inference to obtain pixel coordinates of the bolt.

[0148] Step S811: coordinate transformation to obtain world coordinates. That is, coordinate transformation is performed according to the camera internal parameters, the relative posture between the camera and the collaborative robot, and the height of the station to be operated where the bolt is located to obtain the world coordinates of the bolt.

[0149] Step S812: Control the collaborative robot to move to the workstation to be operated according to the world coordinates.

[0150] Step S813: Complete the tightening operation.

[0151] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effect of the present application, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted schemes are equivalent to the technical schemes described in this application, and therefore will also fall within the scope of protection of this application.

[0152] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above-mentioned embodiment of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electrical carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.

[0153] Another aspect of the present application also provides a computer-readable storage medium.

[0154] In an embodiment of a computer-readable storage medium according to the present application, the computer-readable storage medium can be configured to store a program for executing the collaborative robot guidance and positioning method of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned collaborative robot guidance and positioning method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-temporary computer-readable storage medium.

[0155] Another aspect of the present application also provides a smart device.

[0156] In an embodiment of an intelligent device according to the present application, the intelligent device may include at least one processor; and a memory connected to the at least one processor in communication; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method described in any of the above embodiments is implemented. Figure 7 , Figure 7 FIG. 4 exemplarily shows that the memory 11 and the processor 12 are communicatively connected via a bus.

[0157] The electronic device described in the present application may be, but is not limited to, a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, a vehicle-mounted device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), an augmented reality (AR)\virtual reality (VR) device, etc., and the embodiments of the present application are not limited to this.

[0158] Furthermore, the present application also provides a collaborative robot guidance and positioning system.

[0159] In an embodiment of a collaborative robot guidance and positioning system of the present application, the collaborative robot guidance and positioning system may include a collaborative robot module, a production line control module, and the intelligent device in the above intelligent device embodiment. The collaborative robot module and the production line control module are respectively connected to the intelligent device for communication.

[0160] In one implementation, the smart device may be a host computer. Figure 2 ,like Figure 2 As shown, the collaborative robot guidance and positioning system may include a collaborative robot module, a host computer and a production line control module. The collaborative robot module and the production line control module are respectively connected to the host computer through the TCP communication protocol. The collaborative robot module may include a collaborative robot and a camera. The host computer includes a camera trigger unit, an identification and positioning unit, an integrated communication unit and a trajectory planning unit, through which target detection, automatic hand-eye calibration (i.e., obtaining the relative posture between the camera and the collaborative robot), 3D positioning (i.e., obtaining the station height of the station to be operated), integrated communication and other functions can be realized, thereby finally realizing the preset operation of the station to be operated (e.g., tightening operation).

[0161] So far, the technical solution of the present application has been described in conjunction with an embodiment shown in the accompanying drawings, but it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present application.

Claims

1. A collaborative robot guidance and positioning method, characterized in that: The method is applied to an intelligent device, wherein the intelligent device is respectively connected to a collaborative robot module and a production line control module, wherein the collaborative robot module includes a camera and a collaborative robot; The method comprises: Based on the demand instruction of the production line control module, a motion instruction is sent to the collaborative robot, the collaborative robot module is controlled to move to a preset photographing position, and the camera is controlled to collect an image of the workstation to be operated corresponding to the photographing position to obtain a first image; Based on a preset trained target detection model, the pixel coordinates of the workstation to be operated are acquired according to the first image; Acquire the world coordinates of the workstation to be operated according to the pixel coordinates, the pre-stored relative position between the camera and the collaborative robot, and the workstation height of the workstation to be operated; Send motion instructions to the collaborative robot module according to the world coordinates to control the collaborative robot to move to the waiting operation station to complete the preset operation.

2. The collaborative robot guidance and positioning method according to claim 1, characterized in that: The method comprises training the target detection model according to the following steps: Obtain image data samples and construct training and verification data sets; Training the target detection model according to the training data set to obtain a pre-trained target detection model; The pre-trained target detection model is verified according to the verification data set to obtain the trained target detection model.

3. The collaborative robot guidance and positioning method according to claim 2, characterized in that: The step of verifying the pre-trained target detection model according to the verification data set includes: According to the verification data set, the pre-trained target detection model is verified to obtain a verification accuracy index; Determine whether the verification accuracy index meets the preset requirements; If satisfied, the pre-trained target detection model is saved as a trained target detection model; If not, the parameters of the target detection model are adjusted, and the step of "training the target detection model according to the training data set" is performed.

4. The collaborative robot guidance and positioning method according to claim 1, characterized in that: The method further comprises obtaining the relative position between the camera and the collaborative robot according to the following steps: Sending a motion instruction to the collaborative robot to control the collaborative robot to move to a preset position; Controlling the camera to collect a preset number of second images of the preset positions; According to the preset number of second images, the relative position between the camera and the collaborative robot is acquired.

5. The collaborative robot guidance and positioning method according to claim 4, characterized in that: There are multiple preset positions; The controlling the camera to collect a preset number of second images of the preset positions includes: Controlling the camera to capture a second image at the current position, and determining whether the second image reaches a preset number; If yes, then execute "obtaining the relative position between the camera and the collaborative robot according to the preset number of second images" and control the collaborative robot to move to the initial position; If not, the collaborative robot is controlled to move to the next preset position, and the step of "controlling the camera to capture a second image at the current position, and determining whether the second image reaches a preset number" is executed.

6. The collaborative robot guidance and positioning method according to claim 1, characterized in that: The method further comprises obtaining the station height of the station to be operated according to the following steps: Sending a motion instruction to the collaborative robot to control the collaborative robot to move to a preset photographing position, and obtaining the current position and posture of the collaborative robot; Controlling the camera to collect an image of the calibration plate corresponding to the photographing position to obtain a third image; According to the third image, obtaining a relative posture between the calibration plate and the camera; The station height of the station to be operated is obtained according to the relative posture between the calibration plate and the camera and the relative posture between the camera and the collaborative robot.

7. The collaborative robot guidance and positioning method according to any one of claims 1 to 6, characterized in that: Before sending the motion instruction to the collaborative robot, the method further includes: Connecting the camera and starting a first thread for controlling the camera; Connect the collaborative robot, start a second thread for controlling the collaborative robot, and establish a communication connection with the collaborative robot module.

8. A smart device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the collaborative robot guidance and positioning method according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the collaborative robot guidance and positioning method according to any one of claims 1 to 7.

10. A collaborative robot guidance and positioning system, characterized in that: The system comprises: a collaborative robot module, a production line control module and the intelligent device according to claim 8; the collaborative robot module and the production line control module are respectively connected to the intelligent device for communication.

Images