Equipment transfer method and system and electronic equipment

Abstract

The embodiment of the invention discloses an equipment transfer method and system and electronic equipment, and the method comprises the steps: obtaining the identification information of target equipment at a transfer operation starting position, and determining the camera shooting pose and mechanical arm clamping jaw configuration corresponding to the target equipment based on the identification information; a pose image of the target equipment is obtained through a visual camera according to the camera shooting pose, and mechanical arm grabbing coordinates are determined based on the pose image; and on the basis of the mechanical arm clamping jaw configuration and the mechanical arm grabbing coordinates, the mechanical arm is controlled to grab the target equipment for transfer operation. According to the embodiment of the invention, the process of grabbing and transferring the target equipment by the mechanical arm can be accurately controlled.

Description

Technical Field

[0001] The present invention relates to the technical field of equipment transfer, and particularly to an equipment transfer method, system and electronic device. Background Art

[0002] When performing transfer operations such as inbound and outbound operations on various types of equipment, such as different types of metering equipment, it mainly relies on manual identification and grasping operations, with relatively low efficiency and certain safety risks. Although there are robotic arm grasping and transfer solutions based on image recognition, the robotic arm grasping and transfer solutions in the prior art have poor control effects on the equipment during the transfer process, easily resulting in transfer failures or equipment damage. Summary of the Invention

[0003] Embodiments of the present invention provide an equipment transfer method, system and electronic device, which can precisely control the process of a robotic arm grasping and transferring a target equipment.

[0004] In a first aspect, embodiments of the present invention provide an equipment transfer method, including:

[0005] Obtaining identification information of a target equipment at the starting position of a transfer operation, and determining a camera shooting pose and a robotic arm gripper configuration corresponding to the target equipment based on the identification information;

[0006] Obtaining a pose image of the target equipment by a vision camera in the camera shooting pose, and determining a robotic arm grasping coordinate based on the pose image; and

[0007] Controlling the robotic arm to grasp the target equipment for a transfer operation based on the robotic arm gripper configuration and the robotic arm grasping coordinate.

[0008] In a second aspect, embodiments of the present invention provide an equipment transfer system, including:

[0009] A robotic arm, a vision camera and an equipment transfer device, wherein the pose of the vision camera is adjustable, and the equipment transfer device is used to execute any one of the equipment transfer methods in the embodiments of the present invention.

[0010] In a third aspect, embodiments of the present invention further provide an electronic device, including a memory, a processor and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, it implements any one of the equipment transfer methods in the embodiments of the present invention.

[0011] A device transfer method, system and electronic device provided by an embodiment of the present invention can determine the optimal camera shooting pose and the optimal robotic arm gripper configuration by acquiring and based on the identification information of the target device, and then capture the pose image of the target device based on the optimal camera shooting pose, so as to obtain a high-precision pose image; then determine the accurate robotic arm grasping coordinates based on the high-precision pose image, and control the robotic arm to grasp the target device for transfer operations based on the accurate robotic arm grasping coordinates and the optimal robotic arm grasping configuration, which can reduce manual intervention, improve operation efficiency, precisely control the process of the robotic arm grasping and transferring the device, ensure the safety of the device and personnel, reduce the safety risk during the operation process and avoid operation failures. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] To more clearly illustrate the technical solutions of the present invention, the accompanying drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0013] Figure 1 is a flowchart of the device transfer method provided by an embodiment of the present invention;

[0014] Figure 2 is another flowchart of the device transfer method provided by an embodiment of the present invention;

[0015] Figure 3 is another flowchart of the device transfer method provided by an embodiment of the present invention;

[0016] Figure 4 is another flowchart of the device transfer method provided by an embodiment of the present invention;

[0017] Figure 5 is a structural diagram of the device transfer system provided by an embodiment of the present invention;

[0018] Figure 6 is another structural diagram of the device transfer system provided by an embodiment of the present invention;

[0019] Figure 7 is a structural diagram of an integrated bracket provided by an embodiment of the present invention;

[0020] Figure 8 is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order different from those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0023] Figure 1 FIG. is a schematic flowchart of a device transfer method provided by an embodiment of the present invention. This embodiment is applicable to a scenario where a robotic arm is used to transfer various types of devices. This method can be executed by a device transfer device provided by an embodiment of the present invention, and the device can be implemented in a software and / or hardware manner. In a specific embodiment, the device can be integrated in an electronic device, and the electronic device can be, for example, a computer, a server, etc. The following embodiments will be described by taking the device integrated in the electronic device as an example. Refer to Figure 1 , the method can specifically include the following steps:

[0024] Step 101, obtain the identification information of the target device at the starting position of the transfer operation, and determine the camera shooting pose and robotic arm gripper configuration corresponding to the target device based on the identification information. This step can determine the optimal camera shooting pose and the optimal robotic arm gripper configuration, which is conducive to obtaining a high-precision target device pose image by the vision camera based on the optimal camera shooting pose, and is conducive to gripping the target device by the robotic arm based on the optimal robotic arm gripper configuration for transfer operations.

[0025] Specifically, the above-mentioned target device includes a metering device, and the metering device can include, for example, an electric energy meter, a mutual inductor, a collector, etc.

[0026] Specifically, the above-mentioned identification information may include the identification image of the target device, and the identification image may be an image with relatively low accuracy.

[0027] Optionally, the process of obtaining the identification information of the target device to be transferred includes: obtaining the above-mentioned identification image of the target device through a vision camera.

[0028] Specifically, the above-mentioned identification information may include the identification code signal of the target device, and the identification code signal may specifically be a bar code signal, a two-dimensional code signal, etc.

[0029] Optionally, the process of obtaining the identification information of the target device to be transferred includes: obtaining the identification code signal of the target device through a code reader.

[0030] Specifically, the above-mentioned camera shooting pose may include the height and / or rotation angle of the corresponding camera, etc.

[0031] Specifically, the above-mentioned robotic arm gripper configuration may include the opening degree, grasping force, and / or the material of the gripper contact surface of the robotic arm gripper, etc.

[0032] Optionally, the process of determining the device category and placement state of the target device based on the identification information includes: determining the device category and placement state of the target device based on the identification image through a pre-trained image recognition model.

[0033] Optionally, the above-mentioned image recognition model may be a deep learning algorithm, such as a convolutional neural network, which is obtained through multiple rounds of training based on the pre-collected image data and image feature annotation information of various devices to be transferred. The above-mentioned image features may specifically be key positions such as boundaries and buttons.

[0034] Optionally, an integrated bracket is provided above the starting position of the transfer operation. The vision camera is fixed on the first horizontal side of the integrated bracket through a pose adjustment mechanism. The pose adjustment mechanism is used to adjust the height and rotation angle of the vision camera. A light strip is also provided between the first horizontal side and the vision camera. The light strip is fixed on the first horizontal side for supplementary lighting when obtaining the pose image; the top of the integrated bracket is covered with a black opaque acrylic panel for blocking the top light to avoid reflection interference when obtaining the pose image.

[0035] Specifically, the above-mentioned light strip can also be used when obtaining the above-mentioned identification image of the target device through the vision camera, and / or obtaining the identification code signal of the target device through the code reader.

[0036] Specifically, the above-mentioned integrated bracket may be a horizontal bracket supported by legs, as Figure 7 shown, or it may be a suspended horizontal bracket or a horizontal bracket with other support structures.

[0037] Specifically, the shape of the above-mentioned integrated bracket can be flexibly determined based on the shape of the starting position.

[0038] Specifically, the starting position of the above transfer operation can be on the conveyor belt. Therefore, the shape of the integrated bracket can be specifically set as a rectangle parallel to the conveyor belt, as Figure 7 shown. Then both the vision camera and the light strip can be fixed on the horizontal side parallel to the conveying direction of the conveyor belt.

[0039] Specifically, the brightness of the above light strip can be adjusted to adapt to different target devices to avoid reflection.

[0040] Optionally, the above pose adjustment mechanism can include a servo motor telescopic device.

[0041] Specifically, the aforementioned barcode reader can also be fixed on the integrated bracket.

[0042] Specifically, the above vision camera can also be integrated on the drone.

[0043] It can be understood that in the embodiments of the present invention, by setting an integrated bracket and a pose adjustment mechanism on the integrated bracket to adjust the pose of the vision camera, it is possible to adjust the vision camera to the optimal spectral shooting height and viewing angle during the subsequent process of using the vision camera to capture pose images, avoiding image distortion or blurring caused by the device being too high or too low and inappropriate viewing angles, thereby ensuring that accurate grasping coordinates can be obtained; moreover, the light strip provided on the integrated bracket in the embodiments of the present invention can perform supplementary lighting while ensuring uniform illumination of the device surface and avoiding interference from reflected light; in addition, the top of the integrated bracket in the embodiments of the present invention is covered with a black opaque acrylic panel, which can effectively reduce the interference of external light sources on image acquisition and ensure the image clarity and contrast during the vision recognition process.

[0044] Step 102: Obtain the pose image of the target device through the vision camera at the camera shooting pose, and determine the robotic arm grasping coordinates based on the pose image. In this step, the vision camera captures a high-precision pose image of the target device based on the optimal camera shooting pose, and determines the high-precision robotic arm grasping coordinates based on the high-precision pose image of the target device, which is beneficial for the robotic arm to grasp and transfer the target device based on the high-precision grasping coordinates, and further beneficial for effectively controlling the device during the process of grasping and transferring the target device.

[0045] It can be understood that during the process of the grasping and transfer operation, if the grasping coordinates are inaccurate, problems such as incorrect motion path planning, inability to correctly judge the actual position and state of the object, resulting in too large or too small grasping force, and incorrect adjustment of the robotic arm grasping posture will occur. Therefore, it will lead to the failure of the grasping and transfer and damage to the device. Therefore, obtaining a high-precision pose image of the target device and high-precision robotic arm grasping coordinates is crucial during the process of the robotic arm grasping and transfer operation.

[0046] Optionally, the process of obtaining the pose image of the target device by the vision camera at the camera shooting pose includes: adjusting the pose of the vision camera to the shooting pose, and controlling the vision shooting camera to shoot the target device after the pose adjustment is completed to obtain the pose image.

[0047] Optionally, the process of determining the robotic arm grasping coordinates based on the pose image includes: determining the robotic arm grasping coordinates based on the pose image through a pre-trained grasping coordinate regression model.

[0048] Optionally, the process of determining the robotic arm grasping coordinates based on the pose image through a pre-trained grasping coordinate regression model includes:

[0049] Determining the grasping position feature map of the target device through the coordinate regression model, and determining the robotic arm grasping coordinates based on the grasping position feature map.

[0050] Specifically, the network structure of the coordinate regression model can be the DarkNet-53 backbone of YOLOv3.

[0051] Specifically, the coordinate regression model can be trained based on the Cornell Grasp Dataset (CornellGraspDataset).

[0052] Step 103, based on the robotic arm gripper configuration and the robotic arm grasping coordinates, control the robotic arm to grasp the target device for transfer operation. Based on Steps 101 and 102, this step can control the robotic arm to grasp and transfer the target device based on accurate robotic arm grasping coordinates and the optimal robotic arm grasping configuration, which can reduce manual intervention, improve operation efficiency, precisely control the process of the robotic arm grasping and transferring the device, ensure the safety of the device and personnel, reduce the safety risk during the operation process, and avoid operation failures.

[0053] Optionally, the process of controlling the robotic arm to grasp the target device for transfer operation based on the robotic arm gripper configuration and the robotic arm grasping coordinates includes: planning the transfer path based on the robotic arm grasping coordinates, and transferring the target device by the robotic arm based on the transfer path and the robotic arm gripper configuration.

[0054] Specifically, the transfer path includes the approaching operation path for the robotic arm to approach the target device and the carrying path for the robotic arm to carry the target device to the target position.

[0055] Optionally, the process of planning the transfer path based on the robotic arm grasping coordinates includes:

[0056] On the premise of ensuring that the robotic arm can accurately reach the target device, avoid obstacles between devices, prevent the gripper from colliding with other devices, and avoid dropping the device due to too fast or unstable movements, the above-mentioned approaching operation path and handling path are calculated through a path planning algorithm, such as the A* algorithm or the D* algorithm, etc.

[0057] Optionally, the process of transferring the above-mentioned target device by the robotic arm based on the above-mentioned transfer path and the robotic arm gripper configuration includes:

[0058] Control the robotic arm to approach the target device through the approaching operation path, grab the device with the gripper, and control the robotic arm after grabbing the target device to carry the target device to the target position through the handling path.

[0059] Optionally, the device transfer method provided by the embodiments of the present invention further includes: determining the change relationship between the camera coordinate system of the vision camera and the base coordinate system of the robotic arm through an eye-in-hand calibration method based on a calibration board fixed at the end of the robotic arm.

[0060] It can be understood that the eye-in-hand calibration method allows the camera to be installed at different positions and angles, as long as it can ensure that the camera can clearly observe the robotic arm and the working area. Therefore, accurate and efficient calibration can be completed when the vision camera is at different camera poses.

[0061] The following further introduces the device transfer method provided by the embodiments of the present invention, as Figure 2 shown, which may include the following steps:

[0062] Step 201, establish a device database including the category information, size data, shape feature data, and weight data of each device that needs to perform transfer operations.

[0063] Specifically, the size data of the above-mentioned each device may include data such as the corresponding length, width, height, and thickness.

[0064] Specifically, the shape feature data of the above-mentioned each device may include features such as the corresponding contact surface and shell material.

[0065] Specifically, the weight data of the above-mentioned each device may include data such as the corresponding total weight and center position.

[0066] Step 202, obtain the identification information of the target device at the starting position of the transfer operation, and determine the device category and placement state of the target device based on the identification information.

[0067] Specifically, the above-mentioned placement state may include being placed upright, placed sideways, and placed obliquely.

[0068] Step 203: Query the device database based on the device category to obtain the size data, external shape feature data, and weight data of the target device.

[0069] Step 204: Determine the camera shooting pose based on the size data and placement state, and determine the robotic arm gripper configuration based on the size data, external shape feature data, weight data, and placement state.

[0070] It can be understood that when the device types are different or the placement states of the same type of device are different, for example, when the device is placed upright or sideways, the optimal camera shooting pose for photographing the device is different. Therefore, based on the size data and placement state of the target device, the optimal camera shooting pose can be determined.

[0071] Optionally, the robotic arm gripper configuration includes: the opening degree of the robotic arm gripper, the grasping force, and the material of the gripper contact surface.

[0072] Specifically, the material of the contact surface of the above-mentioned gripper includes a polyurethane gasket.

[0073] Optionally, the process of determining the robotic arm gripper configuration based on the size data, external shape feature data, weight data, and placement state includes: determining the grasping surface of the target device based on the placement state, and determining the opening degree of the robotic arm gripper, the grasping force, and the material of the gripper contact surface based on the grasping surface, size data, external shape feature data, and weight data.

[0074] Optionally, the process of determining the opening degree of the robotic arm gripper, the grasping force, and the material of the gripper contact surface based on the grasping surface, size data, external shape feature data, and weight data includes:

[0075] Adapting the material of the gripper contact surface to the material corresponding to the external shape feature of the grasping surface of the target device, adapting the opening degree of the gripper to the range corresponding to the distance dimension between the two grasping surfaces of the target device, and adapting the grasping force of the gripper to the force magnitude corresponding to the weight of the target device.

[0076] It can be understood that when different types of devices and the same device have different grasping surfaces due to different placement states, the opening degree of the robotic arm gripper and the material of the clamping contact surface required are different. For example, for an electricity meter with a reflective panel on its surface, using a soft polyurethane gasket on the gripper contact surface can avoid damage to the reflective panel or device damage caused by uneven pressure; and different types of devices have different weights or different frictional forces determined by different grasping surface materials, so different grasping forces are required for the gripper. Therefore, determining the opening degree of the robotic arm gripper, the grasping force, and the material of the gripper contact surface based on the grasping surface, size data, external shape feature data, and weight data can better grasp the target device and accurately transfer the target device, and avoid damaging the target device.

[0077] Step 205: Based on the robotic arm gripper configuration and the robotic arm grasping coordinates, control the robotic arm to grasp the target device for transfer operation.

[0078] The embodiments of the present invention can facilitate better grasping of the target device and precise transfer of the target device, so as to further improve the control effect of the process of the robotic arm grasping and transferring the device and avoid damaging the target device.

[0079] The following further introduces the device transfer method provided by the embodiments of the present invention. As Figure 3 shown, the training process of the grasping coordinate regression model provided in the embodiments of the present invention includes:

[0080] Step 301: Determine the true value of the grasping pose based on the sample image, where the sample image includes a plurality of image data obtained from the Cornell grasping dataset.

[0081] Optionally, the process of determining the true value of the grasping pose based on the sample image includes: determining the vertex coordinates of the grasping rectangle corresponding to the target grasping device in the sample image and obtaining the five-parameter method representation result of the grasping rectangle to obtain the above-mentioned true value of the grasping pose.

[0082] Specifically, the four vertex coordinates can be represented by (x1, y1), (x2, y2), (x3, y3), and (x4, y4), respectively. Then, the corresponding five-parameter method representation result can be calculated by the following formula:

[0083]

[0084] Among them, (X0, Y0) represents the coordinates corresponding to the center of the grasping rectangle; H0 represents the maximum height of the two-finger parallel gripper of the robotic arm; W0 represents the finger width of the two-finger parallel gripper of the robotic arm; θ0 represents the angle of the grasping rectangle relative to the horizontal axis.

[0085] Step 302: Input the sample image into the coordinate regression model to be trained, and obtain the predicted value of the grasping pose of the robotic arm through the coordinate regression model.

[0086] Step 303: Determine the function value of the training loss function based on the predicted value of the grasping pose and the true value of the grasping pose, and adjust the network parameters in the coordinate regression model based on the function value of the training loss function.

[0087] Specifically, the mean square error (MSE) can be used as the training loss function, and the stochastic gradient descent method (SGD) can be used as the optimizer. The training loss function can be expressed as:

[0088]

[0089] Among them, (x, y, h, w, θ) are the predicted values, (X0, Y0, H0, W0, θ0) are the true values, λ is the angle weighting value which can be set to 10, and N is the batch size.

[0090] The embodiment of the present invention can train a coordinate regression model with high prediction accuracy when regressively predicting the grasping coordinates.

[0091] The following further introduces the device transfer method provided by the embodiment of the present invention. As Figure 4 shown, it may include the following steps:

[0092] Step 401, obtain the identification code signal of the target device through a barcode reader to obtain the identification information of the target device.

[0093] Optionally, the process of obtaining the identification code signal of the target device through the barcode reader includes: scanning the barcode on the target device through an industrial barcode reader to obtain the barcode reflection signal to obtain the above-mentioned identification code signal of the target device.

[0094] Step 402, decode the identification code signal to obtain the device code of the device, and generate a device transfer record based on the device code.

[0095] Specifically, the above-mentioned device transfer record includes the device inbound and outbound record.

[0096] Optionally, the process of decoding the identification code to obtain the device code of the device includes: performing photoelectric conversion on the barcode reflection signal and then decoding to obtain the above-mentioned device code.

[0097] It can be understood that since objects of different colors have different wavelengths of visible light reflected, white objects can reflect visible light of various wavelengths, while black objects absorb visible light of various wavelengths. Therefore, when the light emitted by the light source of the barcode scanner passes through the diaphragm and convex lens 1 (as shown in the following figure) and irradiates the black-and-white barcode, the reflected light is focused by convex lens 2 and then irradiates the photoelectric converter. Thus, the photoelectric converter receives the reflected light signals of different intensities corresponding to the white and black bars and converts them into corresponding electrical signals, which are output to the amplification and shaping circuit. The shaping circuit converts the analog signal into a digital electrical signal, and then decodes it into digital character information through the decoding interface circuit.

[0098] Step 403, determine the camera shooting pose and the gripper configuration of the robotic arm corresponding to the target device based on the device code.

[0099] Optionally, the above-mentioned device database also stores the device code information of each device.

[0100] Optionally, the process of determining the camera shooting pose and the gripper configuration of the robotic arm corresponding to the target device based on the device code includes:

[0101] Query the device database based on the above device code to obtain the size data, shape feature data, and weight data of the target device, and determine the camera shooting pose and the gripper configuration of the robotic arm based on the above size data, shape feature data, and weight data.

[0102] Step 404: Obtain the pose image of the target device by the vision camera in the camera shooting pose, and determine the robotic arm grasping coordinates based on the pose image.

[0103] Step 405: Control the robotic arm to grasp the target device for transfer operation based on the gripper configuration of the robotic arm and the robotic arm grasping coordinates.

[0104] The embodiment of the present invention can facilitate the precise management of device dynamics and facilitate precise traceability when there are subsequent related requirements, providing strong support for improving the operation process and operation quality.

[0105] Figure 5 It is a structural diagram of a device transfer system provided by an embodiment of the present invention, which includes: a vision camera 501, a robotic arm 502, and a device transfer device 503. The device transfer device is applicable to execute the device transfer method provided by any embodiment of the present invention. As Figure 5 shown, the device may specifically include:

[0106] A shooting pose and gripper configuration determination module 5031, configured to obtain the identification information of the target device at the starting position of the transfer operation, and determine the corresponding camera shooting pose and the gripper configuration of the robotic arm based on the identification information.

[0107] Optionally, the above-mentioned gripper configuration of the robotic arm includes the opening degree, grasping force, and the material of the gripper contact surface of the robotic arm gripper.

[0108] Optionally, the above-mentioned identification information includes the identification image of the target device.

[0109] Optionally, the shooting pose and gripper configuration determination module 5031 can specifically be used to obtain the identification image of the target device through the vision camera, and / or obtain the identification code signal of the target device through the barcode reader, to obtain the identification information of the target device.

[0110] Optionally, the shooting pose and gripper configuration determination module 5031 can specifically be used to determine the device category and placement state of the target device based on the identification image through a pre-trained image recognition model.

[0111] A grasping coordinate determination module 5032, configured to obtain the pose image of the target device by the vision camera in the camera shooting pose, and determine the robotic arm grasping coordinates based on the pose image.

[0112] Optionally, the above-mentioned grasping coordinate determination module 5032 can specifically be used to determine the robotic arm grasping coordinates based on the pose image through a pre-trained grasping coordinate regression model.

[0113] Optionally, the training process of the above-mentioned coordinate regression model includes: determining the true value of the grasping pose based on the sample image, where the sample image includes multiple image data obtained from the Cornell Grasping Dataset; inputting the sample image into the coordinate regression model to be trained, and obtaining the predicted value of the robotic arm's grasping pose through the coordinate regression model; and determining the function value of the training loss function based on the predicted value of the grasping pose and the true value of the grasping pose, and adjusting the network parameters in the coordinate regression model based on the function value of the training loss function.

[0114] The transfer control module 5033 is used to control the robotic arm to grasp the target device for transfer operations based on the robotic arm gripper configuration and the robotic arm grasping coordinates.

[0115] Optionally, the device transfer device provided in the embodiment of the present invention further includes: a device database establishment module, which is used to establish a device database including the category information, size data, shape feature data, and weight data of each device that needs to be transferred before determining the camera shooting pose and the robotic arm gripper configuration corresponding to the target device based on the identification information.

[0116] Optionally, the above-mentioned grasping coordinate determination module 5032 can specifically be used to determine the device category and placement state of the target device based on the identification information; query the device database based on the device category to obtain the size data, shape feature data, and weight data of the target device; and determine the camera shooting pose based on the size data and the placement state, and determine the robotic arm gripper configuration based on the size data, shape feature data, weight data, and placement state.

[0117] Optionally, the above-mentioned grasping coordinate determination module 5032 can specifically be used to determine the grasping surface of the target device based on the placement state, and determine the opening degree, grasping force, and gripper contact surface material of the robotic arm gripper based on the grasping surface, size data, shape feature data, and weight data.

[0118] The device transfer system provided in the embodiment of the present invention can obtain and determine the optimal camera shooting pose and the optimal robotic arm gripper configuration based on the identification information of the target device, and then capture the pose image of the target device based on the optimal camera shooting pose, and can obtain a high-precision pose image; then determine the accurate robotic arm grasping coordinates based on the high-precision pose image, and control the robotic arm to grasp the target device for transfer operations based on the accurate robotic arm grasping coordinates and the optimal robotic arm grasping configuration, thereby being able to reduce manual intervention, improve operation efficiency, precisely control the process of the robotic arm grasping and transferring the device, ensure the safety of the device and personnel, reduce the safety risk during the operation process, and avoid operation failures.

[0119] The device transfer system provided by the embodiments of the present invention will be further introduced below. As Figure 6 shown, it may include: a robotic arm 501, a vision camera 502, a device transfer device 503, and an integrated bracket 604.

[0120] Optionally, the integrated bracket is arranged directly above the starting position of the operation. The vision camera is fixed on the first horizontal side of the integrated bracket through a pose adjustment mechanism, and the pose adjustment mechanism is used to adjust the height and rotation angle of the vision camera. A light strip is also arranged between the first horizontal side and the vision camera, and the light strip is fixed on the first horizontal side for supplementary lighting; the top of the integrated bracket is covered with a black opaque acrylic panel to block the top light and avoid the interference of reflection.

[0121] Specifically, the above-mentioned integrated bracket may be a horizontal bracket supported by legs. As Figure 7 shown, it may also be a suspended horizontal bracket or a horizontal bracket with other support structures.

[0122] Specifically, the shape of the above-mentioned integrated bracket can be flexibly determined based on the shape of the starting position.

[0123] Specifically, the starting position of the above-mentioned transfer operation may be on a conveyor belt. Therefore, the shape of the integrated bracket can be specifically set as a rectangle parallel to the conveyor belt. As Figure 7 shown. Then both the vision camera and the light strip can be fixed on the horizontal side parallel to the conveying direction of the conveyor belt.

[0124] Specifically, the brightness of the above-mentioned light strip can be adjusted.

[0125] Optionally, the above-mentioned pose adjustment mechanism may include a servo motor telescopic device.

[0126] Specifically, the aforementioned barcode reader can also be fixed on the integrated bracket.

[0127] In the device transfer system provided by the embodiments of the present invention, by setting an integrated bracket and a pose adjustment mechanism on the integrated bracket to adjust the pose of the vision camera, it can ensure that during the subsequent process of using the vision camera to capture pose images, the vision camera is at the best shooting height and angle of view, avoiding image distortion or blurring caused by the device being too high or too low, so as to ensure that accurate grasping coordinates can be obtained; moreover, the light strip set on the integrated bracket in the embodiments of the present invention can perform supplementary lighting while ensuring uniform illumination of the device surface and avoiding the interference of reflected light; in addition, the top of the integrated bracket in the embodiments of the present invention is covered with a black opaque acrylic panel, which can effectively reduce the interference of external light sources on image acquisition and ensure the image clarity and contrast during the vision recognition process.

[0128] Optionally, the device transfer system provided by the present invention further includes an industrial barcode reader, a robotic arm fixing bracket, and an electric control system.

[0129] Specifically, the above-mentioned electric control system may include a motor, a sensor, a controller, etc.

[0130] Those skilled in the art can clearly understand that for the convenience and conciseness of description, only the above-mentioned division of each functional module is used as an example. In actual applications, the above-mentioned functions can be allocated to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the above-described functional modules can refer to the corresponding process in the foregoing method embodiments and will not be elaborated herein.

[0131] An embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the device transfer method provided in any one of the above embodiments.

[0132] An embodiment of the present invention further provides a computer-readable medium, on which a computer program is stored. When the program is executed by a processor, it implements the device transfer method provided in any one of the above embodiments.

[0133] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the device transfer method as described in any one of the embodiments of the present invention when executed by a processor.

[0134] Next, refer to Figure 8 , which shows a schematic structural diagram of a computer system 800 of an electronic device suitable for implementing the embodiments of the present invention. Figure 8 The shown electronic device is only an example and should not bring any limitation to the functions and usage scope of the embodiments of the present invention.

[0135] As Figure 8 shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage section 808 into the random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, ROM 802, and RAM 803 are connected to each other through a bus 804. The input / output (I / O) interface 805 is also connected to the bus 804.

[0136] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as required. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 810 as required so that a computer program read therefrom is installed into the storage section 808 as required.

[0137] Specifically, according to an embodiment disclosed by the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment disclosed by the present invention includes a computer program product which includes a computer program carried on a computer-readable medium, and the computer program includes program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from the removable medium 811. When the computer program is executed by a central processing unit (CPU) 801, the above-described functions defined in the system of the present invention are performed.

[0138] It should be noted that the computer-readable medium shown in the present invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and this computer-readable medium can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code, and the above module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0140] The modules and / or units involved in the embodiments of the present invention can be implemented in software or in hardware. The described modules and / or units can also be provided in a processor. For example, it can be described as: a processor includes a shooting pose and gripper configuration determination module, a grasping coordinate determination module, and a transfer control module. Among them, the names of these modules do not constitute a limitation to the module itself in some cases.

[0141] As another aspect, the present invention also provides a computer-readable medium, which can be included in the device described in the above embodiments; or it can exist separately without being assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by the device, the device is enabled to: obtain the identification information of the target device at the starting position of the transfer operation, and determine the camera shooting pose and the robotic arm gripper configuration corresponding to the target device based on the identification information; obtain the pose image of the target device with the camera shooting pose through a vision camera, and determine the robotic arm grasping coordinates based on the pose image; and control the robotic arm to grasp the target device for transfer operation based on the robotic arm gripper configuration and the robotic arm grasping coordinates.

[0142] The above specific implementation manners do not constitute a limitation to the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A device transfer method, characterized in that, Including: Obtain the identification information of the target device at the starting position of the transfer operation, and determine the camera shooting pose and robotic arm gripper configuration corresponding to the target device based on the identification information; Obtain the pose image of the target device by the vision camera in the camera shooting pose, and determine the robotic arm grasping coordinates based on the pose image; And Based on the robotic arm gripper configuration and the robotic arm grasping coordinates, control the robotic arm to grasp the target device for transfer operation.

2. The device transfer method according to claim 1, wherein The obtaining the identification information of the target device at the starting position of the transfer operation includes: Obtain the identification image of the target device by the vision camera, and / or obtain the identification code signal of the target device by the barcode reader, to obtain the identification information of the target device.

3. The device transfer method according to claim 1, wherein An integrated bracket is provided directly above the starting position of the transfer operation, and the vision camera is fixed on the first horizontal side of the integrated bracket through a pose adjustment mechanism, and the pose adjustment mechanism is used to adjust the height and rotation angle of the vision camera; A light strip is further provided between the first horizontal side and the vision camera, and the light strip is fixed on the first horizontal side for filling light when obtaining the pose image; the top of the integrated bracket is covered with a black opaque acrylic panel for blocking the top light to avoid reflection interference when obtaining the pose image.

4. The device transfer method according to claim 2, wherein It further includes: Decode the identification code signal to obtain the device code of the device, and generate a device transfer record based on the device code.

5. The device transfer method according to claim 1, wherein Before determining the camera shooting pose and robotic arm gripper configuration corresponding to the target device based on the identification information, the device transfer method further includes: establishing a device database including the category information, size data, shape feature data and weight data of each device to be transferred; The determining the camera shooting pose and robotic arm gripper configuration corresponding to the target device based on the identification information includes: Determine the device category and placement state of the target device based on the identification information; Query the device database based on the device category to obtain the size data, shape feature data and weight data of the target device; and Determine the camera shooting pose based on the size data and the placement state, and determine the robotic arm gripper configuration based on the size data, the shape feature data, the weight data and the placement state.

6. The device transfer method according to claim 5, wherein The robotic arm gripper configuration includes: the opening degree of the robotic arm gripper, the grasping force and the material of the gripper contact surface; The determining the camera shooting pose based on the size data and the placement state, and determining the robotic arm gripper configuration based on the size data, the shape feature data, the weight data and the placement state includes: Determine the grasping surface of the target device based on the placement state, and determine the opening degree of the robotic arm gripper, the grasping force and the material of the gripper contact surface based on the grasping surface, the size data, the shape feature data and the weight data.

7. The device transfer method according to claim 1, wherein Determining the grasping coordinates of the robotic arm based on the pose image includes: Determining the grasping coordinates of the robotic arm based on the pose image through a pre-trained grasping coordinate regression model; The training process of the coordinate regression model includes: Determining the true value of the grasping pose based on the sample image, where the sample image includes multiple image data obtained from the Cornell Grasping Dataset; Inputting the sample image into the coordinate regression model to be trained, and obtaining the predicted value of the grasping pose of the robotic arm through the coordinate regression model; and Determining the function value of the training loss function based on the predicted value of the grasping pose and the true value of the grasping pose, and adjusting the network parameters in the coordinate regression model based on the function value of the training loss function.

8. An equipment transfer system, characterized in that, Including: A vision camera, a robotic arm, and a device transfer device, where the pose of the vision camera is adjustable, and the device transfer device is used to execute the device transfer method according to any one of claims 1 to 7.

9. The device transfer system according to claim 8, wherein Further including: An integrated bracket; The integrated bracket is arranged directly above the starting position of the operation. The vision camera is fixed on the first horizontal side of the integrated bracket through a pose adjustment mechanism, and the pose adjustment mechanism is used to adjust the height and rotation angle of the vision camera; A light strip is further arranged between the first horizontal side and the vision camera, and the light strip is fixed on the first horizontal side for supplementary lighting; the top of the integrated bracket is covered with a black opaque acrylic panel for blocking the top light to avoid reflection interference.

10. A computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the device transfer method according to any one of claims 1 to 7 when executing the program.

Images