Cargo carrying method and device, carrying equipment and readable storage medium

Abstract

The invention provides a cargo carrying method and device, carrying equipment and a readable storage medium, the method is applied to the technical field of vehicles, and the method comprises the steps that a cargo image of a to-be-carried cargo is acquired; determining a target position of a carrying part of the to-be-carried cargo based on the cargo image; and controlling the mechanical arm to move to the carrying part according to the target position so as to carry the to-be-carried goods. According to the method, the target position of the carrying part of the to-be-carried cargo can be accurately and quickly determined by carrying out image recognition on the cargo image, so that the mechanical arm is controlled to carry out accurate carrying, and the cargo carrying accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and more specifically, to a cargo handling method, device, handling equipment and readable storage medium in the field of vehicle technology. Background Art

[0002] In modern logistics and warehouse management, handling equipment is an important handling tool and is widely used in various production scenarios. However, when handling goods, the driver's field of vision may be blocked, which makes it difficult for the mechanical arm of the handling equipment to align with the handling area between the goods, which may not only damage the goods, but may even cause safety accidents such as cargo dumping. Therefore, there is an urgent need for a cargo handling method that can improve the accuracy of cargo handling. Summary of the invention

[0003] The present application provides a cargo handling method, device, handling equipment and readable storage medium, wherein the method can improve the accuracy of cargo handling by the handling equipment.

[0004] In a first aspect, a cargo handling method is provided, the method comprising:

[0005] Acquire cargo images of cargo to be handled;

[0006] Based on the cargo image, determine the target position of the transport part of the cargo to be transported;

[0007] According to the target position, the robot arm is controlled to move to the transporting position to transport the goods to be transported.

[0008] In the embodiment of the present application, after the controller acquires the cargo image of the cargo to be transported, it determines the target position of the transport part of the cargo to be transported based on the cargo image, and controls the mechanical arm to move to the transport part according to the target position to transport the cargo to be transported. In this way, after the controller of the transporting equipment acquires the cargo image, it performs image recognition on the cargo image, and can accurately and quickly determine the target position of the transport part of the cargo to be transported, thereby controlling the mechanical arm to carry out precise transport, reducing the transport errors that may be caused by human misjudgment or improper operation in traditional methods, thereby improving the accuracy of cargo transportation.

[0009] Furthermore, automated control reduces the need for manual operation and the risk of errors and accidents caused by human factors, thereby improving the safety of cargo handling.

[0010] In combination with the first aspect, in some possible implementations, controlling the robot arm to move to the transporting position according to the target position includes:

[0011] Determine the projection position of the robot arm on the surface of the goods to be handled;

[0012] Based on the projection position, the robot arm is controlled to move to the transport location.

[0013] In the embodiment of the present application, the controller can determine the projection position of the robot arm on the surface of the goods to be transported, and control the robot arm to move to the transporting position based on the projection position. By determining the projection position of the robot arm on the surface of the goods, the controller can determine the relative position relationship between the robot arm and the target position where the transporting part of the goods is located, so that the robot arm can be accurately controlled to move to the target position of the transporting part, reducing the time for the robot arm to align with the target position of the goods, thereby improving the transport efficiency of the goods.

[0014] In combination with the first aspect, in some possible implementations, controlling the robot arm to move to the transporting position based on the projection position includes:

[0015] determining a positional deviation between a target position and a projected position;

[0016] When the position deviation is greater than a preset deviation threshold, the robot arm is controlled to move to the target position according to the projection position and the position deviation.

[0017] In the embodiment of the present application, the controller can determine the position deviation between the target position and the projected position; when the position deviation is greater than a preset deviation threshold, the robot arm is controlled to move to the target position according to the projected position and the position deviation. In this way, the controller can accurately calculate the position deviation between the target position and the projected position by comparing the target position and the projected position, thereby controlling the robot arm to move more accurately, which is conducive to reducing errors in the cargo handling process and ensuring that the robot arm can accurately move to the target position.

[0018] In conjunction with the first aspect, in some possible implementations, determining a target position of a transport part of the cargo to be transported based on the cargo image includes:

[0019] In the case where a plurality of goods to be transported are determined based on the goods image, a target goods to be transported is determined from the plurality of goods to be transported;

[0020] Based on the cargo image, a target position of a transporting part of the target cargo to be transported is determined.

[0021] In the embodiment of the present application, when the controller determines multiple goods to be transported based on the cargo image, the controller can determine the target goods to be transported from the multiple goods to be transported; and determine the target position of the transport part of the target goods to be transported based on the cargo image. In this way, the controller can accurately determine the target goods to be transported from the multiple goods to be transported through the preset rules, thereby improving the transport speed. At the same time, based on the cargo image, the target position of the transport part of the target goods to be transported can be determined, reducing the risk of safety accidents caused by misoperation or misjudgment.

[0022] Optionally, controlling the robot arm to move to the transporting position according to the target position includes:

[0023] In combination with the first aspect, in some possible implementations, controlling the robot arm to move to the transporting position according to the target position includes:

[0024] Determine the movement duration of the robot arm;

[0025] When the moving time is longer than the preset time and the position deviation between the target position and the projection position is larger than the preset deviation threshold, the robot arm is controlled to stop moving.

[0026] In the embodiment of the present application, after controlling the robot arm to start moving, the controller can determine the movement time of the robot arm and the position deviation between the target position and the projection position in real time; when the movement time is greater than the preset time and the position deviation is greater than the preset deviation threshold, the controller can control the robot arm to stop moving. By monitoring the movement time and position deviation of the robot arm in real time and stopping the movement of the robot arm in time when the deviation exceeds the preset range, the controller can effectively avoid the problem of inaccurate positioning of the robot arm due to continuous error accumulation, avoid invalid or inefficient handling operations, and thus improve the overall handling efficiency.

[0027] In combination with the first aspect, in some possible implementations, after controlling the robotic arm to stop moving, the method further includes:

[0028] Display the real-time images of the goods to be transported and the robotic arm;

[0029] A virtual extension portion of the robot arm is displayed in the real-time image; wherein the extension portion is located at the front end of the robot arm displayed in the real-time image, and is used to indicate the relative position between the robot arm and the handling portion.

[0030] In the embodiment of the present application, after the controller controls the robot arm to stop moving, it can display a real-time image of the cargo to be transported and the robot arm on the front display screen; and display the virtual extension part of the robot arm in the real-time image. In this way, when the automatic control of the robot arm movement is abnormal, the operator can intuitively understand the relative position relationship between the front end of the robot arm and the target cargo by observing the virtual extension part in the real-time image, so that the operator can intuitively observe the vertical position of the robot arm, which is conducive to the operator manually controlling the robot arm to align with the target position and reduce the possibility of misoperation.

[0031] In conjunction with the first aspect, in some possible implementations, displaying a virtual extension portion of the robotic arm in a real-time image includes:

[0032] Determine a plurality of preset coordinates; wherein the preset coordinates are located at the front end of the robotic arm displayed in the real-time image;

[0033] A virtual extension portion is displayed at the location of a plurality of preset coordinates.

[0034] In the embodiment of the present application, the controller determines a plurality of preset coordinates and displays a virtual extension portion at the location of the plurality of preset coordinates. By displaying the virtual extension portion at the front end of the robotic arm, the operator can intuitively see the potential extension range or reachable area of ​​the robotic arm, which helps the operator to more accurately judge the relative position between the robotic arm and the goods to be transported, thereby improving the accuracy and efficiency of the operation.

[0035] In a second aspect, a cargo handling device is provided, the device comprising:

[0036] An acquisition module, used for acquiring cargo images of the cargo to be transported;

[0037] A determination module, used to determine a target position of a transport part of the cargo to be transported based on the cargo image;

[0038] The control module is used to control the robot arm to move to the transporting position according to the target position to transport the goods to be transported.

[0039] In a third aspect, there is provided a handling device, comprising a memory for storing executable program code;

[0040] The processor is used to call and run the executable program code from the memory, so that the electronic device executes the method in any possible implementation manner of the first aspect above.

[0041] In a fourth aspect, an executable program code product is provided, the executable program code product comprising: an executable program code, when the executable program code is run on a computer, the computer executes the method in any possible implementation of the first aspect above.

[0042] In a fifth aspect, a readable storage medium is provided, which stores an executable program code. When the executable program code runs on a computer, the computer executes the method in any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of steps of a cargo handling method provided by an embodiment of the present application;

[0044] Figure 2 It is a scene schematic diagram of a front-end display interface of a cargo handling method provided in an embodiment of the present application;

[0045] Figure 3 This is a schematic diagram of an application scenario of a cargo handling method provided in an embodiment of the present application;

[0046] Figure 4 It is a scene schematic diagram of a front-end display interface of a cargo handling method provided in an embodiment of the present application;

[0047] Figure 5 It is a flow chart of steps of a cargo handling method provided by an embodiment of the present application;

[0048] Figure 6 It is a structural schematic diagram of a cargo handling device provided in an embodiment of the present application;

[0049] Figure 7 It is a structural schematic diagram of a handling device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solution in the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0051] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0052] At present, in modern logistics and warehouse management, handling equipment is an important handling tool and is widely used in various production scenarios. However, when handling goods, the driver's field of vision may be blocked, which makes it difficult for the mechanical arm of the handling equipment to align with the handling area between the goods, which may not only damage the goods, but may even cause safety accidents such as cargo dumping. Therefore, there is an urgent need for a cargo handling method that can improve the accuracy of cargo handling.

[0053] In order to solve the above technical problems, an embodiment of the present application provides a cargo handling method, which can be executed by a controller of a handling device. After acquiring a cargo image of the cargo to be handled, the controller determines the target position of the handling part of the cargo to be handled based on the cargo image, and controls the robot arm to move to the handling part according to the target position to handle the cargo to be handled. In this way, after acquiring the cargo image, the controller of the handling device performs image recognition on the cargo image, and can accurately and quickly determine the target position of the handling part of the cargo to be handled, thereby controlling the robot arm to carry out precise handling, thereby improving the accuracy of cargo handling.

[0054] Furthermore, automated control reduces the need for manual operation and the risk of errors and accidents caused by human factors, thereby improving the safety of cargo handling.

[0055] See also Figure 1 , Figure 1 is a flowchart of the steps of a cargo handling method provided by an embodiment of the present application. The execution subject of the method may be a controller of a handling device, such as Figure 1 As shown, the method may include the following steps.

[0056] S101, obtaining a cargo image of the cargo to be transported.

[0057] S102: Determine a target position of a transporting part of the cargo to be transported based on the cargo image.

[0058] S103, controlling the robot arm to move to the transporting position according to the target position to transport the goods to be transported.

[0059] Among them, the goods to be transported refer to items or cargo boxes that need to be moved or transported; the cargo image refers to a real-time image of the goods to be transported obtained by a certain imaging device (such as a camera, a wide-angle camera, etc.), and the real-time image may include but is not limited to data such as the shape, size, quantity, and location information of the goods to be transported, which is used for subsequent image processing and recognition.

[0060] The handling part refers to a specific location or area on the cargo to be handled that is suitable for handling. The handling part can be determined based on the structure, weight distribution and handling requirements of the cargo. For example, if the cargo to be handled is a cargo box, the handling part may be the bottom gap area of ​​the cargo box; for irregularly shaped objects, the handling part may be its balance point or the two sides that are easy to hold.

[0061] The target position refers to the coordinate position of the transport part in the cargo image determined by the image recognition algorithm. For example, the target position can be the pixel area of ​​the transport part in the image pixel coordinate system; it can also be the three-dimensional space area of ​​the transport part in the three-dimensional space coordinate system, and the comparison is not limited.

[0062] The robotic arm refers to the mechanical gripper of the handling equipment. For example, if the handling equipment is a forklift, the robotic arm may be the fork of the forklift; if the handling equipment is a handling robot, the robotic arm may be the end effector of the robot, such as the gripper or suction cup.

[0063] In this embodiment, the controller can obtain the cargo image of the cargo to be transported through the optical device, and determine the target position of the transport part of the cargo to be transported based on the cargo image and the preset image recognition algorithm. After that, the controller can control the mechanical arm of the transport device to move to the transport part according to the target position to transport the cargo to be transported.

[0064] For example, taking a forklift as an example, the operator can install a wide-angle camera on the forklift or fork in advance. When the forklift moves near the goods to be transported, the wide-angle camera can automatically capture the cargo image of the goods to be transported. After that, the controller can obtain the cargo image and identify the bottom gap area of ​​the goods to be transported and the target pixel area of ​​the bottom gap area in the image pixel coordinate system based on the preset image recognition algorithm. The controller controls the forklift fork to move to the target pixel area with a precise path and speed. When the fork is accurately inserted into the bottom gap area of ​​the cargo, the controller will issue a command to make the fork rise and fix the cargo, and transport it to the specified location.

[0065] In the embodiment of the present application, after the controller acquires the cargo image of the cargo to be transported, it determines the target position of the transport part of the cargo to be transported based on the cargo image, and controls the mechanical arm to move to the transport part according to the target position to transport the cargo to be transported. In this way, after the controller of the transporting equipment acquires the cargo image, it performs image recognition on the cargo image, and can accurately and quickly determine the target position of the transport part of the cargo to be transported, thereby controlling the mechanical arm to carry out precise transport, reducing the transport errors that may be caused by human misjudgment or improper operation in traditional methods, thereby improving the accuracy of cargo transportation.

[0066] Furthermore, automated control reduces the need for manual operation and the risk of errors and accidents caused by human factors, thereby improving the safety of cargo handling.

[0067] See also Figure 2 , Figure 2 It is a scene schematic diagram of a front-end display interface of a cargo handling method provided in an embodiment of the present application.

[0068] For example, Figure 2The scene 200 in the figure includes goods to be transported 201 and a target position 202. The operator may install a wide-angle camera on the transport device or the robot arm in advance, and when the transport device moves near the goods to be transported 201, the wide-angle camera may automatically capture the goods image of the goods to be transported 201. Afterwards, the controller may acquire the goods image and identify the target position 202 of the goods to be transported based on a preset image recognition algorithm.

[0069] Optionally, controlling the robot arm to move to the transporting position according to the target position includes:

[0070] Determine the projection position of the robot arm on the surface of the goods to be handled;

[0071] Based on the projection position, the robot arm is controlled to move to the transport location.

[0072] The projection position refers to the vertical projection point of the robot arm on the surface of the cargo to be transported. For example, the projection position can be the pixel area of ​​the robot arm in the image pixel coordinate system; it can also be the three-dimensional space area of ​​the robot arm in the three-dimensional space coordinate system, which is not limited.

[0073] In this embodiment, the controller can first obtain the height of the robot arm and the distance between the robot arm and the goods to be transported, and determine the projection position of the robot arm on the surface of the goods to be transported according to the height and distance. Further, the controller can control the robot arm of the transport equipment to move to the transport position according to the projection position to transport the goods to be transported.

[0074] For example, taking a forklift as an example, the operator can pre-install an ultrasonic sensor and a time-of-flight (TOF) laser sensor on the forklift or fork, wherein the ultrasonic sensor is used to measure the distance between the fork and the cargo, and the TOF laser sensor is used to measure the height of the fork. When the forklift moves near the cargo to be transported, the ultrasonic sensor can automatically measure the distance between the fork and the cargo, and the TOF laser sensor can measure the height of the fork. After that, the controller can obtain the distance between the fork and the cargo and the height of the fork, and determine the projection position of the robot arm on the surface of the cargo to be transported based on the height and distance. Furthermore, the controller can control the movement of the robot arm so that the position error between the projection position of the robot arm and the target position where the transporting part is located is within a preset error threshold, so that the robot arm can accurately transport the cargo to be transported.

[0075] In the embodiment of the present application, the controller can determine the projection position of the robot arm on the surface of the goods to be transported, and control the robot arm to move to the transporting position based on the projection position. By determining the projection position of the robot arm on the surface of the goods, the controller can determine the relative position relationship between the robot arm and the target position where the transporting part of the goods is located, so that the robot arm can be accurately controlled to move to the target position of the transporting part, reducing the time for the robot arm to align with the target position of the goods, thereby improving the transport efficiency of the goods.

[0076] Optionally, controlling the robot arm to move to the handling position based on the projection position includes:

[0077] determining a positional deviation between a target position and a projected position;

[0078] When the position deviation is greater than a preset deviation threshold, the robot arm is controlled to move to the target position according to the projection position and the position deviation.

[0079] In this embodiment, after determining the target position and the projection position, the controller determines whether the target position and the projection position are located in the same coordinate system. If they are located in the same coordinate system, the controller determines the position deviation between the target position and the projection position; if the position deviation is greater than a preset deviation threshold, the controller controls the robot arm to move to the target position according to the projection position and the position deviation. If they are not located in the same coordinate system, the controller converts the target position and the projection position to the same coordinate system according to a preset conversion rule, and then the controller can determine the position deviation between the target position and the projection position; if the position deviation is greater than a preset deviation threshold, the controller controls the robot arm to move to the target position according to the projection position and the position deviation.

[0080] Exemplarily, after determining the target position and the projection position, the controller determines that the target position is the pixel area of ​​the transport part in the image pixel coordinate system, and the projection position is the pixel area of ​​the robot arm in the image pixel coordinate system, that is, the two are located in the same coordinate system. Afterwards, the controller can determine the position deviation between the target position and the projection position; if the position deviation is greater than the preset deviation threshold, the controller can control the robot arm to move to the target position according to the projection position and the position deviation. If the position deviation is not greater than the preset deviation threshold, indicating that the robot arm is located near the target position where the transport part is located, the robot arm can directly transport the goods without moving.

[0081] In another exemplary embodiment, after determining the target position and the projection position, the controller determines that the target position is the pixel area of ​​the transporting part in the image pixel coordinate system, and the projection position is the three-dimensional space area of ​​the robot arm in the three-dimensional space coordinate system, that is, the two are located in different coordinate systems. The controller can convert the three-dimensional space area of ​​the projection position into the pixel area on the image pixel coordinate system according to a preset conversion formula. Afterwards, the controller can determine the position deviation between the target position and the projection position; if the position deviation is greater than the preset deviation threshold, the controller can control the robot arm to move to the target position according to the projection position and the position deviation.

[0082] For example, the controller may determine a target three-dimensional coordinate from the three-dimensional space region of the projection position, and convert the target three-dimensional coordinate into a target pixel coordinate in the image pixel coordinate system, wherein the conversion formula may be expressed as:

[0083]

[0084] in, is the camera's internal parameter matrix, which is the camera's inherent property; [RT] is the camera's external parameter matrix, which represents the rotation and translation relationship between the camera coordinate system and the world coordinate system; where R is the rotation matrix and T is the translation matrix. Both the internal parameter matrix and the external parameter matrix can be obtained by Zhang Zhengyou's calibration method. u represents the coordinate value of the target pixel coordinate along the X-axis in the image pixel coordinate system; v represents the coordinate value of the target pixel coordinate along the Y-axis in the image pixel coordinate system. Z c is the coordinate value of the target's three-dimensional coordinates along the Z axis in the camera coordinate system; f represents the focal length of the camera. w is the coordinate value of the target three-dimensional coordinate along the X axis in three-dimensional space; Y w is the coordinate value of the target three-dimensional coordinate along the Y axis in three-dimensional space; Z w The coordinate value of the target's three-dimensional coordinates in the three-dimensional space along the Z axis; u0 represents the coordinate value of the origin coordinate of the image's physical coordinate system along the X axis; v0 represents the coordinate value of the origin coordinate of the image's physical coordinate system along the Y axis.

[0085] Of course, the conversion formula can also be expressed as other variations or equivalent formulas of the above formula (1).

[0086] In another exemplary embodiment, after determining the target position and the projection position, the controller determines that the target position is the pixel area of ​​the transporting part in the image pixel coordinate system, and the projection position is the three-dimensional space area of ​​the robotic arm in the three-dimensional space coordinate system, that is, the two are located in different coordinate systems. The controller can convert the pixel area of ​​the transporting part in the image pixel coordinate system into the three-dimensional space area of ​​the transporting part in the three-dimensional space coordinate system according to a preset conversion formula. Afterwards, the controller can determine the position deviation between the target position and the projection position; if the position deviation is greater than the preset deviation threshold, the controller can control the robotic arm to move to the target position according to the projection position and the position deviation.

[0087] In the embodiment of the present application, the controller can determine the position deviation between the target position and the projected position; when the position deviation is greater than a preset deviation threshold, the robot arm is controlled to move to the target position according to the projected position and the position deviation. In this way, the controller can accurately calculate the position deviation between the target position and the projected position by comparing the target position and the projected position, thereby controlling the robot arm to move more accurately, which is conducive to reducing errors in the cargo handling process and ensuring that the robot arm can accurately move to the target position.

[0088] Optionally, determining a target position of a transport part of the cargo to be transported based on the cargo image includes:

[0089] In the case where a plurality of goods to be transported are determined based on the goods image, a target goods to be transported is determined from the plurality of goods to be transported;

[0090] Based on the cargo image, a target position of a transporting part of the target cargo to be transported is determined.

[0091] In this embodiment, when the controller determines multiple goods to be transported based on the goods image, the controller can determine the target goods to be transported from the multiple goods to be transported according to the preset transport rules. Afterwards, the controller can determine the target position of the transport part of the target goods to be transported according to the goods image captured by the wide-angle camera and the preset target image recognition algorithm.

[0092] For example, taking a forklift as an example, the operator can pre-set the rules for handling goods, and the handling rules can be set according to the order in which the goods are placed or the priority of the goods. After the controller obtains the image of the goods to be handled, it can determine the target goods to be handled in the order from top to bottom and from left to right according to the order in which the goods are placed. Afterwards, the controller can identify the bottom gap area of ​​the target goods to be handled and the target pixel area of ​​the bottom gap area in the image pixel coordinate system based on the cargo image captured by the wide-angle camera and the preset target image recognition algorithm.

[0093] In another exemplary embodiment, the operator can configure the type of the current cargo box and the number of items to be transported at a time, and the operator can change the default configuration on the front display screen. For example, the goods to be transported are arranged in a vertical arrangement. If the default configuration is that the type of the current cargo box is box type 01 and the number of items to be transported at a time is 1, then after the controller obtains the image of the goods to be transported, it can first select the top cargo box as the target cargo to be transported according to the arrangement of the cargo boxes. Afterwards, the controller can identify the bottom gap area of ​​the target cargo to be transported and the target pixel area of ​​the bottom gap area in the image pixel coordinate system based on the cargo image captured by the wide-angle camera and the preset target image recognition algorithm.

[0094] In another example, the operator can configure the type of the current cargo box and the number of items to be transported at a time, and the operator can change the default configuration on the front display screen. If the operator configures the type of the current cargo box to be box type 01 and the number of items to be transported at a time to be 2, then after the controller obtains the image of the goods to be transported, it can select the two cargo boxes at the top arranged vertically as the target goods to be transported according to the configuration information of the operator. Afterwards, the controller can identify the bottom gap area of ​​the second cargo box in the target goods to be transported and the target pixel area of ​​the bottom gap area in the image pixel coordinate system based on the cargo image captured by the wide-angle camera and the preset target image recognition algorithm.

[0095] In the embodiment of the present application, when the controller determines multiple goods to be transported based on the cargo image, the controller can determine the target goods to be transported from the multiple goods to be transported; and determine the target position of the transport part of the target goods to be transported based on the cargo image. In this way, the controller can accurately determine the target goods to be transported from the multiple goods to be transported through the preset rules, thereby improving the transport speed. At the same time, based on the cargo image, the target position of the transport part of the target goods to be transported can be determined, reducing the risk of safety accidents caused by misoperation or misjudgment.

[0096] Optionally, controlling the robot arm to move to the transporting position according to the target position includes:

[0097] Determine the movement duration of the robot arm;

[0098] When the moving time is longer than the preset time and the position deviation between the target position and the projection position is larger than the preset deviation threshold, the robot arm is controlled to stop moving.

[0099] The moving time refers to the total time from the start of controlling the movement of the robotic arm to the current moment.

[0100] In this embodiment, after controlling the mechanical arm to start moving, the controller can start timing through the timer and determine the position deviation between the target position and the projected position in real time. At the current moment, the controller can determine the movement time of the mechanical arm. If the movement time of the mechanical arm is determined to be greater than the preset time, the controller can determine the position deviation between the target position and the projected position. If the position deviation between the target position and the projected position is greater than the preset deviation threshold, the controller controls the mechanical arm to stop moving. If the position deviation between the target position and the projected position is less than or equal to the preset deviation threshold, the controller can control the mechanical arm to transport the goods to be transported.

[0101] Exemplarily, if the controller determines that the target position of the robotic arm is a pixel coordinate in the image pixel coordinate system, which is (100, 200), the preset duration is 5 seconds, and the position deviation threshold is (5, 5) pixel coordinates. After controlling the robotic arm to start moving, the controller can monitor the movement duration of the robotic arm and the position deviation between the target position and the projection position in real time. At the current moment, the controller determines that the movement duration of the robotic arm is 6 seconds, and compares the movement duration with the preset duration of 5 seconds to determine that the movement duration is greater than the preset duration. Afterwards, the controller determines that the position deviation between the target position and the projection position at the current moment is (6, 10), and compares the position deviation with the preset deviation threshold to determine that the position deviation is greater than the preset deviation threshold. At this time, the controller can control the robotic arm to stop moving.

[0102] In another exemplary embodiment, after controlling the robot arm to start moving, the controller can monitor the movement time of the robot arm and the position deviation between the target position and the projected position in real time. At the current moment, the controller determines that the movement time of the robot arm is 6 seconds, and compares the movement time with the preset time of 5 seconds to determine that the movement time is greater than the preset time. Afterwards, the controller determines that the position deviation between the target position and the projected position at the current moment is (4, 5), and compares the position deviation with the preset deviation threshold to determine that the position deviation is less than the preset deviation threshold. At this time, the controller can control the robot arm to carry the goods to be carried.

[0103] In the embodiment of the present application, after the controller controls the robot arm to start moving, it can determine the movement time of the robot arm and the position deviation between the target position and the projection position in real time; when the movement time is greater than the preset time and the position deviation is greater than the preset deviation threshold, the controller can control the robot arm to stop moving. In this way, the controller can effectively avoid the problem of inaccurate positioning of the robot arm due to continuous error accumulation, avoid invalid or inefficient handling operations, and improve the overall handling efficiency by monitoring the movement time and position deviation of the robot arm in real time and stopping the movement of the robot arm in time when the deviation exceeds the preset range.

[0104] Optionally, after controlling the robotic arm to stop moving, the method further includes:

[0105] Display the real-time images of the goods to be transported and the robotic arm;

[0106] A virtual extension portion of the robot arm is displayed in the real-time image; wherein the extension portion is located at the front end of the robot arm displayed in the real-time image, and is used to indicate the relative position between the robot arm and the handling portion.

[0107] Among them, the real-time image refers to the image of the goods to be transported and the actual robotic arm image obtained by a video camera; the virtual extended part refers to the extended part of the front end of the actual robotic arm displayed in the real-time image, which can also be called a virtual robotic arm.

[0108] In this embodiment, the operator can install a wide-angle camera on the handling equipment or the mechanical arm of the handling equipment, and collect images of the goods to be handled and the mechanical arm in real time. After that, the controller can obtain the real-time image captured by the wide-angle camera after controlling the mechanical arm to stop moving, and display the real-time image on the front display screen. At the same time, the front end of the actual mechanical arm displayed in the real-time image is superimposed with the virtual extension part, and the virtual extension part moves dynamically with the movement of the mechanical arm, which is used to more clearly indicate the relative position between the mechanical arm and the handling part.

[0109] In the embodiment of the present application, after the controller controls the robot arm to stop moving, it can display a real-time image of the cargo to be transported and the robot arm on the front display screen; and display the virtual extension part of the robot arm in the real-time image. In this way, when the automatic control of the robot arm movement is abnormal, the operator can intuitively understand the relative position relationship between the front end of the robot arm and the target cargo by observing the virtual extension part in the real-time image, so that the operator can intuitively observe the vertical position of the robot arm, which is conducive to the operator manually controlling the robot arm to align with the target position and reduce the possibility of misoperation.

[0110] Optionally, a virtual extension of the robot arm is displayed in the real-time image, including:

[0111] Determine a plurality of preset coordinates; wherein the preset coordinates are located at the front end of the robotic arm displayed in the real-time image;

[0112] A virtual extension portion is displayed at the location of a plurality of preset coordinates.

[0113] Among them, the preset coordinates refer to multiple coordinates at the front end of the robotic arm displayed in the real-time image. The preset coordinates can be pixel coordinates in the image pixel coordinate system; they can also be three-dimensional coordinates in the spatial three-dimensional coordinate system, and the comparison is not limited.

[0114] In this embodiment, the controller can determine a plurality of preset coordinates at the front end of the robotic arm displayed in the real-time image. Afterwards, it is determined whether the preset coordinates are pixel coordinates on the image pixel coordinate system. If the preset coordinates are pixel coordinates on the image pixel coordinate system, the points corresponding to the positions of the plurality of preset coordinates are used as anchor points to determine the virtual extension part, and the virtual extension part is superimposed on the front end of the robotic arm displayed in the real-time image, and displayed on the front display screen. If the preset coordinates are three-dimensional coordinates in a three-dimensional spatial coordinate system, the controller converts the preset coordinates into pixel coordinates on the image pixel coordinate system according to a preset conversion rule. Afterwards, the virtual extension part is determined, and the virtual extension part is superimposed on the front end of the robotic arm displayed in the real-time image, and displayed on the front display screen.

[0115] Exemplarily, after displaying the real-time image obtained by photographing the cargo to be transported and the robotic arm, the controller can determine a plurality of preset coordinates at the front end of the robotic arm displayed in the real-time image. If the controller determines that the preset coordinates are pixel coordinates on the image pixel coordinate system, for example, the preset coordinates are (100, 200), (150, 250), (200, 300), the controller uses the preset coordinates as anchor points to determine a straight-line virtual extension portion, for example, a straight line from (100, 200) to (200, 300). Afterwards, the virtual extension portion is superimposed on the front end of the robotic arm displayed in the real-time image and displayed on the front display screen.

[0116] In another exemplary embodiment, after displaying the real-time image obtained by photographing the cargo to be handled and the robotic arm, the controller can determine a plurality of preset coordinates at the front end of the robotic arm displayed in the real-time image. If the controller determines that the preset coordinates are three-dimensional coordinates in a three-dimensional spatial coordinate system, the controller converts the preset coordinates into pixel coordinates in the image pixel coordinate system according to the perspective projection transformation formula. Then, the controller can determine the virtual extension part by interpolation or fitting method based on the converted pixel coordinates. The virtual extension part is superimposed on the front end of the robotic arm displayed in the real-time image and displayed on the front display screen.

[0117] In the embodiment of the present application, the controller determines a plurality of preset coordinates and displays a virtual extension part at the position where the plurality of preset coordinates are located. In this way, the controller can display the virtual extension part at the front end of the robotic arm in the real-time image, so that the operator can more intuitively and clearly understand the potential extension range of the robotic arm and the motion trajectory of the robotic arm, which is conducive to the operator to more accurately judge the relative position between the robotic arm and the goods to be transported, thereby improving the accuracy and efficiency of the cargo handling operation.

[0118] See also Figure 3 , Figure 3It is a schematic diagram of an application scenario of a cargo handling method provided in an embodiment of the present application. The scenario may include: a sensor 10, a computing unit 20, a controller 30 and a display 40; the sensor 10 may include an ultrasonic sensor 101, a laser sensor 102 and a visual sensor 103.

[0119] Among them, the ultrasonic sensor 101 can measure the distance between the robot arm and the goods; the laser sensor 102 can measure the height of the robot arm; and the visual sensor 103 can collect the cargo image of the goods to be transported. After the sensor 10 collects the distance between the robot arm and the goods, the height of the robot arm, and the cargo image, the information can be sent to the calculation unit 20, and the calculation unit 20 analyzes and calculates the various information sent by the sensor 10 to determine the target position of the transport part of the goods to be transported. The calculated information is sent to the controller 30 so that the controller 30 controls the robot arm to move to the transport part according to the target position to transport the goods to be transported. At the same time, the goods to be transported, the robot arm, and the virtual extension part of the robot arm are displayed on the display 40.

[0120] See also Figure 4 , Figure 4 It is a scene schematic diagram of a front-end display interface of a cargo handling method provided in an embodiment of the present application.

[0121] For example, Figure 4 The scene 400 includes goods to be transported 401, a target position 402, a robotic arm 403, a virtual extension portion 404, a virtual button 405 for selecting a box type, a virtual button 406 for the number of items transported at a time, a virtual button 407 for changing the default configuration, and a virtual button 408 for automatic alignment.

[0122] For example, after the operator moves the transport device to the preset position, the operator clicks the virtual button 408 for automatic alignment, and the controller can execute the above steps S101 to S103, so that the controller controls the robot arm to transport the goods to be transported 401. Correspondingly, the operator can also manually control the robot arm to transport the goods to be transported 401 through the virtual extension part 404 in the front display interface.

[0123] When the operator clicks the virtual button 407 for changing the default configuration, the default configuration can be changed by selecting the virtual button 405 for the box type and the virtual button 406 for the single handling quantity. For example, if the operator configures the current box type as box type 01 and the single handling quantity as 1, the controller can first select the top box as the target to-be-handled goods according to the arrangement of the boxes after acquiring the image of the goods to be handled. Afterwards, the controller can identify the bottom gap area of ​​the target to-be-handled goods and the target pixel area of ​​the bottom gap area in the image pixel coordinate system according to the goods image captured by the wide-angle camera and the preset target image recognition algorithm.

[0124] See also Figure 5 , Figure 5 This is a flow chart of the steps of a cargo handling method provided by an embodiment of the present application. Figure 5 As shown, the method may include the following steps.

[0125] S501, determining the distance between the robot arm and the cargo through an ultrasonic sensor.

[0126] S502, determining the height of the robot arm through a laser sensor.

[0127] S503, obtaining a cargo image of the cargo to be transported by using a wide-angle camera.

[0128] S504: Determine a target position of a transporting part of the cargo to be transported based on the cargo image.

[0129] Exemplarily, the controller may determine the target position of the transport part of the cargo to be transported based on the cargo image and a preset image recognition algorithm.

[0130] S505, determining the projection position of the robot arm on the surface of the cargo to be transported.

[0131] Exemplarily, the controller may obtain the height of the robotic arm and the distance between the robotic arm and the goods to be transported, and determine the projection position of the robotic arm on the surface of the goods to be transported according to the height and the distance.

[0132] S506, controlling the robot arm to move to the transporting position based on the projection position.

[0133] Exemplarily, the controller can control the movement of the robotic arm so that the position error between the projected position of the robotic arm and the target position where the transporting part is located is within a preset error threshold, so that the robotic arm can accurately transport the goods to be transported.

[0134] S507, determining the movement duration of the robot arm.

[0135] S508, determining whether the moving time is greater than a preset time, and whether the position deviation between the target position and the projection position is greater than a preset deviation threshold, if so, executing step S409.

[0136] S509, controlling the robotic arm to stop moving.

[0137] S510, displaying a real-time image obtained by photographing the goods to be transported and the robot arm.

[0138] S511, displaying the virtual extended portion of the robot arm in the real-time image.

[0139] Exemplarily, the extension part is located at the front end of the robot arm displayed in the real-time image, which is used to indicate the relative position between the robot arm and the handling part. After controlling the robot arm to stop moving, the controller can obtain the real-time image captured by the wide-angle camera and display the real-time image on the front display screen. At the same time, the front end of the actual robot arm displayed in the real-time image is superimposed with the virtual extension part, and the virtual extension part moves dynamically with the movement of the robot arm, which is used to indicate the relative position between the robot arm and the handling part.

[0140] It should be noted that the achievable methods and technical effects of each step in the embodiments of the present application can refer to the relevant contents in the above embodiments and will not be repeated here.

[0141] See also Figure 6 , Figure 6 It is a structural schematic diagram of a cargo handling device provided in an embodiment of the present application. The cargo handling device may be arranged in a controller and may include: an acquisition module 601, a determination module 602 and a control module 603.

[0142] An acquisition module 601 is used to acquire a cargo image of the cargo to be transported;

[0143] A determination module 602 is used to determine a target position of a transport part of the cargo to be transported based on the cargo image;

[0144] The control module 603 is used to control the robot arm to move to the transporting position according to the target position to transport the goods to be transported.

[0145] Optionally, the control module 603 is specifically used to determine the projection position of the robot arm on the surface of the goods to be transported; and control the robot arm to move to the transporting position based on the projection position.

[0146] Optionally, the control module 603 is specifically used to determine a position deviation between the target position and the projection position; when the position deviation is greater than a preset deviation threshold, the robot arm is controlled to move to the target position according to the projection position and the position deviation.

[0147] Optionally, the determination module 602 is specifically configured to determine a target cargo to be transported from among the multiple cargoes to be transported when multiple cargoes to be transported are determined based on the cargo image; and determine a target position of a transport part of the target cargo to be transported based on the cargo image.

[0148] Optionally, the control module 603 is specifically used to determine the movement time of the robotic arm; when the movement time is greater than a preset time and the position deviation between the target position and the projection position is greater than a preset deviation threshold, the robotic arm is controlled to stop moving.

[0149] Optionally, after the control module 603, the cargo handling device may further include: a first display module and a second display module;

[0150] The first display module is used to display real-time images of the goods to be transported and the robotic arm;

[0151] The second display module is used to display the virtual extension part of the robot arm in the real-time image; wherein the extension part is located at the front end of the robot arm displayed in the real-time image, and is used to indicate the relative position between the robot arm and the handling part.

[0152] Optionally, the second display module is specifically used to determine a plurality of preset coordinates; wherein the preset coordinates are located at the front end of the robotic arm displayed in the real-time image; and a virtual extension portion is displayed at the location of the plurality of preset coordinates.

[0153] The cargo handling device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0154] See also Figure 7 , Figure 7 It is a structural diagram of a controller provided in an embodiment of the present application.

[0155] For example, Figure 7 As shown, the controller 700 includes: a memory 701 and a processor 702, wherein the memory 701 stores an executable program code 7011, and the processor 702 is used to call and execute the executable program code 7011 to perform a cargo handling method.

[0156] In addition, an embodiment of the present application also protects a handling device, which may include a memory and a processor, wherein the memory stores an executable program code, and the processor is used to call and execute the executable program code to execute a cargo handling method provided by an embodiment of the present application.

[0157] This embodiment also provides a readable storage medium, in which executable program code is stored. When the executable program code is run on a computer, the computer executes the above-mentioned related method steps to implement a cargo handling method provided by the above-mentioned embodiment.

[0158] This embodiment also provides an executable program code product. When the executable program code product runs on a computer, the computer is enabled to execute the above-mentioned related steps to implement a cargo handling method provided by the above embodiment.

[0159] Among them, the device, readable storage medium, executable program code product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0160] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0161] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0162] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A cargo handling method, characterized in that: The method comprises: Acquire cargo images of cargo to be handled; Determining a target position of a transport portion of the cargo to be transported based on the cargo image; The robot arm is controlled to move to the transporting position according to the target position to transport the goods to be transported.

2. The method according to claim 1, characterized in that: The controlling the robot arm to move to the transporting position according to the target position comprises: Determining the projection position of the robot arm on the surface of the cargo to be transported; The robot arm is controlled to move to the transporting location based on the projection position.

3. The method according to claim 2, characterized in that The controlling the robot arm to move to the transporting position based on the projection position comprises: determining a positional deviation between the target position and the projection position; When the position deviation is greater than a preset deviation threshold, the robot arm is controlled to move to the target position according to the projection position and the position deviation.

4. The method according to claim 1, characterized in that: The step of determining a target position of a transporting part of the cargo to be transported based on the cargo image includes: In a case where a plurality of the goods to be transported are determined based on the goods image, determining a target goods to be transported from the plurality of goods to be transported; Based on the cargo image, a target position of a transporting part of the target cargo to be transported is determined.

5. The method according to claim 1, characterized in that The controlling the robot arm to move to the transporting position according to the target position comprises: Determining the movement duration of the robotic arm; When the moving time is longer than a preset time and the position deviation between the target position and the projection position is larger than a preset deviation threshold, the robot arm is controlled to stop moving.

6. The method according to claim 5, characterized in that After controlling the mechanical arm to stop moving, the method further includes: Displaying a real-time image obtained by photographing the goods to be transported and the robotic arm; A virtual extension portion of the robotic arm is displayed in the real-time image, wherein the extension portion is located at the front end of the robotic arm displayed in the real-time image, and is used to indicate the relative position between the robotic arm and the transporting portion.

7. The method according to claim 6, characterized in that The step of displaying the virtual extension part of the robotic arm in the real-time image comprises: Determine a plurality of preset coordinates; wherein the preset coordinates are located at the front end of the robotic arm displayed in the real-time image; The virtual extension part is displayed at the positions where the plurality of preset coordinates are located.

8. A cargo handling device, characterized in that: The device comprises: An acquisition module, used for acquiring cargo images of the cargo to be transported; A determination module, used for determining a target position of a transport part of the cargo to be transported based on the cargo image; The control module is used to control the robot arm to move to the transporting position according to the target position, so as to transport the goods to be transported.

9. A handling device, characterized in that: The handling equipment comprises: A memory for storing executable program codes; A processor, configured to call and run the executable program code from the memory, so that the handling device executes the method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores executable program codes, and when the executable program codes are executed, the method according to any one of claims 1 to 7 is implemented.

Images