Container control method and electronic equipment
By automatically calibrating the photographic pose of the robotic arm, the target photographic pose is obtained, which solves the problem of insufficient image acquisition accuracy of the vision component, realizes higher precision spatial visual positioning and more accurate cargo handover operation, and improves handover efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- MEITUAN TECH CO LTD
- Filing Date
- 2024-10-30
- Publication Date
- 2026-05-05
AI Technical Summary
The existing technology for spatial visual positioning based on images of preset recognition beacons acquired by vision components is not accurate enough, which affects the accuracy of cargo handover operations.
By automatically calibrating the photographic pose of the robotic arm, the target photographic pose is obtained, and images of visual recognition beacons are acquired using vision components, thereby improving the accuracy of spatial visual positioning.
This improved the accuracy of the robotic arm in performing handover operations and increased the efficiency of cargo handover.
Smart Images
Figure CN121974074A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of cargo handling technology, specifically to a container control method and electronic equipment. Background Technology
[0002] Performing a handover operation for goods refers to receiving goods and transporting them to a designated location. Before conducting the handover operation, it is necessary to spatially locate the handover location based on visual spatial positioning technology in order to achieve a precise handover operation based on the positioning results.
[0003] In the process of spatially locating the cargo handover position using visual spatial positioning technology, it is necessary to use a vision component to acquire images of preset identification beacons, and then use these images to achieve spatial visual positioning of the cargo handover position. In related technologies, the positioning accuracy of spatial visual positioning based on these preset identification beacon images needs improvement, which may affect the accuracy of the cargo handover operation. Summary of the Invention
[0004] The purpose of this disclosure is to provide a container control method, apparatus, storage medium, electronic device, and program product.
[0005] In a first aspect, this disclosure provides a container control method, the container including a movable robotic arm, the robotic arm being equipped with a vision component and an end effector located at the end of the robotic arm; the method includes: In response to receiving a handover instruction for the target goods, the robotic arm's photographing pose is calibrated according to the target handover position indicated by the handover instruction to obtain the target photographing pose. The target photographing pose is the photographing pose of the robotic arm when it takes a picture of the first position marker through the vision component. The first position marker is used to identify the target handover position, which is the handover position of the target goods. After controlling the robotic arm to move to the target photographing pose, the first image of the first location marker is acquired through the vision component; Based on the first image, the robotic arm is controlled to perform a handover operation on the target cargo via the end effector.
[0006] Optionally, calibrating the robotic arm's photographic pose according to the target handover position indicated by the handover command to obtain the target photographic pose includes: Obtain the preset photo pose corresponding to the target handover position, wherein the preset photo pose is the photo pose of the robotic arm obtained by pre-calibration; After controlling the robotic arm to move to the preset photo-taking pose, the second image of the first location marker is acquired through the vision component; The robot arm's photographic pose corresponding to the target handover position is recalibrated based on the second image to obtain the target photographic pose.
[0007] Optionally, the step of recalibrating the photographic pose of the robotic arm corresponding to the target intersection position based on the second image to obtain the target photographic pose includes: The second image is subjected to quality inspection to obtain a quality inspection result, which characterizes whether the second image includes the first location marker and / or the clarity of the first location marker; If, based on the quality inspection results, it is determined that the second image does not contain the first location marker, or the clarity of the first location marker is less than or equal to a preset clarity threshold, the photographing pose of the robotic arm corresponding to the target handover position is recalibrated to obtain the target photographing pose.
[0008] Optionally, the photographing pose of the robotic arm corresponding to the target handover position is recalibrated to obtain the target photographing pose, including: Multiple preset photo points corresponding to the target handover position are obtained, and different preset photo points represent different photo poses of the robotic arm; For each preset photo-taking point, after controlling the robotic arm to move to the preset photo-taking point, the third image of the first position marker collected by the vision component is obtained; After performing hand-eye calibration on the third image corresponding to each preset shooting point, the target shooting pose is obtained.
[0009] Optionally, the container includes multiple storage compartments, each of which is equipped with a mounting mechanism for hanging goods. The step of controlling the robotic arm to perform a handover operation on the target cargo via the end effector based on the first image includes: Based on the first image, the location of the first location marker is spatially visually located to obtain the first spatial location; The first spatial location is verified, and after the verification is passed, a third image of the second location marker is acquired through the vision component. The second location marker is a visual identification beacon set on the target mounting mechanism. The target mounting mechanism includes a mounting mechanism in the target storage compartment for storing the target goods, or the target mounting mechanism includes a mounting mechanism on the vehicle for transporting the target goods. Based on the third image, the robotic arm is controlled to perform a handover operation on the target cargo via the end effector.
[0010] Optionally, the vehicle may be a drone, an unmanned vehicle, or a robot.
[0011] Optionally, controlling the robotic arm to perform a handover operation on the target cargo via the end effector based on the third image includes: Based on the third image, the target intersection position is spatially visually located to obtain the second spatial position; Based on the second spatial location, trajectory planning is performed to obtain the target motion trajectory; The robotic arm is controlled to move according to the target motion trajectory, so that the end effector moves to the second spatial position; After the end effector moves to the second spatial position, a handover operation is performed on the target cargo through the end effector.
[0012] Optionally, the handover instruction includes a first instruction to load the target cargo from the end effector to the target loading mechanism; The step of performing a handover operation on the target cargo via the end effector after the end effector moves to the second spatial position includes: When the handover instruction is the first instruction, after the end effector moves to the second spatial position, the end effector is controlled to transfer the target cargo from the end effector to the target mounting mechanism; Alternatively, the handover instruction may include a second instruction to transfer the target cargo from the target mounting mechanism to the end effector; The step of performing a handover operation on the target cargo via the end effector after the end effector moves to the second spatial position includes: When the handover instruction is the second instruction, after the end effector moves to the second spatial position, control the end effector to transfer the target cargo from the target mounting mechanism to the end effector.
[0013] Optionally, the method further includes: In response to receiving a notification message that the handover operation has been completed, the handover operation result is verified to obtain a verification result, which indicates whether the target goods have been successfully handed over. If the handover of the target goods fails based on the verification results, a preset fault handling strategy is executed.
[0014] Optionally, the verification of the handover operation result to obtain the verification result includes: The vision component acquires a fourth image of the target storage compartment, performs image recognition on the fourth image, and determines the verification result based on the image recognition result; and / or, The pressure signal collected by the pressure sensor is acquired, and the verification result is determined based on the change of the pressure signal within a preset time period. The pressure sensor is deployed inside the target storage compartment or on the end effector.
[0015] Secondly, this disclosure provides a container control device, wherein a movable robotic arm is disposed within the container, the robotic arm is provided with a vision component and an end effector located at the end of the robotic arm; the device includes: The calibration module is used to respond to the acquisition of a handover instruction for the target goods, and to calibrate the photographing pose of the robotic arm according to the target handover position indicated by the handover instruction, so as to obtain the target photographing pose. The target photographing pose is the photographing pose of the robotic arm when it takes a picture of the first position mark through the vision component. The first position mark is used to identify the target handover position, which is the handover position of the target goods. The image acquisition module is used to control the robotic arm to move to the target photographing pose and then acquire the first image of the first position marker through the vision component; The control module is used to control the robotic arm to perform a handover operation on the target cargo through the end effector based on the first image.
[0016] Thirdly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the container control method described in the first aspect of this disclosure.
[0017] Fourthly, this disclosure provides an electronic device, comprising: A memory on which computer programs are stored; A processor is configured to execute the computer program in the memory to implement the steps of the container control method described in the first aspect of this disclosure.
[0018] Fifthly, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the container control method described in the first aspect of this disclosure.
[0019] The above technical solution allows for the automatic calibration of the robotic arm's photographic pose based on the target handover position indicated by the handover command, thus obtaining the target photographic pose. The robotic arm then uses this real-time automatically calibrated target photographic pose to acquire a first image of the visual recognition beacon via a vision component. Compared to images acquired using a fixed photographic pose, this first image provides higher accuracy for spatial visual positioning, enabling more precise control of the robotic arm to perform the handover operation and improving the efficiency of the handover process.
[0020] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description
[0021] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings: Figure 1 A schematic diagram of a robotic arm installed inside a shipping container is shown.
[0022] Figure 2 This is a flowchart illustrating a container control method according to an exemplary embodiment.
[0023] Figure 3 It is based on Figure 2 The illustrated embodiment shows a flowchart of a container control method.
[0024] Figure 4 It is based on Figure 2 The illustrated embodiment shows a flowchart of a container control method.
[0025] Figure 5 It is based on Figure 2 The illustrated embodiment shows a flowchart of a container control method.
[0026] Figure 6 This is a block diagram illustrating a container control device according to an exemplary embodiment.
[0027] Figure 7 It is based on Figure 6 The illustrated embodiment shows a block diagram of a container control device.
[0028] Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation
[0029] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.
[0030] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.
[0031] First, the application scenarios of this disclosure are described. This disclosure is mainly applied to scenarios involving the control of vending machines (such as express delivery lockers, food delivery lockers, etc.) equipped with robotic arms. The robotic arm is movably installed inside the locker. Through this robotic arm, goods can be moved between different storage compartments within the locker, and goods can also be transferred between the carrier and the locker. It should be understood that the carrier can be various goods delivery equipment such as drones, unmanned vehicles, and robots. For ease of description, this disclosure only uses a drone delivery scenario as an example.
[0032] Figure 1 A schematic diagram of a robotic arm installed inside a shipping container is shown, such as... Figure 1 As shown, the container (not shown in the figure) is equipped with a track 21, on which the robotic arm 2 can move up and down. When the robotic arm 2 moves up and down, it can also drive the end effector 3 of the robotic arm 2 to move up and down. For example, the robotic arm 2 can be controlled to move upward on the track 21 to move the end effector 3 and the cargo 1 hanging on the end effector 3 to the top opening 20, so that the end effector 3 can be placed at the top opening 20 to hand over the cargo 1 to the incoming drone. In addition, the robotic arm 2 is also equipped with a vision component (not shown in the figure, but can be arranged above the end effector 3). This vision component is used to collect images of the location markers at the cargo handover position. The location markers can be visual identification beacons (such as QR codes). The cargo handover position refers to the position where the cargo handover operation needs to be carried out, such as the position of the mounting mechanism arranged in each storage compartment of the container (or on the drone). After spatial visual positioning based on the image, the location information of the cargo handover position is identified. Then, the movement posture of the robotic arm 2 is adjusted according to this location information, thereby adjusting the relative position of the end effector 3 and the cargo handover position to ensure that the cargo 1 can be successfully handed over. It should be understood that for ground delivery equipment such as unmanned vehicles or robots, the opening 20 corresponds to the side or bottom of the container.
[0033] In the aforementioned process, before using the vision component to acquire images of the location markers at the cargo handover position, it is necessary to control the robotic arm to move to a preset photographing pose corresponding to that cargo handover position. This allows for the acquisition of higher-quality images of the visual beacons based on the extrinsic parameters of the vision component within this preset photographing pose. Related technologies typically employ fixed photographing poses to acquire images of location markers. However, over random time, the position of the visual beacon may shift, and the positioning accuracy of spatial visual positioning based on beacon images acquired from fixed poses needs improvement, thus affecting the accuracy of the cargo handover operation.
[0034] To address the aforementioned problems, this disclosure provides a container control method, apparatus, storage medium, electronic device, and program product. The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.
[0035] Figure 2 This is a flowchart illustrating a container control method according to an exemplary embodiment. This method can be applied to a container or a server. If the container control method is applied to a server, the container can communicate with the server, allowing the server to send control commands to the container to control it.
[0036] The internal layout of the container involved in this disclosure is as follows: Figure 1 The movable robotic arm 2 shown is equipped with a vision unit and an end effector 3 located at the end of the arm. Here, "movable robotic arm" refers to an arm capable of vertical movement and, after reaching a certain height in the vertical direction, multi-degree-of-freedom spatial motion. For example, the robotic arm could be a six-degree-of-freedom arm with six independent drive joints, each capable of independent movement, thus achieving complex spatial motion. In practical applications, the robotic arm can be moved vertically along a track within a container by controlling an RGV (Rail Guided Vehicle), and then further complex spatial motion can be achieved by controlling at least one drive joint of the robotic arm.
[0037] The pose of the robotic arm changes by controlling the movement of at least one drive joint. As the pose of the robotic arm changes, the pose of the vision component on the robotic arm also changes. Vision components with different poses have different extrinsic parameters. Additionally, the container may include multiple storage compartments, which may further include buffer compartments and regular compartments. Users can open the regular compartments to retrieve goods. The buffer compartment is used to buffer goods when no regular compartment is available, and to transfer goods from the buffer compartment to a regular compartment when a regular compartment is determined to be free.
[0038] like Figure 2 As shown, the container control method includes the following steps: In step S101, in response to receiving a handover instruction for the target goods, the robotic arm's photographing pose is calibrated according to the target handover position indicated by the handover instruction to obtain the target photographing pose. The target photographing pose is the photographing pose of the robotic arm when it takes a picture of the first position marker through the vision component. The first position marker is used to identify the target handover position, which is the handover position of the target goods.
[0039] The target goods may include takeout food packages, express delivery items, etc. The handover instruction may include a first instruction to attach the target goods from the end effector to the target mounting mechanism, or a second instruction to transfer the target goods from the target mounting mechanism to the end effector. The target mounting mechanism refers to a mounting mechanism installed in the target storage compartment (a storage compartment that already contains the target goods or will contain them) or a mounting mechanism installed on the drone (for example, the mounting mechanism may be a gripper or hook capable of attaching goods), which can be used to carry the goods. The drone is used to transport the target goods.
[0040] The following describes the triggering scenarios for obtaining the handover instruction for the target cargo.
[0041] The container control method disclosed herein may involve the following three cargo delivery scenarios: delivering target goods from a drone to a container (e.g., a drone placing a takeout meal package into a takeout locker for user pickup), transferring target goods from a container to a drone (e.g., a merchant placing a meal package into a takeout locker, and the drone retrieving the meal from the locker for delivery), and transferring target goods from one storage compartment to another within the container (e.g., transferring a meal package from a buffer compartment to a regular compartment). In each of these three cargo delivery scenarios, when using a robotic arm configured within the container for cargo transfer, it is necessary to attach the target goods from the end effector to the target attachment mechanism and / or transfer the target goods from the target attachment mechanism to the end effector.
[0042] For example, during the delivery of target cargo from a drone to a container, the drone lands on a landing platform on top of the container. This platform has a push rod to right the drone. The drone can send a cargo delivery notification message to the container. Upon receiving this notification message, the container controls its top cover to open and moves the RGV to a preset waiting position outside the cargo compartment, so that the robotic arm also moves to the preset waiting position outside the cargo compartment to prepare for cargo reception. The container can then generate a second instruction for the target cargo and send this second instruction to the drone. This second instruction instructs the transfer of the target cargo from the target mounting mechanism on the drone to the end effector of the robotic arm. The second instruction also indicates that the target handover location for this cargo transfer operation can be the location of the drone's target mounting mechanism. Then, by executing steps S101-S103, the transfer of the target cargo from the drone's target mounting mechanism to the end effector of the robotic arm can be achieved. Next, the container can select any available storage compartment (which can be a buffer compartment or a regular compartment) from multiple storage compartments as the target storage compartment. After controlling the RGV to move to the height of the target storage compartment, a first instruction is generated. This first instruction instructs the target cargo to be mounted from the end effector of the robotic arm to the target mounting mechanism within the target storage compartment. The first instruction also indicates that the target handover position for this cargo transfer operation can be the location of the target mounting mechanism within the target storage compartment. Subsequently, by executing steps S101-S103, the handover operation of mounting the target cargo from the end effector of the robotic arm to the target mounting mechanism within the target storage compartment is realized. Therefore, the process of delivering the target cargo from the drone to the container includes two handover operations for the target cargo.
[0043] It should also be noted that after the target goods are delivered from the drone to the container, the container can send a delivery completion notification message to the server and / or the drone. At this point, the drone can take off, and the server can then schedule it for other delivery tasks. After delivering the target goods from the drone to the container, the container can also control the RGV to move to a preset waiting position inside the compartment, so that the RGV can wait for the next delivery task. In another possible application scenario, after delivering the target goods from the drone to the container, the RGV can also remain at the track height corresponding to the target storage compartment opening, waiting for the next task scheduling.
[0044] The above describes the two handover commands involved in delivering target goods from a drone to a container, and the handover process for the target goods in response to each handover command. The process of transferring target goods from the container to the drone using a robotic arm includes two sub-processes: transferring the target goods from the target loading mechanism in the target storage compartment of the container to the end effector of the robotic arm (corresponding to the second command), and loading the target goods from the end effector of the robotic arm to the target loading mechanism of the drone (corresponding to the first command). Similarly, the process of transferring target goods from one storage compartment to another within the container using a robotic arm includes two sub-processes: transferring the target goods from the target loading mechanism in the first storage compartment of the container to the end effector of the robotic arm (corresponding to the second command), and loading the target goods from the end effector of the robotic arm to the target loading mechanism in the second storage compartment (corresponding to the first command).
[0045] In other words, in all three of the above-mentioned cargo delivery scenarios, the container can be triggered to receive a first instruction to mount the target cargo from the end effector to the target mounting mechanism and a second instruction to transfer the target cargo from the target mounting mechanism to the end effector.
[0046] In addition, the robotic arm in this disclosure is also equipped with a vision component. As the pose of the robotic arm changes, the pose of the vision component (which can be understood as the extrinsic parameter of the vision component) also changes. The image content acquired by the vision component will also be different under different poses. Therefore, in order to accurately acquire the image of the first position marker located at the target intersection position, it is necessary to first calibrate the pose of the vision component. Here, the photographing pose of the robotic arm refers to the photographing pose when taking a picture of the first position marker through the vision component on the robotic arm. This disclosure adjusts the pose of the vision component by controlling the change of the pose of the robotic arm. Therefore, in order to adjust the pose of the vision component, the photographing pose of the robotic arm can be calibrated to obtain the target photographing pose. After controlling the robotic arm to move to the target photographing pose, the vision component on it will also be in the optimal photographing pose.
[0047] The first location identifier includes a visual identification beacon (such as a QR code) corresponding to the target handover location. For example, the first location identifier can be set at the target storage compartment opening to spatially locate the position of the target storage compartment opening.
[0048] In step S102, after controlling the robotic arm to move to the target photographing pose, the first image of the first position marker is acquired through the vision component.
[0049] After obtaining the target image pose, the robotic arm can be planned according to the target image pose. The planned motion trajectory includes the position adjustment information of at least one drive joint of the robotic arm, so that the at least one drive joint of the robotic arm can be controlled to move according to the planned position adjustment information, so that the robotic arm can reach the target image pose.
[0050] After the robotic arm moves to the target image pose, the vision component on the robotic arm is in a relatively ideal target pose, and the first image of the first position marker is acquired by the vision component in the target pose.
[0051] In step S103, the robotic arm is controlled to perform a handover operation on the target goods through the end effector based on the first image.
[0052] The handover operation may include transferring the target cargo from the end effector of the robotic arm to the target mounting mechanism, or transferring the target cargo from the target mounting mechanism to the end effector of the robotic arm.
[0053] Using the above method, the robotic arm's photographic pose can be automatically calibrated according to the target handover position indicated by the handover command, thus obtaining the target photographic pose. The robotic arm uses the real-time calibrated target photographic pose to acquire the first image of the visual recognition beacon through the vision component. Compared with the image acquired using a fixed photographic pose, this first image has higher accuracy for spatial visual positioning, thereby enabling more precise control of the robotic arm to perform the handover operation and improving the efficiency of the handover work.
[0054] Figure 3 It is based on Figure 2 The illustrated embodiment shows a flowchart of a container control method, as follows: Figure 3 As shown, step S101 includes the following sub-steps: In step S1011, a preset photo pose corresponding to the target handover position is obtained, wherein the preset photo pose is the photo pose of the robotic arm obtained by pre-calibration.
[0055] In step S1012, after controlling the robotic arm to move to the preset photo-taking pose, the second image of the first position marker is acquired through the vision component.
[0056] In one possible application scenario, the corresponding robotic arm's photographing pose can be pre-marked for each possible cargo handover position corresponding to the container. These possible cargo handover positions could include, for example, the positions of mounting mechanisms within each storage compartment or the positions of mounting mechanisms configured on a drone. Thus, after obtaining the target handover position based on the handover command, the preset photographing pose of the robotic arm corresponding to that target handover position can be determined based on a pre-defined correspondence. This preset photographing pose can include the target movement position of at least one drive joint on the robotic arm. Therefore, by controlling each of these at least one drive joint to move to its corresponding target movement position, the robotic arm can move to the preset photographing pose.
[0057] After the robotic arm moves to the preset photo-taking pose, the vision component on it can be controlled to capture a second image of the first position marker.
[0058] In step S1013, the photographic pose of the robotic arm corresponding to the target intersection position is recalibrated based on the second image to obtain the target photographic pose.
[0059] In this step, the second image can be quality inspected to obtain the quality inspection result. The quality inspection result indicates whether the second image includes the first position marker and / or the clarity of the first position marker. If the quality inspection result determines that the second image does not include the first position marker, or the clarity of the first position marker is less than or equal to a preset clarity threshold, the photographing pose of the robotic arm corresponding to the target handover position can be recalibrated to obtain the target photographing pose.
[0060] In practical applications, the first location marker (such as a QR code) located at the target intersection may shift, become obscured, or be partially damaged over time. If the second image is still captured according to the pre-calibrated preset shooting pose, the second image may not be able to accurately locate the target intersection spatially. Therefore, by performing this step, the quality of the second image is checked. If the quality check results determine that the second image does not include the first location marker, or only includes a part of the first location marker, or the clarity of the first location marker in the second image is low, the shooting pose of the robotic arm corresponding to the target intersection can be recalibrated to obtain the target shooting pose. Compared with the preset shooting pose, the shooting angle of the vision component for the first location marker and the relative position between the vision component and the first location marker will change in the target shooting pose. Therefore, based on the target shooting pose, the image of the first location marker with the best quality at the current moment can be captured, thereby improving the accuracy of spatial positioning.
[0061] When it is determined that the robot arm's photo pose corresponding to the target handover position needs to be recalibrated, multiple preset photo points corresponding to the target handover position can be obtained. Different preset photo points represent different photo poses of the robot arm. For each preset photo point, after controlling the robot arm to move to the preset photo point, the third image of the first position marker collected by the vision component is obtained. After performing hand-eye calibration based on the third image corresponding to each preset photo point, the target photo pose is obtained.
[0062] The multiple photographic poses describe the different positions and orientations of the robotic arm in the world coordinate system.
[0063] Based on the third image corresponding to each preset shooting point, hand-eye calibration is performed to obtain the target shooting pose that optimizes the image quality. The specific steps for hand-eye calibration can be found in relevant literature and are not specifically limited here.
[0064] Figure 4 It is based on Figure 2 The illustrated embodiment shows a flowchart of a container control method, as follows: Figure 4 As shown, step S103 includes the following sub-steps: In step S1031, the position of the first location marker is spatially visually located based on the first image to obtain the first spatial position.
[0065] The first location marker can be, for example, a visual identification beacon (such as a QR code) at the target storage compartment opening. In this step, the first spatial location marked by the first location marker can be determined based on the first image using spatial visual positioning technology. When the first location marker is a visual identification beacon at the target storage compartment opening, the first spatial location represents the location of the target storage compartment opening.
[0066] In step S1032, the first spatial position is verified, and after the verification is passed, a third image of the second position identifier is acquired by the vision component. The second position identifier is a visual identification beacon set on the target mounting mechanism. The target mounting mechanism includes a mounting mechanism in the target storage compartment for storing the target goods, or the target mounting mechanism includes a mounting mechanism on the vehicle used to transport the target goods.
[0067] The vehicle can be any of the various cargo delivery equipment such as drones, unmanned vehicles, and robots.
[0068] After acquiring the first spatial position, the vision component can send it to the robotic arm, which then verifies the position. This verification may include comparing the first spatial position with location information of the same position obtained from historical positioning, or comparing the first spatial position with a preset recorded position. If the distance difference between the two is less than or equal to a preset distance threshold (which can be set according to control precision), the verification of the first spatial position is considered successful; otherwise, the verification fails.
[0069] In this disclosure, the first location marker is a marker located at the opening of the target storage compartment; therefore, the first spatial location is the location of the target storage compartment opening. However, the handover operation of the target goods usually occurs between the end effector of the robotic arm and the target connecting mechanism. Therefore, in order to accurately realize the handover operation of the target goods, it is also necessary to accurately locate the location of the target connecting mechanism. In one implementation of this disclosure, the process of obtaining the first spatial location by visual positioning based on the first image of the first location marker can be regarded as the initial positioning process. After successful initial positioning, the location of the target mounting mechanism can be finely positioned based on the second location marker to further determine the location of the target mounting mechanism. The second location marker may include a visual identification beacon located on the target mounting mechanism. The target mounting mechanism may include a mounting mechanism inside the target storage compartment storing the target goods, or a mounting mechanism on a drone used to transport the target goods.
[0070] In step S1033, the robotic arm is controlled to perform a handover operation on the target goods through the end effector based on the third image.
[0071] In this step, the target handover location can be spatially visually located based on the third image to obtain a second spatial location. Since this second location marker can be a visual identification beacon located on the target mounting mechanism, the second spatial location obtained by positioning based on the third image of the second location marker is the location of the target mounting mechanism to participate in the handover operation. Then, trajectory planning can be performed based on the second spatial location to obtain the target motion trajectory. The robotic arm is controlled to move according to the target motion trajectory so that the end effector moves to the second spatial location. Thus, after the end effector moves to the second spatial location, the handover operation on the target cargo can be accurately performed using the end effector. This handover operation may include, for example, a gripper mounting action.
[0072] The trajectory planning based on the second spatial position to obtain the target motion trajectory may include planning the movement trajectory of each drive joint of the robotic arm with the goal of moving the end effector of the robotic arm to the second spatial position, obtaining the movement trajectory of each drive joint, and then controlling the movement of each drive joint of the robotic arm according to the movement trajectory corresponding to the drive joint, thereby realizing the movement of the end effector of the robotic arm to the second spatial position.
[0073] As described above, the handover instruction may include a first instruction or a second instruction. In the case where the handover instruction is a first instruction to load the target cargo from the end effector to the target loading mechanism, the end effector can be controlled to transfer the target cargo from the end effector to the target loading mechanism after it moves to the second spatial position.
[0074] For example, if the target goods are takeout meal packages and the target mounting mechanism is located within the target storage compartment, the takeout meal package can be placed into the target storage compartment of the container by transferring it from the end effector to the target mounting mechanism, allowing the user to retrieve the meal from that compartment. Similarly, if the target goods are takeout meal packages and the target mounting mechanism is located on a drone, the takeout meal package can be mounted from the container to the drone by transferring it from the end effector to the target mounting mechanism, enabling unmanned delivery of the takeout meal package via drone. This is merely an example and is not intended to limit the scope of the invention.
[0075] In the case where the handover instruction is a second instruction to transfer the target cargo from the target mounting mechanism to the end effector, the end effector can be controlled to transfer the target cargo from the target mounting mechanism to the end effector after it moves to the second spatial position.
[0076] For example, if the target goods are takeout meal packs and the target mounting mechanism is located within a target storage compartment, the takeout meal pack can be transferred from the target mounting mechanism to the end effector via a robotic arm to another storage compartment (e.g., from a buffer compartment to a regular compartment). Alternatively, the takeout meal pack can be transferred from the target storage compartment to the end effector of the robotic arm of the container, so that the takeout meal pack can be transported and mounted on the target mounting mechanism of a drone (thus enabling drone delivery of the takeout meal pack). Or, the takeout meal pack mounted on the target mounting mechanism of the drone can be transferred from the drone to the end effector of the robotic arm of the container, so that the takeout meal pack can be stored in the container via the robotic arm. These examples are merely illustrative and are not intended to limit the scope of this disclosure.
[0077] Figure 5 It is based on Figure 2 The illustrated embodiment shows a flowchart of a container control method, as follows: Figure 5 As shown, the method also includes the following steps: In step S104, in response to receiving a notification message that the handover operation has been completed, the handover operation result is verified to obtain a verification result, which indicates whether the target goods have been successfully handed over.
[0078] In this step, a fourth image inside the target storage compartment can be acquired by a vision component, the fourth image can be image recognized, and the verification result can be determined based on the image recognition result; and / or, the pressure signal acquired by the pressure sensor can be acquired, and the verification result can be determined based on the change of the pressure signal within a preset time period, wherein the pressure sensor is deployed inside the target storage compartment or on the end effector.
[0079] For example, assuming that by performing this handover operation, the target cargo is transferred from the target mounting mechanism in the target storage compartment to the end effector of the robotic arm, image recognition can be performed on the fourth image. If, based on the image recognition result, it is determined that the target cargo is no longer in the target storage compartment, and / or, based on the signal change of the pressure sensor in the target storage compartment, it is determined that the target cargo in the target storage compartment has been removed, the handover can be considered successful. Otherwise, the handover operation is considered a failure.
[0080] If the current target cargo handover operation is confirmed to be successful, the robotic arm can be controlled to move to a preset waiting position (which can be located inside the container) to wait for the next task scheduling.
[0081] In step S105, when it is determined that the handover of the target goods has failed based on the verification result, a preset fault handling strategy is executed.
[0082] The preset fault handling strategy may include, for example, controlling the vision component on the robotic arm to re-perform visual positioning, so as to re-execute the handover operation based on the re-positioned target handover location. Fault notifications may also be provided through a target terminal (e.g., outputting fault information), which may be a maintenance terminal bound to the container or the container itself. Feedback messages indicating failed goods handover may also be sent to the server.
[0083] Figure 6 This is a block diagram illustrating a container control device according to an exemplary embodiment, wherein a movable robotic arm is disposed within the container, and the robotic arm is provided with a vision component and an end effector located at the end of the robotic arm; as shown below. Figure 6 As shown, the device includes: The calibration module 601 is used to respond to the acquisition of a handover instruction for the target goods, and to calibrate the photographing pose of the robotic arm according to the target handover position indicated by the handover instruction, so as to obtain the target photographing pose. The target photographing pose is the photographing pose of the robotic arm when it takes a picture of the first position mark through the vision component. The first position mark is used to identify the target handover position, which is the handover position of the target goods. Image acquisition module 602 is used to control the robotic arm to move to the target photographing pose and then acquire the first image of the first position marker through the vision component; The control module 603 is used to control the robotic arm to perform a handover operation on the target cargo through the end effector based on the first image.
[0084] Optionally, the calibration module 601 is used to obtain a preset photographing pose corresponding to the target intersection position, wherein the preset photographing pose is a pre-calibrated photographing pose of the robotic arm; after controlling the robotic arm to move to the preset photographing pose, a second image of the first position marker is acquired through the vision component; and the photographing pose of the robotic arm corresponding to the target intersection position is recalibrated according to the second image to obtain the target photographing pose.
[0085] Optionally, the calibration module 601 is used to perform quality inspection on the second image to obtain a quality inspection result, wherein the quality inspection result characterizes whether the second image includes the first position marker and / or the clarity of the first position marker; if the quality inspection result determines that the second image does not include the first position marker, or the clarity of the first position marker is less than or equal to a preset clarity threshold, the photographing pose of the robotic arm corresponding to the target intersection position is recalibrated to obtain the target photographing pose.
[0086] Optionally, the calibration module 601 is used to acquire multiple preset photo points corresponding to the target intersection position, where different preset photo points represent different photo poses of the robotic arm; for each preset photo point, after controlling the robotic arm to move to the preset photo point, the third image of the first position marker collected by the vision component is acquired; and after performing hand-eye calibration based on the third image corresponding to each preset photo point, the target photo pose is obtained.
[0087] Optionally, the container includes multiple storage compartments, each of which is equipped with a mounting mechanism for hanging goods. The control module 603 is configured to perform spatial visual positioning of the position of the first location marker based on the first image to obtain a first spatial position; verify the first spatial position, and after the verification is passed, acquire a third image of the second location marker through the vision component, wherein the second location marker is a visual recognition beacon set on the target mounting mechanism, the target mounting mechanism includes a mounting mechanism in the target storage compartment for storing the target goods, or the target mounting mechanism includes a mounting mechanism on a drone for transporting the target goods; and control the robotic arm to perform a handover operation on the target goods through the end effector based on the third image.
[0088] Optionally, the control module 603 is configured to perform spatial visual positioning of the target handover position based on the third image to obtain a second spatial position; perform trajectory planning based on the second spatial position to obtain a target motion trajectory; control the robotic arm to move according to the target motion trajectory so that the end effector moves to the second spatial position; and after the end effector moves to the second spatial position, perform a handover operation on the target goods through the end effector.
[0089] Optionally, the handover instruction includes a first instruction to load the target cargo from the end effector to the target loading mechanism; the control module 603 is used to control the end effector to transfer the target cargo from the end effector to the target loading mechanism after the end effector moves to the second spatial position when the handover instruction is the first instruction.
[0090] Optionally, the handover instruction includes a second instruction to transfer the target cargo from the target mounting mechanism to the end effector; the control module 603 is used to control the end effector to transfer the target cargo from the target mounting mechanism to the end effector after the end effector moves to the second spatial position when the handover instruction is the second instruction.
[0091] Optionally, Figure 7 It is based on Figure 6 The illustrated embodiment shows a block diagram of a container control device, such as Figure 7 As shown, the device also includes: The verification module 604 is used to verify the handover operation result in response to receiving a notification message that the handover operation has been completed, and to obtain a verification result. The verification result indicates whether the handover of the target goods was successful. If it is determined that the handover of the target goods has failed based on the verification result, a preset fault handling strategy is executed.
[0092] Optionally, the verification module 604 is used to acquire a fourth image inside the target storage compartment through the vision component, perform image recognition on the fourth image, and determine the verification result based on the image recognition result; and / or, acquire a pressure signal collected by a pressure sensor, and determine the verification result based on the change of the pressure signal within a preset time period, wherein the pressure sensor is deployed inside the target storage compartment or on the end effector.
[0093] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0094] Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Figure 8 As shown, the electronic device 700 may include a processor 701 and a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.
[0095] The processor 701 controls the overall operation of the electronic device 700 to complete all or part of the steps in the aforementioned container control method. The memory 702 stores various types of data to support the operation of the electronic device 700. This data may include, for example, instructions for any application or method operating on the electronic device 700, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 703 may include a screen and audio components. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 702 or transmitted via communication component 705. The audio component also includes at least one speaker for outputting audio signals. I / O interface 704 provides an interface between processor 701 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.
[0096] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the container control method described above.
[0097] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the container control method described above. For example, the computer-readable storage medium may be the memory 702 including program instructions, which may be executed by the processor 701 of the electronic device 700 to complete the container control method described above.
[0098] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a programmable device, the computer program having a code portion for performing the container control method described above when executed by the programmable device.
[0099] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.
[0100] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.
[0101] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.
Claims
1. A container control method, characterized in that, The container is equipped with a movable robotic arm, which is equipped with a vision unit and an end effector located at the end of the robotic arm; the method includes: In response to receiving a handover instruction for the target goods, the robotic arm's photographing pose is calibrated according to the target handover position indicated by the handover instruction to obtain the target photographing pose. The target photographing pose is the photographing pose of the robotic arm when it takes a picture of the first position marker through the vision component. The first position marker is used to identify the target handover position, which is the handover position of the target goods. After controlling the robotic arm to move to the target photographing pose, the first image of the first location marker is acquired through the vision component; Based on the first image, the robotic arm is controlled to perform a handover operation on the target cargo via the end effector.
2. The method according to claim 1, characterized in that, The step of calibrating the robotic arm's photographic pose according to the target handover position indicated by the handover command to obtain the target photographic pose includes: Obtain the preset photo pose corresponding to the target handover position, wherein the preset photo pose is the photo pose of the robotic arm obtained by pre-calibration; After controlling the robotic arm to move to the preset photo-taking pose, the second image of the first location marker is acquired through the vision component; The robot arm's photographic pose corresponding to the target handover position is recalibrated based on the second image to obtain the target photographic pose.
3. The method according to claim 2, characterized in that, The step of recalibrating the photographic pose of the robotic arm corresponding to the target intersection position based on the second image to obtain the target photographic pose includes: The second image is subjected to quality inspection to obtain a quality inspection result, which characterizes whether the second image includes the first location marker and / or the clarity of the first location marker; If, based on the quality inspection results, it is determined that the second image does not contain the first location marker, or the clarity of the first location marker is less than or equal to a preset clarity threshold, the photographing pose of the robotic arm corresponding to the target handover position is recalibrated to obtain the target photographing pose.
4. The method according to claim 3, characterized in that, The robotic arm's photographic pose corresponding to the target handover position is recalibrated to obtain the target photographic pose, including: Multiple preset photo points corresponding to the target handover position are obtained, and different preset photo points represent different photo poses of the robotic arm; For each preset photo-taking point, after controlling the robotic arm to move to the preset photo-taking point, the third image of the first position marker collected by the vision component is obtained; After performing hand-eye calibration on the third image corresponding to each preset shooting point, the target shooting pose is obtained.
5. The method according to claim 1, characterized in that, The container includes multiple storage compartments, each of which is equipped with a mounting mechanism for hanging goods. The step of controlling the robotic arm to perform a handover operation on the target cargo via the end effector based on the first image includes: Based on the first image, the location of the first location marker is spatially visually located to obtain the first spatial location; The first spatial location is verified, and after the verification is passed, a third image of the second location marker is acquired through the vision component. The second location marker is a visual identification beacon set on the target mounting mechanism. The target mounting mechanism includes a mounting mechanism in the target storage compartment for storing the target goods, or the target mounting mechanism includes a mounting mechanism on the vehicle for transporting the target goods. Based on the third image, the robotic arm is controlled to perform a handover operation on the target cargo via the end effector.
6. The method according to claim 5, characterized in that, The step of controlling the robotic arm to perform a handover operation on the target cargo via the end effector based on the third image includes: Based on the third image, the target intersection position is spatially visually located to obtain the second spatial position; Based on the second spatial location, trajectory planning is performed to obtain the target motion trajectory; The robotic arm is controlled to move according to the target motion trajectory, so that the end effector moves to the second spatial position; After the end effector moves to the second spatial position, a handover operation is performed on the target cargo through the end effector.
7. The method according to claim 6, characterized in that, The handover instruction includes a first instruction to load the target cargo from the end effector to the target loading mechanism; The step of performing a handover operation on the target cargo via the end effector after the end effector moves to the second spatial position includes: When the handover instruction is the first instruction, after the end effector moves to the second spatial position, the end effector is controlled to transfer the target cargo from the end effector to the target mounting mechanism; or, The handover instruction includes a second instruction to transfer the target cargo from the target mounting mechanism to the end effector; The step of performing a handover operation on the target cargo via the end effector after the end effector moves to the second spatial position includes: When the handover instruction is the second instruction, after the end effector moves to the second spatial position, control the end effector to transfer the target cargo from the target mounting mechanism to the end effector.
8. The method according to claim 5, characterized in that, The method further includes: In response to receiving a notification message that the handover operation has been completed, the handover operation result is verified to obtain a verification result, which indicates whether the target goods have been successfully handed over. If the handover of the target goods fails based on the verification results, a preset fault handling strategy is executed.
9. The method according to claim 8, characterized in that, The verification of the handover operation results includes the following verification results: The vision component acquires a fourth image of the target storage compartment, performs image recognition on the fourth image, and determines the verification result based on the image recognition result; and / or, The pressure signal collected by the pressure sensor is acquired, and the verification result is determined based on the change of the pressure signal within a preset time period. The pressure sensor is deployed inside the target storage compartment or on the end effector.
10. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-9.