Multi-camera based robotic docking method, device, and storage medium

Abstract

The application discloses a multi-camera-based robot docking method and device, and a storage medium. The robot is provided with a downward-looking camera and a forward-looking camera. The downward-looking camera is used to identify a code strip on the ground, and the forward-looking camera is used to identify a shelf identifier of a shelf. The method comprises the following steps: controlling the robot to move to a pre-labeled preparation point. The preparation point is a position for adjusting the pose of the robot before the robot docks with the shelf. If the downward-looking camera does not identify the code strip during the movement of the robot to the preparation point, the forward-looking camera is used to obtain a forward-looking identification result of the shelf identifier. The robot is guided to move according to the forward-looking identification result until the code strip is identified by the downward-looking camera. The docking position of a docking point is determined according to a downward-looking identification result of the downward-looking camera and a docking parameter of the pre-labeled docking point. The docking point is a position for docking the robot with the shelf. The robot is controlled to move to the docking position to dock with the shelf. The above scheme can improve the docking accuracy of the robot when the robot performs a task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a robot docking method based on multiple cameras, equipment and a storage medium. BACKGROUND

[0002] With the gradual maturity and perfection of robot technology, the robot industry is being further developed in today's society. Among various operating scenarios of robots, high-precision docking has always been the top priority of various task requirements.

[0003] Taking robots such as AGV (Automated Guided Vehicle) and AMR (Autonomous Mobile Robot) as examples, the traditional method sets a specific coded target in the operating scenario of the robot, and the camera provided on the robot is used to collect and recognize the image of the coded target to achieve a visual docking solution.

[0004] However, the simple visual docking solution has low precision today, and with the development of robot navigation and positioning technology, the terminal point precision of centimeters cannot meet the requirements of some high-precision processes. Moreover, the target needs to be observed throughout the docking process, otherwise if there is no real-time feedback quantity (for example, the robot moves off or the identification code band used for visual navigation is contaminated, damaged, etc.), the subsequent docking cannot be completed, and the fault tolerance is low. SUMMARY

[0005] The present application provides at least a robot docking method based on multiple cameras, a device, equipment and a computer readable storage medium.

[0006] The first aspect of the present application provides a robot docking method based on multiple cameras. The robot is provided with a downward-looking camera and a forward-looking camera. The downward-looking camera is used to identify a preset code band on the ground where the robot travels, and the forward-looking camera is used to identify a shelf identification on a shelf to which the robot is docked. The method comprises: controlling the robot to move to a pre-labeled preparation point, which is a position for adjusting the pose of the robot before docking with the shelf; in response to the downward-looking camera not identifying the preset code band during the movement of the robot to the preparation point, obtaining a forward-looking identification result of the shelf identification by the forward-looking camera; guiding the robot to move according to the forward-looking identification result until the downward-looking camera identifies the preset code band; determining a docking position of a docking point according to a downward-looking identification result of the preset code band by the downward-looking camera and a docking parameter of the docking point pre-labeled, the docking point being a position for docking the robot with the shelf; and controlling the robot to move to the docking position to dock with the shelf.

[0007] In an embodiment, the front-view recognition result comprises distance information between the robot and the shelf marker, and the guiding the robot to move according to the front-view recognition result comprises: comparing the distance information with a safety distance threshold to obtain a distance comparison result; and guiding the robot to move according to the distance comparison result.

[0008] In an embodiment, the guiding the robot to move according to the distance comparison result comprises: in response to the distance comparison result representing that the distance information is greater than or equal to the safety distance threshold, controlling the robot to continue moving; in response to the distance comparison result representing that the distance information is less than the safety distance threshold, pausing the robot movement and adjusting the boom of the robot to align with the cartridge on the shelf according to the front-view recognition result; and controlling the robot with the adjusted boom to continue moving.

[0009] In an embodiment, the adjusting the boom of the robot to align with the cartridge on the shelf according to the front-view recognition result comprises: determining, according to the front-view recognition result, a vertical docking deviation and a horizontal docking deviation between a boom center of the boom and a cartridge center of the cartridge; and adjusting the robot according to the vertical docking deviation and the horizontal docking deviation until the boom center aligns with the cartridge center.

[0010] In an embodiment, the front-view recognition result further comprises an identified height difference and an identified horizontal difference between the boom center of the boom and the shelf marker, and the determining the vertical docking deviation and the horizontal docking deviation between the boom center of the boom and the cartridge center of the cartridge according to the front-view recognition result comprises: obtaining a placement height difference between the shelf marker and the cartridge center; determining the vertical docking deviation according to the identified height difference and the placement height difference; and determining the identified horizontal difference as the horizontal docking deviation.

[0011] In an embodiment, the determining the docking position of the docking point according to the lower-view recognition result of the preset code band by the lower-view camera and the docking parameter of the docking point pre-calibrated comprises: obtaining a current pose of the robot and a lower-view camera extrinsic parameter of the lower-view camera; and determining the docking position according to the lower-view recognition result, the current pose, the lower-view camera extrinsic parameter, and the docking parameter.

[0012] In one embodiment, the preset code band includes multiple identifier codes. After guiding the robot to move according to the forward-looking recognition result until the downward-looking camera recognizes the preset code band, the method further includes: obtaining the downward-looking recognition result of the downward-looking camera on the preset code band and the number of identifier codes corresponding to the downward-looking recognition result of each frame; in response to the number of identifier codes in the current frame being greater than the number of identifier codes in the historical frames, updating the docking position according to the downward-looking recognition result of the current frame, wherein the acquisition timing of the current frame is later than the acquisition timing of the historical frames.

[0013] In one embodiment, after obtaining the downward-looking recognition result of the downward-looking camera on the preset code band and the number of identification codes corresponding to the downward-looking recognition result of each frame, the method further includes: in response to the number of identification codes in the current frame being less than the number of identification codes in the historical frames, determining the recognition deviation between the downward-looking recognition result of the current frame and the docking position; in response to the recognition deviation being greater than the recognition deviation threshold, and the pose deviation between the current recognition result of the robot in the current frame and the historical recognition result of the robot in the historical frames being greater than the pose deviation threshold, ignoring the downward-looking recognition result of the current frame.

[0014] A second aspect of this application provides a multi-camera robot docking device. The robot is equipped with a downward-looking camera and a forward-looking camera. The downward-looking camera is used to identify a preset code strip on the ground on which the robot travels, and the forward-looking camera is used to identify a shelf identifier on the shelf to which the robot docks. The device includes: a docking preparation module for controlling the robot to move towards a pre-marked preparation point, the preparation point being the position where the robot adjusts its posture before docking with the shelf; a downward-looking recognition module for acquiring the forward-looking recognition result of the forward-looking camera on the shelf identifier in response to the robot not recognizing the preset code strip during its movement to the preparation point; a forward-looking guidance module for guiding the robot to move according to the forward-looking recognition result until the downward-looking camera recognizes the preset code strip; a docking determination module for determining the docking position of the docking point based on the downward-looking recognition result of the downward-looking camera on the preset code strip and the docking parameters of the pre-marked docking point; the docking point being the position where the robot docks with the shelf; and a docking module for controlling the robot to move to the docking position and dock with the shelf.

[0015] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described multi-camera-based robot docking method.

[0016] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described multi-camera-based robot docking method.

[0017] The above scheme equips the robot with a downward-facing camera and a forward-facing camera. The downward-facing camera identifies a pre-set code strip on the ground where the robot travels, while the forward-facing camera identifies shelf markings on the shelf the robot will dock with. The robot is pre-calibrated at a docking point and a preparation point. The docking point is the position where the robot docks with the shelf, and the preparation point is the position where the robot adjusts its posture before docking. During the docking task, the robot moves towards the pre-calibrated preparation point. If the downward-facing camera fails to identify the pre-set code strip while moving to the preparation point, the forward-facing camera's recognition result of the shelf markings is obtained. The robot is guided to move using the forward-facing recognition result until the downward-facing camera identifies the pre-set code strip. Then, the downward-facing camera's recognition result of the pre-set code strip and the docking parameters of the pre-calibrated docking point are used to determine the docking position. The robot is then controlled to move to the docking position and dock with the shelf, completing the docking task. This allows the robot to be guided by the recognition result of the forward-facing camera when the downward-facing camera fails to identify the pre-set code strip, increasing the probability of the robot scanning the code strip and improving the docking accuracy and success rate during task execution.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0020] Figure 1 This is a schematic diagram of an exemplary single-cantilever robot application scenario in the multi-camera-based robot docking method of this application;

[0021] Figure 2 This is a flowchart illustrating an exemplary embodiment of the multi-camera-based robot docking method of this application;

[0022] Figure 3 This is an exemplary scenario diagram of a front-view camera recognizing shelf markings in the multi-camera-based robot docking method of this application;

[0023] Figure 4 This is an exemplary code strip cutting scenario diagram in the multi-camera-based robot docking method of this application;

[0024] Figure 5This is a block diagram illustrating a multi-camera-based robot docking device as an exemplary embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0026] Figure 7 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0027] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0028] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0029] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0030] To facilitate understanding, some technical terms involved in the application scenarios of this application are explained below.

[0031] Common robots typically include, but are not limited to, the following components:

[0032] Odometry: Provides movement parameters such as linear velocity and angular velocity, enabling a preliminary estimate of the change in robot distance traveled. A common type is the wheel encoder, which estimates the robot's current position based on the position change between consecutive time points and the position at the previous time point.

[0033] Two-dimensional lidar: A sensor used to acquire two-dimensional planar information. It can detect the two-dimensional planar contour information of the surrounding environment and can also detect highly reflective materials.

[0034] Multi-sensor fusion refers to the process in AGV and / or AMR systems that integrates observation data from multiple sensors, uses timestamps and extrinsic parameters between sensors for data synchronization and fusion, and ultimately achieves high-precision observation and calculation results that are consistent in time and space.

[0035] In addition, some artificial visual markers are also needed in application scenarios: artificially designed structured markers for visual recognition, such as QR codes or other graphic codes.

[0036] Traditional methods typically use relative navigation to control the robot to meet and dock with the shelf. This requires ensuring that the robot's vision sensors can observe the target throughout the entire docking process. Otherwise, if there is no real-time feedback, it will be difficult to complete the subsequent docking. This method has a low fault tolerance rate and cannot meet the high precision and high stability requirements of complex scenarios.

[0037] The robot docking method in this application is based on a multi-camera solution. In addition to setting the above-mentioned example components, it also sets up a downward-looking camera and a forward-looking camera to ensure the accuracy and reliability of the robot in performing docking tasks.

[0038] It should be noted that the method in this application is mainly illustrated using a single cantilever robot as an example, but it does not limit the application of the method to other types of robots, which will not be elaborated upon hereafter.

[0039] You can refer to, for example Figure 1 As shown, Figure 1 This is an exemplary application scenario diagram of a single cantilever robot in the multi-camera-based robot docking method of this application. In this application, the robot docks with the shelf mainly for picking up or placing materials. In the operation scenario of the single cantilever robot, the materials are mainly stored in the form of material cylinders or rolls. The robot can pick up the materials by extending the cantilever into the material cylinder. For details, please refer to relevant materials in this field, which will not be elaborated here.

[0040] For example, in Figure 1 In the example application scenario, the chassis of a single-cantilever AMR robot can be equipped with at least one downward-looking camera. The downward-looking camera is used to identify a preset code strip on the ground on which the robot travels to obtain relevant pose information. The preset code strip can contain several two-dimensional identification codes (downward-looking identification codes). The robot's cantilever can be equipped with at least one forward-looking camera. The acquisition direction of the forward-looking camera is parallel to the robot's movement direction when performing docking tasks (or it can be said to be parallel to the axis of the cantilever). The forward-looking camera is used to identify the shelf markings on the shelf that the robot docks with. The alignment of the cantilever with the material cylinder can be adjusted based on the identification results of the shelf markings by the forward-looking camera.

[0041] Before the robot can perform the docking task, the docking point and preparation point need to be pre-calibrated. The calibration parameters of the robot at the docking point and preparation point can be stored in the form of the relative pose between the robot and the downward-looking identifier (e.g., docking point). and preparation points The preparation point is the position where the robot adjusts its posture before docking with the shelf (for example, the robot adjusts its posture at the preparation point to ensure that the center of the cantilever aligns with the center of the material cylinder so that the cantilever can be correctly extended into the material cylinder during the subsequent docking process); the docking point is the position where the robot docks with the shelf (for example, after the robot reaches the docking point, that is, after the cantilever has been correctly extended into the material cylinder, it can control the gripping parts on the cantilever to grip the material cylinder on the shelf).

[0042] In subsequent practical applications, whenever the robot's downward-facing camera successfully identifies a preset code strip, it can calculate the positions of the preparation point and docking point based on calibration parameters. Path coverage using the downward-facing recognition results can eliminate the cumulative error of the odometer and improve docking accuracy.

[0043] Please see Figure 2 , Figure 2 This is a flowchart illustrating an exemplary embodiment of the multi-camera-based robot docking method of this application. Specifically, it may include the following steps:

[0044] Step S110: Control the robot to move towards the pre-marked preparation point, which is the position where the robot adjusts its posture before docking with the shelf.

[0045] For example, during the execution of a complete docking task, the robot typically receives the docking task at a certain location and then uses LiDAR and odometry navigation to reach the preparation point corresponding to the docking task. This process is a common navigation method in this field and will not be elaborated upon here.

[0046] However, traditional navigation methods may cause the robot to deviate from the preparation point due to accumulated errors in the odometry. Therefore, the method in this application can pre-calibrate the preparation point position when the robot begins to move towards the preparation point. As a reference value, during the movement to the preparation point, as long as the downward-looking camera can successfully identify the preset code strip and obtain the downward-looking recognition result, the position of the preparation point can be re-determined based on the downward-looking recognition result, and the path can be directly covered (such as using the preparation point position determined by the real-time downward-looking recognition result to replace the reference value) to eliminate the cumulative error of the odometer.

[0047] It should be noted that since the robot may still reach the preparation point when navigating based on the reference value (with or without deviation), this application does not require the robot's downward-facing camera to identify the preset code strip on the ground during the process of moving to the preparation point.

[0048] Step S120: In response to the robot not recognizing the preset code strip during the process of moving to the preparation point, the forward-looking camera obtains the forward-looking recognition result of the shelf marking.

[0049] In conjunction with the steps described above, if the robot's downward-facing camera fails to recognize the preset code tape during its movement to the preparation point (this may be due to a significant deviation of the robot during its movement, or the code tape being dirty or damaged), then the traditional navigation method may cause the robot to eventually arrive at a location that is somewhat different from the preparation point.

[0050] To address the issue that the robot's downward-facing camera cannot recognize the preset code strip, this application utilizes the robot's forward-facing camera to identify the shelf markings, guiding the robot to adjust its posture and continue moving to attempt to scan other downward-facing marking codes within the preset code strip.

[0051] In this case, the robot can activate the forward-looking camera for identification as soon as it receives the docking task, or it can activate the forward-looking camera for identification if the downward-looking camera still fails to identify the preset code strip after the robot has moved a certain distance to the preparation point. There is no limitation here.

[0052] For example, the method for determining whether to turn on the front-view camera may include, but is not limited to: setting a preset distance threshold (such as about 2-3cm, which is not limited here); obtaining the robot's current position according to a navigation method such as LiDAR + odometry; if the distance difference between the current position and the preparation point position (reference value) is less than the preset distance threshold, and the lower-view camera has not yet recognized the preset code strip (that is, the preset code strip has not been recognized when the robot is about to reach the preparation point), then the front-view camera can be turned on for recognition.

[0053] Furthermore, one can refer to, for example Figure 3 As shown, Figure 3 This is an exemplary scenario diagram of a front-view camera recognizing shelf markings in the multi-camera-based robot docking method of this application. The plane on which the shelf markings are located can be perpendicular to the acquisition direction of the front-view camera, and the front-view camera can be positioned at the center of the cantilever.

[0054] Specifically, when the robot is about to reach the preparation point, if its downward-facing camera has not yet recognized the preset code strip, the forward-facing camera can be activated to recognize the shelf markings on the shelf, thus obtaining the forward-facing recognition result. The forward-facing recognition result can represent the relative pose of the forward-facing camera (or the cantilever center) and the shelf markings.

[0055] Step S130: Guide the robot to move according to the forward-looking recognition result until the downward-looking camera recognizes the preset code strip.

[0056] In conjunction with the preceding steps, the prerequisite for successful docking between the robot and the shelf is that the robot's cantilever arm can accurately extend into the center of the material cylinder on the shelf. Otherwise, the cantilever arm may collide with the shelf or other objects. Therefore, in the method of this application, if the robot's downward-facing camera is to continue attempting to identify the preset code strip, the robot's movement process needs to be guided based on the forward-facing recognition results to avoid problems such as the robot's cantilever arm colliding with other objects.

[0057] For example, the height difference between the shelf label and the center of the material cylinder is preset, and the horizontal difference between the shelf label and the center of the material cylinder is also preset. In this application, the distance (such as height difference or horizontal difference) between the cantilever center and the shelf label can be determined based on the forward-looking recognition result of the shelf label by the forward-looking camera. Then, the vertical height that the cantilever needs to be adjusted is determined based on the height difference between the cantilever center and the shelf label, and the height difference between the shelf label and the center of the material cylinder. Similarly, the horizontal offset that the cantilever needs to be adjusted can be determined based on the horizontal difference between the cantilever center and the shelf label, and the horizontal difference between the shelf label and the center of the material cylinder.

[0058] Optionally, when setting shelf markers, the straight line formed by the center of the shelf marker and the center of the material cylinder can usually be made perpendicular to the ground on which the robot travels. This makes the positional deviation between the center of the shelf marker and the center of the material cylinder almost non-existent in the horizontal direction, which facilitates subsequent pose calculation and other processing (that is, the horizontal difference between the cantilever center and the shelf marker center is equivalent to the horizontal difference between the cantilever center and the center of the material cylinder).

[0059] Therefore, in this application, adjusting the robot's cantilever according to the forward recognition result to align the center of the cantilever with the center of the barrel, and then controlling the robot to continue moving, can minimize the risk of collisions during the robot's continued movement.

[0060] Understandably, pre-set barcode tapes are typically placed partly inside the shelf and partly outside. Barcode tapes outside the shelf are generally more susceptible to contamination and damage (the further away from the shelf, the higher the probability and extent of damage), while the tapes inside the shelf are relatively clean. Therefore, in this application, if the downward-facing camera still cannot identify the barcode tape when the robot is about to move to the preparation point, the cantilever can be adjusted based on the forward-facing camera's recognition of the shelf markings, guiding the robot to continue moving and docking, allowing the robot to attempt to identify the clean barcode tape inside the shelf.

[0061] Step S140: Determine the docking position of the docking point based on the downward recognition result of the downward-looking camera on the preset code tape and the docking parameters of the pre-calibrated docking point; the docking point is the position where the robot docks with the shelf.

[0062] Based on the steps described above, if the downward-looking camera successfully identifies the code tape under the guidance of the forward-looking recognition result, the docking position of the docking point can be determined according to the downward-looking recognition result of the downward-looking camera on the preset code tape and the docking parameters of the pre-calibrated docking point.

[0063] For example, when the downward-facing camera recognizes the downward-facing identifier code in the code band, the current downward-facing recognition result is obtained (the relative pose of the recognized downward-facing identifier code and the current position of the downward-facing camera). Pre-calibrated external parameters of the downward-looking camera The robot's current pose and the pre-calibrated docking parameters of the docking points. This allows us to determine the docking position of the docking point. Its mathematical expression can be:

[0064]

[0065] Step S150: Control the robot to move to the docking position and dock with the shelf.

[0066] In conjunction with the steps described above, after the robot's downward-facing camera recognizes the preset code strip, the location of the target point that the robot needs to go to (such as the docking position of the docking point) can be determined based on the downward-facing camera's recognition result of the preset code strip and some parameters obtained through pre-calibration.

[0067] Therefore, in this step, the robot can be controlled to navigate and move to the docking position (i.e., docking point) for docking. After reaching the docking position, the cantilever can also be controlled to pick up materials on the shelf, which will not be elaborated here.

[0068] As can be seen, this application equips the robot with a downward-looking camera and a forward-looking camera. The downward-looking camera is used to identify a preset code strip on the ground where the robot travels, and the forward-looking camera is used to identify the shelf markings on the shelf the robot will dock with. The robot is pre-calibrated at a docking point and a preparation point. The docking point is the position where the robot docks with the shelf, and the preparation point is the position where the robot adjusts its posture before docking with the shelf. During the docking task, the robot is controlled to move towards the pre-calibrated preparation point. If the downward-looking camera does not identify the preset code strip while the robot is moving to the preparation point, the forward-looking recognition result of the shelf markings from the forward-looking camera is obtained. The robot is guided to move using the forward-looking recognition result until the downward-looking camera identifies the preset code strip. Then, the docking position of the docking point is determined by the downward-looking recognition result of the preset code strip from the downward-looking camera and the docking parameters of the pre-calibrated docking point. The robot is then controlled to move to the docking position and dock with the shelf, completing the docking task. This allows the robot to be guided by the recognition results of the front-view camera when the robot's downward-facing camera cannot recognize the preset code tape, increasing the probability of the robot scanning the code tape and improving the docking accuracy and success rate of the robot when performing tasks.

[0069] Based on the above embodiments, the embodiments of this application describe the pre-calibration process before step S110. Specifically, the pre-calibration process mainly uses a downward-looking camera and an odometer to determine the location of the docking point and the location of the preparation point.

[0070] The pre-calibration sequence can be either calibrating the docking point first and then the preparation point, or vice versa; there is no limitation here. To improve calibration efficiency and ensure that the robot can always travel from the preparation point to the docking point, this application can choose to calibrate the docking point first and then the preparation point. That is, first control the robot to reach the docking point inside the shelf for calibration, and then control the robot to move to the preparation point outside the shelf for calibration.

[0071] For example, during the pre-calibration process, the robot's cantilever hollowly extends into the material cylinder on the shelf and stops at a position where the cantilever can dock with the material cylinder. At this time, the relative pose of the downward-looking camera and the reference code on the recognized preset code tape is calculated and denoted as... This pose is the docking parameter between the robot and the shelf at the docking point.

[0072] At the same time, record the robot's pose at that point as When the robot docks with the shelf, it needs to first reach a preparation point outside the shelf. The length of the pre-set code strip on the ground is usually longer than the distance between the preparation point and the shelf; let's denote the length of the pre-set code strip outside the shelf as LM (e.g., 10cm). Under normal circumstances, the robot can reach the preparation point by recognizing the pre-set code strip using a downward-facing camera. After adjusting the robot body and cantilever at the preparation point, it then reaches the docking point by recognizing the pre-set code strip again using the downward-facing camera.

[0073] The method for adjusting the robot body and cantilever at the preparation point may include, but is not limited to: using the forward-looking camera set on the cantilever to identify the shelf identification code on the shelf to adjust the vertical and horizontal position of the cantilever (if the cantilever does not support horizontal adjustment, the cantilever can be adjusted horizontally by adjusting the robot chassis), so that the center of the cantilever is aligned with the center of the material cylinder, and then the entire robot begins to enter the shelf for docking along the preset code strip.

[0074] It should be noted that since the robot usually enters in a straight line during the docking process with the shelf (from the preparation point to the docking point), it also exits in a straight line when it moves to the preparation point after the docking point is calibrated. Therefore, it can be reasonably assumed that both the docking point and the preparation point are on the robot's straight travel path.

[0075] For example, taking the docking direction of the robot as the x-axis of the map coordinate system, the difference between the preparation point and the docking point is only in the x-axis direction, while the values ​​of the other five dimensions are the same.

[0076] Specifically, when the robot is at the preparation point, the depth distance between the front-view camera and the shelf identification code is denoted as L1. Additionally, the length of each material cylinder on the shelf is known in advance as L2, and the distance between any two material cylinders is L3. If the number of material cylinders on the shelf is N, then the mathematical expression of the calibration parameters (preparation parameters) of the preparation point on the x-axis dimension can be:

[0077]

[0078] Similarly, the robot's pose at the preparation point The robot's pose at the docking point can also be determined by the position of the robot. It is derived from calculation. For example, its mathematical expression can be:

[0079]

[0080] Based on the above embodiments, this application embodiment describes step S130. Specifically, the method of guiding the robot to move according to the forward-looking recognition result in step S130 may further include the following steps S131 to S132. The forward-looking recognition result may include distance information between the robot and the shelf label.

[0081] Step S131: Compare the distance information with the safe distance threshold to obtain the distance comparison result.

[0082] Understandably, if the robot is guided to continue moving based on the forward recognition result when the downward-facing camera cannot recognize the preset code tape, the closer the robot is to the shelf, the higher the probability of collision between the cantilever and the shelf or the material cylinder. Alternatively, the robot may dock with the shelf in the wrong posture (even if no collision occurs, it may still fail to successfully grip the material cylinder).

[0083] Therefore, this application also proposes a method for guiding the robot to move based on the forward recognition results: different guiding movement methods can be adopted according to the distance between the robot (or cantilever) and the shelf to ensure the safety and reliability of the robot during the guided movement process.

[0084] The safe distance threshold is used to determine how close the robot is to the shelf, and also represents the risk level of the forward-looking camera guiding the robot to continue moving.

[0085] For example, when the forward-looking camera recognizes a shelf label, it can obtain the depth value (i.e., distance information) between the forward-looking camera and the shelf label. There are various methods for obtaining depth values ​​in this technical field, which will not be elaborated here. The real-time distance information is compared with a preset safe distance threshold to obtain a distance comparison result.

[0086] Step S132: Guide the robot to move based on the distance comparison results.

[0087] Distance comparison results can include distance information greater than or equal to the safe distance threshold (relatively safe), or distance information less than the safe distance threshold (relatively dangerous).

[0088] For example, if the robot is far from the shelf, it can continue moving using the existing navigation method. If the robot is close to the shelf, its pose needs to be adjusted first (including but not limited to adjusting the robot chassis and / or cantilever) before moving.

[0089] Based on the above embodiments, this application embodiment describes step S132. Specifically, the method of guiding the robot to move according to the distance comparison result in step S132 may further include the following steps S1321 to S1323.

[0090] Step S1321: In response to the distance comparison result indicating that the distance information is greater than or equal to the safe distance threshold, control the robot to continue moving.

[0091] For example, taking a safe distance threshold of 4cm as an example. Let the relative pose of the forward-looking camera and the shelf label be when the forward-looking camera recognizes the shelf label. The coordinate system for setting shelf labels can be configured with the x-axis pointing vertically upwards and the y-axis pointing to the left, creating a right-handed coordinate system.

[0092] when When the depth value (distance information) is greater than or equal to 4cm, it indicates that the robot body is relatively far from the shelf, and the robot's current movement is relatively safe. Even if the robot's pose crosses the preparation point... It can also safely continue moving into the shelf. Therefore, the robot can be controlled to continue moving.

[0093] Furthermore, once the downward-looking camera recognizes the preset code strip during continued movement, the final docking point (docking position) is calculated based on the currently recognized downward-looking identifier code. The calculation method is similar to the method for calculating the docking position provided in the previous example, but the target point to be reached requires the use of corresponding calibration parameters. That is, the docking parameters in the aforementioned method need to be replaced with the parameters that the robot has pre-calibrated at the preparation point (preparation parameters, denoted as...). ).

[0094] At this point, you can choose to stop immediately and adjust the height of the cantilever based on the height difference between the cantilever and the shelf identification code as detected by the forward-looking camera, and the height difference between the shelf identification code and the center of the material cylinder. Similarly, use the left-right deviation of the robot chassis scanned by the forward-looking camera to adjust the left-right position of the cantilever (or the cantilever can support independent left-right adjustment), so that the cantilever can be aligned with the center of the material cylinder on the same central axis. After adjusting the cantilever, the downward-looking camera is used to calculate the position of the docking point.

[0095] In step S1322, in response to the distance comparison result indicating that the distance information is less than the safe distance threshold, the robot movement is paused and the robot's cantilever is adjusted to align with the material cylinder on the shelf according to the forward recognition result.

[0096] when When the depth value (distance information) is less than 4cm, it indicates that the robot body is too close to the shelf, and the robot's current movement is relatively dangerous. If the downward-facing camera still cannot recognize the preset code tape at this time, the robot's movement needs to be paused. Adjust the robot's cantilever and / or chassis according to the forward-facing recognition results to align the robot's cantilever with the material cylinder on the shelf.

[0097] Step S1323: Control the robot to continue moving after the cantilever has been adjusted.

[0098] After adjusting the cantilever based on the forward-looking recognition results to ensure alignment between the cantilever and the material cylinder, the cantilever will extend into the center of the material cylinder to avoid collisions as the robot continues to move according to the docking direction. Therefore, the robot can be controlled to continue moving after the cantilever adjustment.

[0099] Based on the above embodiments, this application embodiment describes step S1322. Specifically, the method of adjusting the alignment of the robot's cantilever with the material cylinder on the shelf according to the forward recognition result in step S1322 may further include steps S1324 to S1325.

[0100] Step S1324: Determine the vertical and horizontal deviations between the cantilever center and the barrel center of the cantilever based on the forward recognition results.

[0101] Referring to the foregoing embodiments, the vertical deviation of the docking is equivalent to the vertical distance deviation between the cantilever center (forward-looking camera) and the shelf markings. Here, the vertical direction refers to the direction perpendicular to the ground, or it can be considered as the vertical direction.

[0102] The horizontal deviation of the docking is equivalent to the horizontal distance deviation between the cantilever center (forward-looking camera) and the shelf markings. Here, the horizontal direction refers to the direction perpendicular to the docking direction and parallel to the ground, which can also be considered as the radial direction of the cantilever parallel to the ground, or it can be called the lateral direction.

[0103] For example, by using the method provided in the foregoing embodiments, after obtaining the forward recognition result (the relative pose of the forward camera and the shelf identification code), the vertical deviation and horizontal deviation of the docking between the cantilever center and the material cylinder center can be determined.

[0104] Step S1325: Adjust the robot according to the vertical and horizontal deviations of the docking until the center of the cantilever is aligned with the center of the barrel.

[0105] For example, by adjusting the vertical height of the robot's cantilever and the lateral movement of the chassis (which moves the cantilever left and right), vertical and horizontal deviations during docking can be eliminated, ultimately aligning the center of the cantilever with the center of the material cylinder. This means the robot can extend the cantilever into the center of the material cylinder simply by moving according to the docking direction, avoiding collisions or incorrect docking.

[0106] Based on the above embodiments, this application embodiment describes step S1324. The forward-looking recognition result further includes the recognition height difference and recognition horizontal difference between the cantilever center and the shelf marking. Specifically, the method for determining the vertical and horizontal alignment deviations between the cantilever center and the material cylinder center based on the forward-looking recognition result in step S1324 may further include steps S1326 to S1328.

[0107] Step S1326: Obtain the height difference between the shelf marking and the center of the material cylinder.

[0108] Here, the placement height difference refers to the distance difference between the shelf label and the center of the material cylinder in the direction perpendicular to the ground. For example, for ease of calculation and analysis, it can be the distance difference between the geometric center of the shelf label and the center of the cross-section in the docking direction of the material cylinder. Taking a cylindrical material cylinder as an example, the center of the material cylinder here also corresponds to the center of the circular cross-section of the material cylinder perpendicular to the docking direction.

[0109] Step S1327: Determine the docking vertical deviation based on the identification height difference and the placement height difference.

[0110] The height difference refers to the distance difference in the height direction obtained by the forward-looking camera recognizing the shelf identification code.

[0111] The vertical deviation of the docking refers to the height that the cantilever needs to be adjusted when the robot docks, which is also equivalent to the height of the cantilever center from the center of the barrel.

[0112] If the height difference is H bc The height difference is H. t Then, the mathematical expression for the vertical deviation H of the docking can be:

[0113] H = H bc +H t

[0114] Step S1328: Identify the horizontal difference as the docking horizontal deviation.

[0115] The horizontal difference in recognition refers to the difference in distance in the horizontal direction (lateral direction) obtained by the forward-looking camera recognizing the shelf identification code.

[0116] For example, taking the application scenario of step S1322 as an example (distance information is less than the safe distance threshold). If the distance information between the robot and the cantilever front-view camera and the shelf identification code is less than 4cm, and the downward-view camera still cannot scan the preset code strip, then the robot will stop.

[0117] At this point, the vertical deviation of the docking is determined by the forward-looking camera's recognition of the shelf identification code, and the height of the cantilever is adjusted. Similarly, the horizontal deviation of the docking relative to the shelf identification code, identified by the forward-looking camera, is used to adjust the position of the chassis so that the cantilever is aligned with the material cylinder on the same central axis.

[0118] For example, if the front-view camera detects that the robot is 2cm to the left of the shelf label but has no height deviation, then the robot chassis is adjusted 2cm laterally to the right. At this point, the robot continues to move a certain preset distance (e.g., 0.5m) along the current orientation using the odometer navigation mode. Tests have shown that the lateral deviation of the chassis odometer is not too large when moving 0.5m, and it is within a safe and controllable range.

[0119] Therefore, as the robot moves a predetermined distance into the shelf, once the downward-facing camera identifies the pre-defined code strip on the ground, the docking point is calculated based on the downward-facing camera's recognition result and the current odometer pose. Even though the pre-defined code strip outside the shelf is easily contaminated and damaged due to lack of obstruction, the pre-defined code strip inside the shelf is relatively clean. Therefore, by guiding the robot to continue moving based on the forward-facing camera's recognition result of the shelf identification code, there is a high probability that the robot will identify the code strip inside the shelf.

[0120] It should be noted that the method described in the foregoing embodiments can be used to guide the robot with the forward-looking camera to find the downward-looking identifier code corresponding to the preparation point, or it can be used to guide the robot with the forward-looking camera to find the downward-looking identifier code corresponding to the docking point. For example, if the downward-looking camera has already identified a certain downward-looking identifier code in the code tape before the robot reaches the preparation point, but cannot identify the corresponding downward-looking identifier code at the preparation point, making it impossible to directly calculate the docking point inside the shelf, then the forward-looking camera can be used in the same way to guide the robot to find the code.

[0121] It should also be noted that if the robot arrives If the forward-facing camera fails to recognize the shelf identification code, it indicates that the robot has strayed too far from the designated path or that the shelf identification code is contaminated or damaged. Therefore, it can generate corresponding prompts and report them promptly.

[0122] Based on the above embodiments, this application embodiment describes step S140. Specifically, the method for determining the docking position of the docking point in step S140 based on the downward view recognition result of the downward view camera on the preset code band and the docking parameters of the pre-calibrated docking point may further include the following steps S141 to S142.

[0123] Step S141: Obtain the robot's current pose and the external parameters of the downward-facing camera.

[0124] Referring to the foregoing embodiments, the robot's current pose at time t is denoted as... The external parameters of the downward-viewing camera are denoted as follows: .

[0125] Step S142: Determine the docking position based on the downward view recognition result, current pose, downward view camera extrinsic parameters, and docking parameters.

[0126] Based on the current downward-looking recognition results Current pose External parameters of the downward-looking camera and the docking parameters pre-calibrated at the docking point This allows you to determine the docking position.

[0127] Based on the above embodiments, this application embodiment describes the steps after step S130. The preset code band includes multiple identification codes (downward-looking identification codes). Specifically, after guiding the robot to move according to the forward-looking recognition result in step S130 until the downward-looking camera recognizes the preset code band, the method may further include steps S230 to S231.

[0128] Step S230: Obtain the downward-view recognition result of the downward-view camera on the preset code band and the number of identification codes corresponding to each frame of downward-view recognition result.

[0129] It should be noted that in most scenarios, when the downward-looking camera acquires images of a preset code band, each frame may include multiple downward-looking identifiers (e.g., four). However, if one or more downward-looking identifiers are dirty or damaged, only one or two may be identified. Although the downward-looking identifiers are identified, such identification results may contain anomalies. Therefore, this application also proposes an outlier removal method during the downward-looking camera's identification of the preset code band, which removes identified outliers to ensure a high docking success rate.

[0130] For example, when obtaining the downward view recognition result of each frame of the preset code band by the downward view camera, it is also necessary to obtain the number of identification codes identified in each frame of the downward view recognition result.

[0131] Step S231: In response to the fact that the number of identification codes in the current frame is greater than the number of identification codes in the historical frames, the docking position is updated according to the downward recognition result of the current frame. The acquisition timing of the current frame is later than the acquisition timing of the historical frames.

[0132] It should also be noted that if the number of identification codes recognized by the downward-looking camera decreases during the robot's docking with the shelf, it may indicate that there are dirty or damaged identification codes in the current downward-looking recognition results.

[0133] Therefore, in this embodiment, the number of identifier codes identified in the current frame can be compared with the number of identifier codes identified in the historical frames to determine whether there are any outliers. The acquisition timing of the current frame is later than that of the historical frames, and the current frame and the historical frames can be adjacent or non-adjacent frames; this is not limited here.

[0134] For example, when the downward-facing camera first recognizes the downward-facing identifier and obtains the corresponding downward-facing recognition result (such as the docking point position in the map coordinate system), it will record the first downward-facing recognition result as a reference value. Simultaneously, the number of identifier codes corresponding to this downward-looking recognition result is recorded as N. bt0 .

[0135] During the robot's subsequent movements, the initial downward-looking recognition result becomes the downward-looking recognition result of the historical frame. If the number of identifiers recognized in the current frame is greater than N... bt0 This means that the current frame's bottom-view recognition result is more reliable than the bottom-view recognition result of historical frames (the current reference value, such as the docking position of the docking point determined based on the bottom-view recognition result of historical frames).

[0136] Furthermore, the reference value of the docking position can be updated using the downward-looking recognition result of the current frame. During the robot's subsequent movement, the updated docking position can be used as a new position reference for the target point, and the robot can then move accordingly.

[0137] Based on the above embodiments, this application embodiment describes the steps after step S230. Specifically, after obtaining the downward view recognition result of the downward view camera on the preset code band and the number of identification codes corresponding to each frame of downward view recognition result in step S230, the method may further include the following steps S232 to S233.

[0138] Step S232: In response to the fact that the number of identification codes in the current frame is less than the number of identification codes in the historical frames, determine the identification deviation between the downward recognition result of the current frame and the docking position.

[0139] In conjunction with the foregoing embodiments, if the number of identifier codes identified in the current frame is less than or equal to Nbt0, the downward recognition result of the current frame may have an abnormality.

[0140] For example, in this embodiment, the main focus is on combining the downward-looking recognition results of the current frame with the downward-looking recognition results (reference values) of historical frames. The lateral deviation between the two is determined as the recognition deviation, and then it is used to determine whether there are any abnormal problems in the downward recognition results of the current frame.

[0141] Step S233: In response to the recognition deviation being greater than the recognition deviation threshold, and the pose deviation between the robot's current recognition result in the current frame and the robot's historical recognition result in the historical frame being greater than the pose deviation threshold, the downward recognition result of the current frame is ignored.

[0142] Among them, pose deviation refers to the deviation between the current recognition result of the current frame and the historical recognition result of the historical frame. It is mainly used to measure whether there is a large shift in the downward recognition result between two adjacent frames.

[0143] It should be noted that, numerically, historical recognition results may be the same as or different from the baseline value (depending on whether the baseline value has been updated). However, in practice, comparing the current recognition result with historical recognition results and comparing the current recognition result with the baseline value have different effects.

[0144] In conjunction with the steps described above, determine whether the recognition deviation is greater than the preset recognition deviation threshold σy (e.g., 2cm, which can be set according to the accuracy requirements and is not limited here).

[0145] If the identification deviation is greater than the identification deviation threshold, it indicates that the docking point determined based on the downward identification result of the current frame is abnormal (it may be due to abnormal downward identification result or abnormal odometry pose estimation, which will not be elaborated here).

[0146] Furthermore, by simultaneously using the downward-looking recognition results of the current frame and the downward-looking recognition results of historical frames (such as the frame preceding the current frame), the pose deviation between the current recognition result (the pose of the docking point determined in the current frame) and the historical recognition result (the pose of the docking point determined in the historical frame) is calculated. Pose deviations include angle deviations and left-right displacement deviations, and the pose deviation threshold can also include angle deviation thresholds and displacement deviation thresholds.

[0147] Specifically, if the angle deviation is greater than the preset angle threshold σa (e.g., 0.5°) or the left / right displacement deviation is greater than the displacement deviation threshold σt (e.g., 0.01m), then the current recognition result of the current frame may be abnormal. Therefore, the current recognition result of the current frame is ignored (i.e., the current recognition result is not selected for path coverage, and the reference value is not updated).

[0148] If the downward gaze recognition results for multiple consecutive frames (e.g., 5 frames) are ignored according to the example method described above, it may indicate that the robot's current pose has shifted significantly compared to the pose 5 frames ago. In this case, the reference value can be updated based on the current downward gaze recognition results. The robot is then controlled to move towards the updated reference value, thus adjusting the robot's pose.

[0149] Based on the above embodiments, this application also proposes a code tape cutting method to remove the identification results of the robot for the code tape parts that are easily dirty or damaged, so as to ensure docking accuracy.

[0150] For example, reference can be made to, such as Figure 4 As shown, Figure 4 This is an exemplary schematic diagram of a code tape cutting scenario in the multi-camera-based robot docking method of this application.

[0151] It is understandable that the barcode tape on the outside of the shelf is easily damaged by wheels or contaminated by on-site operators, which may cause the robot to fail to detect the downward-facing barcode in that area or result in incorrect recognition results.

[0152] Therefore, in this embodiment, the starting point id of the code tape cutting can be pre-set as S, and the ending point id as E, for the code tape located outside the shelf. In practical applications, when the robot detects the starting point S, it records the current odometry pose T.os The downward view recognition result of this frame is The distance between the starting point S and the ending point E of the code tape cutting is L. SE Among them, L SE It can be pre-calibrated, or it can be calculated based on the size of the downward view identifier and the code spacing between each downward view identifier; there is no limitation here.

[0153] As the robot moves to the end point E of the code tape cutting process, if the current ID of the currently identified downward-looking identifier is between S and E, and the odometry pose T corresponding to the current frame... ot To T os The distance is less than L SE If the robot's downward-facing camera is still in the code strip cutting section, then the current downward-facing recognition result can be ignored, and the downward-facing recognition result before entering the code strip cutting section should still be reported. This serves as a reference value during the robot's navigation process.

[0154] Since the downward-facing camera is located in the code strip cutting section, the function of removing outliers based on the number of identification codes in the aforementioned embodiment can be temporarily disabled.

[0155] The robot continues to Navigation is performed based on the code strip. When the robot's downward-facing camera recognizes the code strip cutting endpoint E, the function of removing outliers based on the number of identification codes in the aforementioned embodiment can be restored; and according to the method provided in the aforementioned embodiment, the preset code strip is recognized to obtain the pose of the currently determined target point, and the real-time updated downward-facing recognition results are reported for path coverage.

[0156] In summary, this application marks a docking point inside the docking rack and records the corresponding robot pose. The preparation parameters and coordinates of the preparation point are calculated based on the length of the material rolls on the rack, the roll gaps, and the depth value of the forward-looking camera at the preparation point. During the docking process, a downward-looking camera is used to identify the code tape on the ground. Once identification is successful, the preparation point coordinates are calculated using the preparation parameters; otherwise, the coordinates of the preparation point are calculated using the coordinates of the docking point. During the journey to the preparation point, if the downward-looking camera cannot scan the preset code tape, the forward-looking camera is used for guided movement to attempt to identify code tape on other parts of the ground.

[0157] When the robot reaches the preparation point but still fails to recognize the preset code strip, the process of using the forward-looking camera to guide the movement can be divided into two layers.

[0158] The first layer works as follows: if the depth value of the shelf identification code detected by the forward-looking camera is within a safe range, the robot is directly guided towards the shelf docking direction. If a downward-looking identification code is detected during the guidance process, the subsequent navigation is directly taken over by the downward-looking recognition result of the downward-looking camera.

[0159] The second layer is as follows: If the depth value detected by the forward-looking camera exceeds the safe range and the ground marking strip is still not detected, the robot's cantilever and chassis positions are adjusted based on the relative pose of the shelf marking strip detected by the forward-looking camera, and then the odometry mode is used to continue moving a certain distance. If a downward-looking marking strip is detected during the guidance process, the downward-looking recognition result from the downward-looking camera directly takes over the subsequent navigation.

[0160] During the overall docking process, the downward-looking recognition results can continuously cover the path, without requiring recognition results for each frame of the downward-looking recognition process. For example, the downward-looking recognition results obtained in the first instance and the recognition results calculated by the odometry pose can be used as reference values ​​in the subsequent navigation movement process, and the reference values ​​can be updated based on the number of detected downward-looking identifiers.

[0161] Furthermore, the lateral deviation between the current recognition result and the baseline value result can be checked. If the number of identifiers detected in the current frame is less than or equal to the number of identifiers detected in the previous frame, the decision on whether to use the current recognition result of the current frame for path coverage is made based on the lateral displacement deviation and angular deviation between the current frame and the previous frame. If the recognition result is ignored multiple times, the recognition result of the latest frame can be used to update the baseline value.

[0162] In addition, this application also supports the segmentation of the code tape recognition process to avoid the recognition results being affected by easily dirty or damaged code tape areas, which could lead to docking failure.

[0163] It should be further noted that the execution entity of the multi-camera-based robot docking method can be a multi-camera-based robot docking device. For example, the multi-camera-based robot docking method can be executed by a terminal device, a server, or other processing devices. The terminal device can be a user equipment (UE), computer, mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc. In some possible implementations, this multi-camera-based robot docking method can be implemented by a processor calling computer-readable instructions stored in memory.

[0164] Figure 5 This is a block diagram illustrating a multi-camera-based robot docking device, as shown in an exemplary embodiment of this application. Figure 5 As shown, the exemplary multi-camera-based robot docking device 500 includes: a docking preparation module 510, a downward-looking recognition module 520, a forward-looking guidance module 530, a docking determination module 540, and a docking module 550. Specifically:

[0165] The docking preparation module 510 is used to control the robot to move towards a pre-marked preparation point, which is the position where the robot adjusts its posture before docking with the shelf.

[0166] The downward-facing recognition module 520 is used to obtain the forward-facing recognition result of the shelf markings from the forward-facing camera when the robot fails to recognize the preset code strip during its movement to the preparation point.

[0167] The forward guidance module 530 is used to guide the robot to move based on the forward recognition results until the downward camera recognizes the preset code strip.

[0168] The docking determination module 540 is used to determine the docking position of the docking point based on the downward recognition result of the downward camera on the preset code tape and the docking parameters of the pre-calibrated docking point. The docking point is the position where the robot docks with the shelf.

[0169] The docking module 550 is used to control the robot to move to the docking position and dock with the shelf.

[0170] In this exemplary multi-camera-based robot docking device, a downward-looking camera and a forward-looking camera are mounted on the robot. The downward-looking camera identifies a preset code strip on the ground where the robot travels, while the forward-looking camera identifies shelf markings on the shelf the robot will dock with. The robot is pre-calibrated at a docking point and a preparation point. The docking point is the position where the robot docks with the shelf, and the preparation point is the position where the robot adjusts its posture before docking. During the docking task, the robot moves towards the pre-calibrated preparation point. If the downward-looking camera does not identify the preset code strip while moving to the preparation point, the forward-looking recognition result of the shelf markings from the forward-looking camera is obtained. The robot is guided to move using the forward-looking recognition result until the downward-looking camera identifies the preset code strip. The docking position is then determined by comparing the downward-looking recognition result of the preset code strip with the docking parameters of the pre-calibrated docking point. The robot is then controlled to move to the docking position and dock with the shelf, completing the docking task. This allows the robot to be guided by the recognition results of the front-view camera when the robot's downward-facing camera cannot recognize the preset code tape, increasing the probability of the robot scanning the code tape and improving the docking accuracy and success rate of the robot when performing tasks.

[0171] It should be noted that the apparatus and method provided in the above embodiments belong to the same concept, and the specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the apparatus provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation.

[0172] The functions of each module can be found in the embodiment of the robot docking method based on multiple cameras, and will not be repeated here.

[0173] Please see Figure 6 , Figure 6 This is a schematic diagram of an embodiment of the electronic device of this application. The electronic device 100 includes a memory 101 and a processor 102. The processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in any of the above embodiments of the multi-camera-based robot docking method. In a specific implementation scenario, the electronic device 100 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 100 may also include mobile devices such as laptops and tablets, which are not limited here.

[0174] Specifically, processor 102 controls itself and memory 101 to implement the steps in any of the above-described embodiments of the multi-camera-based robot docking method. Processor 102 can also be referred to as a CPU (Central Processing Unit). Processor 102 may be an integrated circuit chip with signal processing capabilities. Processor 102 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 102 can be implemented using integrated circuit chips.

[0175] In this exemplary electronic device, a downward-looking camera and a forward-looking camera are mounted on the robot. The downward-looking camera identifies a preset code strip on the ground where the robot travels, while the forward-looking camera identifies shelf markings on the shelf the robot will dock with. The robot is pre-calibrated at a docking point and a preparation point. The docking point is the position where the robot docks with the shelf, and the preparation point is the position where the robot adjusts its posture before docking with the shelf. During the docking task, the robot is controlled to move towards the pre-calibrated preparation point. If the downward-looking camera does not identify the preset code strip while moving to the preparation point, the forward-looking camera's recognition result of the shelf markings is obtained. The robot is guided to move using the forward-looking recognition result until the downward-looking camera identifies the preset code strip. Then, the docking position of the docking point is determined by the downward-looking camera's recognition result of the preset code strip and the docking parameters of the pre-calibrated docking point. The robot is then controlled to move to the docking position and dock with the shelf, completing the docking task. This allows the robot to be guided by the recognition results of the front-view camera when the robot's downward-facing camera cannot recognize the preset code tape, increasing the probability of the robot scanning the code tape and improving the docking accuracy and success rate of the robot when performing tasks.

[0176] Please see Figure 7 , Figure 7 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor. The program instructions 111 are used to implement the steps in any of the above embodiments of the multi-camera-based robot docking method.

[0177] In this exemplary storage medium, by running the program instructions stored in the medium, the robot is controlled to move towards a pre-marked preparation point during the docking task. The preparation point is the position where the robot adjusts its posture before docking with the shelf. The robot is equipped with a downward-looking camera and a forward-looking camera. The downward-looking camera identifies a preset code strip on the ground where the robot travels, and the forward-looking camera identifies the shelf markings on the shelf to be docked with. If the downward-looking camera does not identify the preset code strip during the robot's movement to the preparation point, the forward-looking camera's recognition result of the shelf markings is obtained. The robot is guided to move using the forward-looking recognition result until the downward-looking camera identifies the preset code strip. Then, the downward-looking recognition result of the preset code strip by the downward-looking camera and the docking parameters of the pre-marked docking point are used to determine the docking position, which is the position where the robot docks with the shelf. The robot is then controlled to move to the docking position and dock with the shelf, completing the docking task. This allows the robot to be guided by the recognition result of the forward-looking camera when the downward-looking camera cannot identify the preset code strip, increasing the probability of the robot scanning the code strip and improving the docking accuracy and success rate during task execution.

[0178] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0179] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0180] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0181] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A robot docking method based on multiple cameras, characterized in that, The robot is equipped with a downward-facing camera and a forward-facing camera. The downward-facing camera is used to identify a preset code strip on the ground where the robot travels, and the forward-facing camera is used to identify shelf markings on the shelf that the robot docks with. The method includes: The robot is controlled to move towards a pre-marked preparation point, which is the position where the robot adjusts its posture before docking with the shelf; In response to the robot failing to recognize the preset code strip during its movement to the preparation point, the forward-looking camera obtains the forward-looking recognition result of the shelf marking. The robot is guided to move according to the forward-looking recognition result until the downward-looking camera recognizes the preset code strip; the preset code strip includes multiple identification codes. The system acquires the downward-looking recognition result of the preset code band by the downward-looking camera and the number of identification codes corresponding to the downward-looking recognition result of each frame; determines the docking position of the docking point based on the downward-looking recognition result of the preset code band by the downward-looking camera and the docking parameters of the pre-calibrated docking point; the docking point is the position where the robot docks with the shelf; in response to the number of identification codes in the current frame being greater than the number of identification codes in the historical frames, the docking position is updated based on the downward-looking recognition result of the current frame, wherein the acquisition time of the current frame is later than the acquisition time of the historical frames; Control the robot to move to the updated docking position and dock with the shelf.

2. The method according to claim 1, characterized in that, The forward-looking recognition result includes distance information between the robot and the shelf label. Guiding the robot to move based on the forward-looking recognition result includes: The distance information is compared with the safe distance threshold to obtain the distance comparison result; The robot is guided to move based on the distance comparison results.

3. The method according to claim 2, characterized in that, The step of guiding the robot to move based on the distance comparison result includes: In response to the distance comparison result indicating that the distance information is greater than or equal to the safe distance threshold, the robot is controlled to continue moving; In response to the distance comparison result indicating that the distance information is less than the safe distance threshold, the robot movement is paused and the robot's cantilever is adjusted to align with the material cylinder on the shelf according to the forward recognition result; The robot continues to move after the control arm has been adjusted.

4. The method according to claim 3, characterized in that, The step of adjusting the robot's cantilever to align with the material cylinder on the shelf based on the forward-looking recognition result includes: Based on the forward recognition results, determine the vertical and horizontal deviations between the cantilever center and the barrel center of the cantilever. The robot is adjusted according to the vertical and horizontal docking deviations until the center of the cantilever is aligned with the center of the barrel.

5. The method according to claim 4, characterized in that, The forward-looking recognition result also includes the recognition height difference and recognition horizontal difference between the cantilever center and the shelf marking. Determining the vertical and horizontal alignment deviations between the cantilever center and the material cylinder center based on the forward-looking recognition result includes: Obtain the height difference between the shelf marking and the center of the material cylinder; The docking vertical deviation is determined based on the identification height difference and the placement height difference; The identification level difference is determined as the docking level deviation.

6. The method according to claim 1, characterized in that, The step of determining the docking position of the docking point based on the downward-view recognition result of the preset code band by the downward-view camera and the docking parameters of the pre-calibrated docking point includes: Obtain the current pose of the robot and the extrinsic parameters of the downward-facing camera; The docking position is determined based on the downward view recognition result, the current pose, the extrinsic parameters of the downward view camera, and the docking parameters.

7. The method according to claim 1, characterized in that, After obtaining the downward-view recognition result of the downward-view camera on the preset code band and the number of identifier codes corresponding to each frame of downward-view recognition result, the method further includes: In response to the fact that the number of identifier codes in the current frame is less than the number of identifier codes in the historical frames, the identification deviation between the downward recognition result of the current frame and the docking position is determined; In response to the recognition deviation being greater than the recognition deviation threshold, and the pose deviation between the robot's current recognition result in the current frame and the robot's historical recognition result in the historical frame being greater than the pose deviation threshold, the downward recognition result of the current frame is ignored.

8. An electronic device, characterized in that, The method includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images