A robot warehouse entering visual identification method and device, electronic equipment and medium

Abstract

The present disclosure relates to the technical field of intelligent robots, and provides a robot upper bin visual identification method and device, electronic equipment and a medium. The method comprises the following steps: obtaining the pose of the robot box relative to the chassis and the detection frame position according to the pose of the upper bin in the environment and the pose of the chassis in the environment; calculating the pose Tbc of the box in the chassis camera coordinate system; obtaining each vertex set V of the detection frame according to the pose of the robot box relative to the chassis and the detection frame position; calculating the position [ui, vi] of each vertex set V of the detection frame in the camera pixel coordinate system; traversing and comparing the position [ui, vi] of each vertex set V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinate [umax, vmax] and the lowest point coordinate [umin, vmin]; labeling the data; and calculating the pose of the robot upper bin relative to the chassis. The method does not need to externally paste a two-dimensional code, is beautiful and convenient, is not affected by the external environment, has a wide range of applications, and has high identification accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent robot technology, and in particular to a robot upper warehouse visual recognition method, device, electronic device and medium. Background Technology

[0002] With the rapid development of technology, robotics is also advancing by leaps and bounds. Among these advancements, robots with a separate chassis and upper compartment design are particularly popular in the market. For robots with this design, obtaining the position and orientation of the upper compartment relative to the chassis has become an important research topic.

[0003] Existing technologies generally involve either pasting QR codes on the upper part of the warehouse or using deep learning. The former has too much impact on the appearance and is not aesthetically pleasing; the latter's visual effect is always affected by lighting, such as when the lights are turned off, it becomes completely ineffective. Summary of the Invention

[0004] This disclosure provides a visual recognition method, device, electronic device, and medium for robot upper compartments, including: obtaining the pose of the robot compartment relative to the chassis and the position of the detection frame based on the pose of the upper compartment in the environment and the pose of the chassis in the environment; calculating the pose Tbc of the upper compartment in the chassis camera coordinate system; obtaining the set of vertices V of the detection frame based on the pose of the robot compartment relative to the chassis and the position of the detection frame; calculating the position [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system; traversing and comparing the positions [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system to obtain the coordinates of the highest point [umax,vmax] and the lowest point [umin,vmin]; labeling the data; and calculating the pose of the robot upper compartment relative to the chassis. Compared with the prior art, this disclosure obtains the pose of the robot upper compartment relative to the chassis by calculation, without the need for external QR code pasting, which is aesthetically pleasing and convenient; at the same time, it is not affected by the external environment, has a wide range of applications, and high recognition accuracy.

[0005] To achieve the above objectives, the present disclosure adopts the following technical solution:

[0006] The first aspect of this disclosure provides a robot-assisted warehouse visual recognition method, including:

[0007] Based on the poses of the upper compartment and the chassis in the environment, the pose of the robot body relative to the chassis and the position of the detection frame are obtained.

[0008] Calculate the pose Tbc of the enclosure in the chassis camera coordinate system:

[0009] Tbc = Twb -1 ×Twc

[0010] In the formula, Twb is the pose of the chassis camera at a certain moment, and Twc is the pose of the enclosure in the environment.

[0011] Based on the pose of the robot's housing relative to the chassis and the position of the detection frame, the set of vertices V of the detection frame is obtained: {v1, v2, ..., vn}.

[0012] Calculate the position [ui,vi] of each vertex set V of the detection box in the camera pixel coordinate system:

[0013] [ui,vi] = 1 / z × K × Twb × vi

[0014] In the formula, K is the intrinsic parameter of the camera, z is the depth value of the set of vertices V of the detection box in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment.

[0015] Iterate through and compare the positions [ui,vi] of each vertex set V of the detection box in the camera pixel coordinate system to obtain the coordinates of the highest point [umax,vmax] and the lowest point [umin,vmin].

[0016] Label the data.

[0017] Calculate the pose of the robot's upper compartment relative to the chassis.

[0018] Furthermore, the aforementioned robot-assisted visual recognition method includes data annotation, including:

[0019] The coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection frame are marked, and the orientation of the box is obtained in the pose Tbc of the chassis camera coordinate system.

[0020] or,

[0021] Label the coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection box, as well as the visual key points.

[0022] Furthermore, the aforementioned robot loading and unloading visual recognition method, which annotates visual key points, includes:

[0023] Let the visual key points of the upper warehouse be F: {f1, ..., fm}.

[0024] Calculate the position [uf, vf] of the upper visual key point F in the camera pixel coordinate system:

[0025] [uf, vf] = 1 / z × K × Twb × vi

[0026] In the formula, K is the intrinsic parameter of the camera, z is the depth value of the upper visual key point F in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment.

[0027] Furthermore, the aforementioned robot loading and unloading visual recognition method calculates the pose of the robot loading and unloading relative to the chassis, including:

[0028] A target detection network is selected to obtain the pose of the robot's upper compartment relative to the chassis.

[0029] or,

[0030] The pose of the robot's upper compartment relative to the chassis is calculated based on the set of visual key point coordinates.

[0031] Furthermore, in the aforementioned robot upper warehouse visual recognition method, the pose of the upper warehouse in the environment includes:

[0032] The pose of the upper compartment in the environment is obtained by a motion capture camera.

[0033] Furthermore, in the aforementioned robot upper warehouse visual recognition method, the pose of the chassis in the environment includes:

[0034] The chassis pose in the environment is obtained by using a motion capture camera or chassis positioning.

[0035] Furthermore, in the aforementioned robot upper warehouse visual recognition method, the pose of the chassis in the environment includes:

[0036] The chassis position in the environment is obtained through chassis positioning.

[0037] Furthermore, in the aforementioned robot-assisted visual recognition method, the target detection network includes:

[0038] The target detection network is YOLO, SSD, or Mobilenet.

[0039] Furthermore, in the aforementioned robot-assisted visual recognition method, the target detection network includes:

[0040] The target detection network is SSD.

[0041] Furthermore, the aforementioned robot loading and unloading visual recognition method, employing a target detection network, acquires the pose of the robot loading and unloading relative to the chassis, including:

[0042] A target detection network is selected, and the L2 loss of angle error is used to obtain the pose of the robot's upper compartment relative to the chassis.

[0043] A second aspect of this disclosure provides a robot upper warehouse visual recognition device, comprising:

[0044] The first acquisition unit is used to obtain the pose of the robot body relative to the chassis and the position of the detection frame based on the pose of the upper compartment in the environment and the pose of the chassis in the environment.

[0045] The first calculation unit is used to calculate the pose Tbc of the box body in the chassis camera coordinate system:

[0046] Tbc = Twb -1 ×Twc

[0047] In the formula, Twb is the pose of the chassis camera at a certain moment, and Twc is the pose of the enclosure in the environment.

[0048] The second acquisition unit is used to acquire the set of vertices V of the detection frame: {v1, v2, ..., vn} based on the pose of the robot's box relative to the chassis and the position of the detection frame.

[0049] The second calculation unit is used to calculate the position [ui,vi] of each vertex set V of the detection box in the camera pixel coordinate system:

[0050] [ui,vi] = 1 / z × K × Twb × vi

[0051] In the formula, K is the intrinsic parameter of the camera, z is the depth value of the set of vertices V of the detection box in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment.

[0052] The third acquisition unit is used to traverse and compare the positions [ui,vi] of each vertex set V of the detection box in the camera pixel coordinate system, and obtain the coordinates of the highest point [umax,vmax] and the coordinates of the lowest point [umin,vmin].

[0053] Labeling units are used to label data.

[0054] The third calculation unit is used to calculate the pose of the robot's upper compartment relative to the chassis.

[0055] Furthermore, the labeling unit of the aforementioned robot upper warehouse visual recognition device includes:

[0056] The first annotation module is used to annotate the coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection frame, as well as the orientation of the box in the pose Tbc of the chassis camera coordinate system.

[0057] or,

[0058] The second annotation module is used to annotate the coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection box, as well as the visual key points.

[0059] Furthermore, the third computing unit of the aforementioned robot upper warehouse visual recognition device includes:

[0060] The first calculation module is used to select a target detection network to obtain the pose of the robot's upper compartment relative to the chassis.

[0061] or,

[0062] The second calculation module is used to calculate the pose of the robot's upper compartment relative to the chassis based on the set of visual key point coordinates.

[0063] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, performs the steps of the method described above.

[0064] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0065] This disclosure provides a visual recognition method, device, electronic device, and medium for robot upper compartments, including: obtaining the pose of the robot compartment relative to the chassis and the position of the detection frame based on the pose of the upper compartment in the environment and the pose of the chassis in the environment; calculating the pose Tbc of the upper compartment in the chassis camera coordinate system; obtaining the set of vertices V of the detection frame based on the pose of the robot compartment relative to the chassis and the position of the detection frame; calculating the position [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system; traversing and comparing the positions [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system to obtain the coordinates of the highest point [umax,vmax] and the lowest point [umin,vmin]; labeling the data; and calculating the pose of the robot upper compartment relative to the chassis. Compared with the prior art, this disclosure obtains the pose of the robot upper compartment relative to the chassis by calculation, without the need for external QR code pasting, which is aesthetically pleasing and convenient; at the same time, it is not affected by the external environment, has a wide range of applications, and high recognition accuracy. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a schematic diagram of a robot loading and unloading visual recognition method according to an embodiment of the present disclosure;

[0068] Figure 2 This is a schematic diagram of another robot loading and unloading visual recognition method in this embodiment of the present disclosure;

[0069] Figure 3 This is a schematic diagram of the structural composition of a robot upper warehouse visual recognition device according to an embodiment of the present disclosure;

[0070] Figure 4 This is a schematic diagram of the composition structure of another robot upper warehouse visual recognition device in an embodiment of this disclosure;

[0071] Figure 5 This is a schematic diagram of the composition structure of a robot upper warehouse visual recognition electronic device according to an embodiment of this disclosure. Detailed Implementation

[0072] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0073] Unless otherwise defined, all technical and scientific terms used in this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs; the terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the terms “comprising” and “having” and any variations thereof in the description and claims of this disclosure and the foregoing drawings, but is intended to cover non-exclusive inclusion.

[0074] In the description of the embodiments of this disclosure, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary or secondary relationship of the indicated technical features. In the description of the embodiments of this disclosure, "a plurality of" means two or more, unless otherwise explicitly defined.

[0075] In the description of the embodiments of this disclosure, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0076] In the description of the embodiments of this disclosure, the term "multiple" refers to two or more (including two), similarly, "multiple groups" refers to two or more (including two groups), and "multiple pieces" refers to two or more (including two pieces).

[0077] In the description of the embodiments of this disclosure, the technical terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this disclosure and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this disclosure.

[0078] In the description of the embodiments of this disclosure, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of this disclosure according to the specific circumstances.

[0079] Example 1

[0080] This disclosure provides a robot-assisted visual recognition method for loading onto a warehouse, such as... Figure 1 As shown, it includes:

[0081] 101. Based on the pose of the upper compartment and the pose of the chassis in the environment, obtain the pose of the robot body relative to the chassis and the position of the detection frame.

[0082] Robot: refers to an intelligent robot that can automatically plan its route to a target location and move automatically along the route.

[0083] Upper warehouse: This is used to hold the goods that users select and purchase at the sales counter.

[0084] Chassis: A robot is a combination of multiple technologies and functions. Apart from software functions, other important parts are concentrated on the robot chassis, which carries multiple functions such as robot positioning, navigation, and obstacle avoidance.

[0085] 102. Calculate the pose Tbc of the box body in the chassis camera coordinate system:

[0086] Tbc = Twb -1 ×Twc

[0087] In the formula, Twb is the pose of the chassis camera at a certain moment, and Twc is the pose of the enclosure in the environment.

[0088] 103. Based on the pose of the robot's housing relative to the chassis and the position of the detection frame, obtain the set of vertices V of the detection frame: {v1, v2, ..., vn}.

[0089] 104. Calculate the position [ui,vi] of each vertex set V of the detection box in the camera pixel coordinate system:

[0090] [ui,vi] = 1 / z × K × Twb × vi

[0091] In the formula, K is the intrinsic parameter of the camera, z is the depth value of the set of vertices V of the detection box in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment.

[0092] 105. Traverse and compare the positions [ui,vi] of each vertex set V of the detection box in the camera pixel coordinate system, and obtain the coordinates of the highest point [umax,vmax] and the lowest point [umin,vmin].

[0093] 106. Label the data.

[0094] 107. Calculate the pose of the robot's upper compartment relative to the chassis.

[0095] This disclosure provides a visual recognition method for robot upper compartments, comprising: obtaining the pose of the robot body relative to the chassis and the position of the detection frame based on the pose of the upper compartment in the environment and the pose of the chassis in the environment; calculating the pose Tbc of the upper compartment in the chassis camera coordinate system; obtaining the set of vertices V of the detection frame based on the pose of the robot body relative to the chassis and the position of the detection frame; calculating the position [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system; traversing and comparing the positions [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system to obtain the coordinates of the highest point [umax,vmax] and the lowest point [umin,vmin]; labeling the data; and calculating the pose of the robot upper compartment relative to the chassis. Compared with the prior art, this disclosure obtains the pose of the robot upper compartment relative to the chassis by calculation, without the need for external QR code pasting, which is aesthetically pleasing and convenient; at the same time, it is not affected by the external environment, has a wide range of applications, and high recognition accuracy.

[0096] Example 2

[0097] This disclosure provides a robot-assisted visual recognition method for loading onto a warehouse, such as... Figure 2 As shown, it includes:

[0098] 201. Based on the pose of the upper compartment and the chassis in the environment, obtain the pose of the robot body relative to the chassis and the position of the detection frame.

[0099] The pose of the upper compartment in the environment can be obtained through a motion capture camera, and the pose of the chassis in the environment can be obtained through a motion capture camera or chassis positioning. In this embodiment, the pose of the chassis in the environment is obtained through chassis positioning. In other embodiments, the implementer can determine the method for obtaining the pose of the upper compartment and the chassis in the environment according to their own needs.

[0100] 202. Calculate the pose Tbc of the box body in the chassis camera coordinate system:

[0101] Tbc = Twb -1 ×Twc

[0102] In the formula, Twb is the pose of the chassis camera at a certain moment, and Twc is the pose of the enclosure in the environment.

[0103] 203. Based on the pose of the robot's housing relative to the chassis and the position of the detection frame, obtain the set of vertices V of the detection frame: {v1, v2, ..., vn}.

[0104] 204. Calculate the position [ui,vi] of each vertex set V of the detection box in the camera pixel coordinate system:

[0105] [ui,vi] = 1 / z × K × Twb × vi

[0106] In the formula, K is the intrinsic parameter of the camera, z is the depth value of the set of vertices V of the detection box in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment.

[0107] 205. Traverse and compare the positions [ui,vi] of each vertex set V of the detection box in the camera pixel coordinate system, and obtain the coordinates of the highest point [umax,vmax] and the lowest point [umin,vmin].

[0108] 206. Label the data.

[0109] The coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection frame are marked, and the orientation of the box is obtained in the pose Tbc of the chassis camera coordinate system.

[0110] or,

[0111] Label the coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection box, as well as the visual key points.

[0112] 207. Calculate the pose of the robot's upper compartment relative to the chassis.

[0113] A target detection network is selected, and the L2 loss of angle error is used to obtain the pose of the robot's upper compartment relative to the chassis.

[0114] The target detection network can be YOLO, SSD, or MobileNet; in this embodiment, SSD is selected as the target detection network. In other embodiments, implementers may choose other target detection networks, and this embodiment does not impose any restrictions on this.

[0115] In this embodiment, a target detection network is selected, and the pose of the robot's upper compartment relative to the chassis is obtained by combining the L2 loss of the angle error. The position of the upper compartment relative to the chassis is set as (x, y). It is generally assumed that the bottom edge of the detection frame is on the ground, and the midpoint of the bottom edge is approximately considered as the position of the upper compartment. After the projection equation limits the upper compartment coordinate z to 0, x and y can be obtained.

[0116] or,

[0117] The pose of the robot's upper compartment relative to the chassis is calculated based on the set of visual key point coordinates.

[0118] Let the visual key points of the upper warehouse be F: {f1, ..., fm};

[0119] Calculate the position [uf, vf] of the upper visual key point F in the camera pixel coordinate system:

[0120] [uf, vf] = 1 / z × K × Twb × vi

[0121] In the formula, K is the intrinsic parameter of the camera, z is the depth value of the upper visual key point F in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment.

[0122] The calculation of the robot's upper compartment's pose relative to the chassis involves three steps: The first step is standard object detection, which can reuse existing object detection networks and results from the chassis; the second step is visual keypoint detection, similar to HRNet, outputting the predicted position [uf, vf] of the visual keypoint F in the image coordinate system; the third step calculates the upper compartment's pose relative to the chassis using the set of visual keypoint coordinates, employing a projection formula:

[0123] [ui,vi] = 1 / z × K × Tbc × fi,

[0124] In the formula, K is the camera intrinsic parameter matrix, fi is the position of the i-th feature point in the upper vehicle coordinate system. According to the known structure, ui and vi are the predicted positions of the visual key points. Tbc is calculated. Since the upper vehicle and the chassis are on the same plane, there are 6 unknowns in Tbc. The above projection formula can provide two constraints, so more than 3 feature points can form an overdetermined equation, and the pose of the upper vehicle relative to the chassis can be solved.

[0125] This disclosure provides a visual recognition method for robot upper compartments, comprising: obtaining the pose of the robot body relative to the chassis and the position of the detection frame based on the pose of the upper compartment in the environment and the pose of the chassis in the environment; calculating the pose Tbc of the upper compartment in the chassis camera coordinate system; obtaining the set of vertices V of the detection frame based on the pose of the robot body relative to the chassis and the position of the detection frame; calculating the position [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system; traversing and comparing the positions [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system to obtain the coordinates of the highest point [umax,vmax] and the lowest point [umin,vmin]; labeling the data; and calculating the pose of the robot upper compartment relative to the chassis. Compared with the prior art, this disclosure obtains the pose of the robot upper compartment relative to the chassis by calculation, without the need for external QR code pasting, which is aesthetically pleasing and convenient; at the same time, it is not affected by the external environment, has a wide range of applications, and high recognition accuracy.

[0126] Example 3

[0127] This disclosure provides a robot-mounted warehouse visual recognition device, such as... Figure 3 As shown, it includes:

[0128] The first acquisition unit 31 is used to obtain the pose of the robot body relative to the chassis and the position of the detection frame based on the pose of the upper compartment in the environment and the pose of the chassis in the environment.

[0129] The first calculation unit 32 is used to calculate the pose Tbc of the box body in the chassis camera coordinate system:

[0130] Tbc = Twb -1 ×Twc

[0131] In the formula, Twb is the pose of the chassis camera at a certain moment, and Twc is the pose of the enclosure in the environment.

[0132] The second acquisition unit 33 is used to acquire the set of vertices V of the detection frame: {v1, v2, ..., vn} based on the pose of the robot's box relative to the chassis and the position of the detection frame.

[0133] The second calculation unit 34 is used to calculate the position [ui,vi] of each vertex set V of the detection box in the camera pixel coordinate system:

[0134] [ui,vi] = 1 / z × K × Twb × vi

[0135] In the formula, K is the intrinsic parameter of the camera, z is the depth value of the set of vertices V of the detection box in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment.

[0136] The third acquisition unit 35 is used to traverse and compare the positions [ui,vi] of each vertex set V of the detection box in the camera pixel coordinate system, and obtain the coordinates of the highest point [umax,vmax] and the coordinates of the lowest point [umin,vmin].

[0137] Labeling unit 36 ​​is used to label the data.

[0138] The third calculation unit 37 is used to calculate the pose of the robot's upper compartment relative to the chassis.

[0139] It should be noted that the detailed description of each component in this embodiment can be found in other embodiments, and will not be repeated here.

[0140] This disclosure provides a robot upper compartment visual recognition device, comprising: a first obtaining unit that obtains the robot body's pose relative to the chassis and the position of the detection frame based on the pose of the upper compartment in the environment and the pose of the chassis in the environment; a first calculating unit that calculates the pose Tbc of the upper compartment in the chassis camera coordinate system; a second obtaining unit that obtains the set of vertices V of the detection frame based on the robot body's pose relative to the chassis and the position of the detection frame; a second calculating unit that calculates the position [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system; a third obtaining unit that iterates and compares the positions [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system to obtain the coordinates of the highest point [umax,vmax] and the lowest point [umin,vmin]; a labeling unit that labels the data; and a third calculating unit that calculates the pose of the robot upper compartment relative to the chassis. Compared with the prior art, this disclosure obtains the pose of the robot upper compartment relative to the chassis through calculation, eliminating the need for external QR code pasting, which is aesthetically pleasing and convenient; at the same time, it is not affected by the external environment, has a wide range of applications, and high recognition accuracy.

[0141] Example 4

[0142] This disclosure provides a robot-mounted warehouse visual recognition device, such as... Figure 4 As shown, it includes:

[0143] The first acquisition unit 41 is used to obtain the pose of the robot body relative to the chassis and the position of the detection frame based on the pose of the upper compartment in the environment and the pose of the chassis in the environment.

[0144] The first calculation unit 42 is used to calculate the pose Tbc of the box body in the chassis camera coordinate system:

[0145] Tbc = Twb -1 ×Twc

[0146] In the formula, Twb is the pose of the chassis camera at a certain moment, and Twc is the pose of the enclosure in the environment.

[0147] The second acquisition unit 43 is used to acquire the set of vertices V of the detection frame: {v1, v2, ..., vn} based on the pose of the robot's box relative to the chassis and the position of the detection frame.

[0148] The second calculation unit 44 is used to calculate the position [ui,vi] of each vertex set V of the detection box in the camera pixel coordinate system:

[0149] [ui,vi] = 1 / z × K × Twb × vi

[0150] In the formula, K is the intrinsic parameter of the camera, z is the depth value of the set of vertices V of the detection box in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment.

[0151] The third acquisition unit 45 is used to traverse and compare the positions [ui,vi] of each vertex set V of the detection box in the camera pixel coordinate system, and obtain the coordinates of the highest point [umax,vmax] and the coordinates of the lowest point [umin,vmin].

[0152] Labeling unit 46 is used to label the data.

[0153] The first annotation module 461 is used to annotate the coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection frame, as well as the orientation of the box in the pose Tbc of the chassis camera coordinate system.

[0154] or,

[0155] The second annotation module 462 is used to annotate the coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection box, as well as the visual key points.

[0156] The third calculation unit 47 is used to calculate the pose of the robot's upper compartment relative to the chassis.

[0157] The first calculation module 471 is used to select a target detection network to obtain the pose of the robot's upper compartment relative to the chassis.

[0158] or,

[0159] The second calculation module 472 is used to calculate the pose of the robot's upper compartment relative to the chassis based on the set of visual key point coordinates.

[0160] It should be noted that the detailed description of each component in this embodiment can be found in other embodiments, and will not be repeated here.

[0161] This disclosure provides a robot upper compartment visual recognition device, comprising: a first obtaining unit that obtains the robot body's pose relative to the chassis and the position of the detection frame based on the pose of the upper compartment in the environment and the pose of the chassis in the environment; a first calculating unit that calculates the pose Tbc of the upper compartment in the chassis camera coordinate system; a second obtaining unit that obtains the set of vertices V of the detection frame based on the robot body's pose relative to the chassis and the position of the detection frame; a second calculating unit that calculates the position [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system; a third obtaining unit that iterates and compares the positions [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system to obtain the coordinates of the highest point [umax,vmax] and the lowest point [umin,vmin]; a labeling unit that labels the data; and a third calculating unit that calculates the pose of the robot upper compartment relative to the chassis. Compared with the prior art, this disclosure obtains the pose of the robot upper compartment relative to the chassis through calculation, eliminating the need for external QR code pasting, which is aesthetically pleasing and convenient; at the same time, it is not affected by the external environment, has a wide range of applications, and high recognition accuracy.

[0162] Example 5

[0163] This disclosure provides a robot-mounted warehouse visual recognition electronic device, such as... Figure 5 As shown, it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described steps of robot visual recognition on the warehouse.

[0164] The processing device (e.g., central processing unit, graphics processing unit, etc.) 51 can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 52 or a program loaded from storage device 58 into random access memory (RAM) 53. The RAM 53 also stores various programs and data required for the operation of the electronic device. The processing device 51, ROM 52, and RAM 53 are interconnected via bus 54. An input / output (I / O) interface 55 is also connected to bus 54.

[0165] Typically, the following devices can be connected to I / O interface 55: input devices 56 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 57 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 58 including, for example, magnetic tapes, hard disks, etc.; and communication devices 59. Communication device 59 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively. Figure 5 Each box shown can represent a device or multiple devices as needed.

[0166] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 59, or installed from a storage device 58, or installed from a ROM 52. When the computer program is executed by the processing device 51, it performs the functions defined in the methods of some embodiments of this disclosure.

[0167] This disclosure provides a robot upper compartment visual recognition electronic device, comprising: obtaining the pose of the robot body relative to the chassis and the position of the detection frame based on the pose of the upper compartment in the environment and the pose of the chassis in the environment; calculating the pose Tbc of the upper compartment in the chassis camera coordinate system; obtaining the set of vertices V of the detection frame based on the pose of the robot body relative to the chassis and the position of the detection frame; calculating the position [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system; traversing and comparing the positions [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system to obtain the coordinates of the highest point [umax,vmax] and the lowest point [umin,vmin]; labeling the data; and calculating the pose of the robot upper compartment relative to the chassis. Compared with the prior art, this disclosure obtains the pose of the robot upper compartment relative to the chassis by calculation, without the need for external QR code pasting, which is aesthetically pleasing and convenient; at the same time, it is not affected by the external environment, has a wide range of applications, and high recognition accuracy.

[0168] Example 6

[0169] This disclosure provides a computer-readable storage medium for robot loading and unloading visual recognition. The computer-readable storage medium stores a computer program, which, when executed by one or more processors, implements the above-described robot loading and unloading visual recognition method.

[0170] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0171] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0172] The aforementioned computer-readable medium may be included in the aforementioned device; or it may exist independently without being assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: obtain the pose of the robot's housing relative to the chassis and the position of the detection frame based on the pose of the upper compartment in the environment and the pose of the chassis in the environment; calculate the pose Tbc of the housing in the chassis camera coordinate system; obtain the set of vertices V of the detection frame based on the pose of the robot's housing relative to the chassis and the position of the detection frame; calculate the position [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system; traverse and compare the positions [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system to obtain the coordinates of the highest point [umax,vmax] and the lowest point [umin,vmin]; label the data; and calculate the pose of the robot's upper compartment relative to the chassis.

[0173] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0174] This disclosure provides a computer-readable storage medium for visual recognition of a robot upper compartment, comprising: obtaining the pose of the robot body relative to the chassis and the position of the detection frame based on the pose of the upper compartment and the chassis in the environment; calculating the pose Tbc of the upper compartment in the chassis camera coordinate system; obtaining the set of vertices V of the detection frame based on the pose of the robot body relative to the chassis and the position of the detection frame; calculating the position [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system; traversing and comparing the positions [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system to obtain the coordinates of the highest point [umax,vmax] and the lowest point [umin,vmin]; labeling the data; and calculating the pose of the robot upper compartment relative to the chassis. Compared with the prior art, this disclosure obtains the pose of the robot upper compartment relative to the chassis by calculation, without the need for external QR code pasting, which is aesthetically pleasing and convenient; at the same time, it is not affected by the external environment, has a wide range of applications, and high recognition accuracy.

[0175] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0176] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a receiving unit, an information acquisition unit, a target determination unit, and a delivery unit. The names of these units do not necessarily limit the specific unit itself.

[0177] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0178] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention. In particular, as long as there is no structural conflict, the various technical features mentioned in the embodiments can be combined in any way. The present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A visual recognition method for robot loading into a warehouse, characterized in that, include: Based on the pose of the upper compartment and the pose of the chassis in the environment, the pose of the robot body relative to the chassis and the position of the detection frame are obtained. Calculate the pose Tbc of the enclosure in the chassis camera coordinate system: Tbc = Twb -1 ×Twc In the formula, Twb is the pose of the chassis camera at a certain moment, and Twc is the pose of the enclosure in the environment; Based on the pose of the robot's housing relative to the chassis and the position of the detection frame, obtain the set of vertices V of the detection frame: {v1, v2, ..., vn}; Calculate the position [ui, vi] of each vertex set V of the detection box in the camera pixel coordinate system: [ui, vi] = 1 / z×K×Twb×vi In the formula, K is the intrinsic parameter of the camera, z is the depth value of the set of vertices V of the detection box in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment; Iterate through and compare the positions [ui, vi] of each vertex set V of the detection box in the camera pixel coordinate system to obtain the coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin]. Label the data; Calculate the pose of the robot's upper compartment relative to the chassis; Data annotation includes: The coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection frame are marked, and the orientation of the box is obtained in the pose Tbc of the chassis camera coordinate system. or, Label the coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection box, as well as the visual key points; Mark visual key points, including: Let the visual key points of the upper warehouse be F: {f1, ..., fm}; Calculate the position [uf, vf] of the upper visual key point F in the camera pixel coordinate system: [uf, vf] = 1 / z × K × Twb × vi In the formula, K is the intrinsic parameter of the camera, z is the depth value of the upper visual key point F in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment. Calculating the pose of the robot's upper compartment relative to the chassis includes: A target detection network is selected to obtain the pose of the robot's upper compartment relative to the chassis; or, Based on the set of visual key point coordinates, calculate the pose of the robot's upper compartment relative to the chassis; Using a target detection network, the pose of the robot's upper compartment relative to the chassis is obtained, including: A target detection network is selected, and the L2 loss of angle error is used to obtain the pose of the robot's upper compartment relative to the chassis.

2. The robot loading and unloading visual recognition method according to claim 1, characterized in that, The pose of the upper compartment in the environment includes: The pose of the upper compartment in the environment is obtained by a motion capture camera.

3. The robot loading and unloading visual recognition method according to claim 1, characterized in that, The chassis's position and orientation in the environment include: The chassis pose in the environment is obtained by using a motion capture camera or chassis positioning.

4. The robot loading and unloading visual recognition method according to claim 1, characterized in that, The chassis's position and orientation in the environment include: The chassis position in the environment is obtained through chassis positioning.

5. The robot loading and unloading visual recognition method according to claim 1, characterized in that, The target detection network includes: The target detection network is YOLO, SSD, or Mobilenet.

6. The robot loading and unloading visual recognition method according to claim 5, characterized in that, The target detection network includes: The target detection network is SSD.

7. A robot loading and unloading visual recognition device, performing the robot loading and unloading visual recognition method as described in any one of claims 1-6, characterized in that, include: The first acquisition unit is used to obtain the pose of the robot body relative to the chassis and the position of the detection frame based on the pose of the upper compartment in the environment and the pose of the chassis in the environment. The first calculation unit is used to calculate the pose Tbc of the box body in the chassis camera coordinate system: Tbc = Twb -1 ×Twc In the formula, Twb is the pose of the chassis camera at a certain moment, and Twc is the pose of the enclosure in the environment; The second acquisition unit is used to acquire the set of vertices V of the detection frame: {v1, v2, ..., vn} based on the pose of the robot's box relative to the chassis and the position of the detection frame; The second calculation unit is used to calculate the position [ui,vi] of each vertex set V of the detection box in the camera pixel coordinate system: [ui, vi] = 1 / z×K×Twb×vi In the formula, K is the intrinsic parameter of the camera, z is the depth value of the set of vertices V of the detection box in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment; The third acquisition unit is used to traverse and compare the positions [ui, vi] of each vertex set V of the detection box in the camera pixel coordinate system, and obtain the coordinates of the highest point [umax, vmax] and the coordinates of the lowest point [umin, vmin]. Labeling units are used to label data; The third calculation unit is used to calculate the pose of the robot's upper compartment relative to the chassis; The annotation unit includes: The first annotation module is used to annotate the coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection box, as well as the orientation of the box in the pose Tbc of the chassis camera coordinate system. or, The second annotation module is used to annotate the coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection box, as well as the visual key points; Mark visual key points, including: Let the visual key points of the upper warehouse be F: {f1, ..., fm}; Calculate the position [uf, vf] of the upper visual key point F in the camera pixel coordinate system: [uf, vf] = 1 / z × K × Twb × vi In the formula, K is the intrinsic parameter of the camera, z is the depth value of the upper visual key point F in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment. The third calculation unit includes: The first calculation module is used to select a target detection network to obtain the pose of the robot's upper compartment relative to the chassis; or, The second calculation module is used to calculate the pose of the robot's upper compartment relative to the chassis based on the set of visual key point coordinates. Based on the set of visual key point coordinates, calculate the pose of the robot's upper compartment relative to the chassis; Using a target detection network, the pose of the robot's upper compartment relative to the chassis is obtained, including: A target detection network is selected, and the L2 loss of angle error is used to obtain the pose of the robot's upper compartment relative to the chassis.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images