Visual identification method and device for warehouse loading of robot, electronic equipment and medium

Abstract

The invention relates to the technical field of intelligent robots, and provides a robot warehouse loading visual identification method and device, electronic equipment and a medium. Comprising the steps that the pose of a robot box relative to a chassis and the position of a detection frame are obtained according to the pose of an upper bin in the environment and the pose of the chassis in the environment; calculating the pose Tbc of the box body in a chassis camera coordinate system; according to the pose of the robot box relative to the chassis and the position of the detection frame, a vertex set V of the detection frame is obtained; calculating the position [ui, vi] of each vertex set V of the detection frame in a camera pixel coordinate system; traversing and comparing the position [ui, vi] of each vertex set V of the detection frame in a camera pixel coordinate system, and obtaining a highest point coordinate [umax, vmax] and a lowest point coordinate [umin, vmin]; marking the data; and calculating the pose of the robot upper cabin relative to the chassis. The method does not need to externally paste a two-dimensional code, and is beautiful and convenient; meanwhile, the method is not affected by external environment, and is wide in application range and high in recognition precision.

Description

Technical Field

[0001] The present disclosure relates to the technical field of intelligent robots, and in particular to a robot upper warehouse visual recognition method, device, electronic equipment and medium. Background Art

[0002] With the rapid development of science and technology, robot technology is also changing with each passing day. Among them, robots with a chassis and upper compartment separated from each other are very popular in the market. For robots with a chassis and upper compartment separated from each other, obtaining the position of the upper compartment relative to the chassis has become an important research topic.

[0003] The existing technology generally adopts the method of pasting a QR code on the upper warehouse or the deep learning method. The former has too great an impact on the appearance and is not beautiful; the latter's vision is always affected by light, for example, it will be completely ineffective when the lights are turned off. Summary of the invention

[0004] The present invention provides a robot upper warehouse visual recognition method, device, electronic device and medium, including: obtaining the posture and detection frame position of the robot box relative to the chassis according to the posture of the upper warehouse in the environment and the posture of the chassis in the environment; calculating the posture Tbc of the box in the chassis camera coordinate system; obtaining the vertex set V of the detection frame according to the posture of the robot box relative to the chassis and the detection frame position; calculating the position [ui,vi] of the vertex set V of the detection frame in the camera pixel coordinate system; traversing and comparing the position [ui,vi] of the vertex set V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinates [umax,vmax] and the lowest point coordinates [umin,vmin]; marking the data; calculating the posture of the robot upper warehouse relative to the chassis. Compared with the prior art, the present invention obtains the posture of the robot upper warehouse relative to the chassis by calculation, without the need to externally paste a QR code, which is beautiful and convenient; at the same time, it is not affected by the external environment, has a wide range of applications and high recognition accuracy.

[0005] In order to achieve the above objectives, the present invention adopts the following technical solutions:

[0006] A first aspect of the present disclosure provides a robot warehouse visual recognition method, comprising:

[0007] According to the posture of the upper warehouse and the posture of the chassis in the environment, the posture of the robot box relative to the chassis and the position of the detection frame are obtained.

[0008] Calculate the position Tbc of the box in the chassis camera coordinate system:

[0009] Tbc=Twb -1 ×Twc

[0010] Wherein, Twb is the position of the chassis camera at a certain moment, and Twc is the position of the box in the environment.

[0011] According to the posture of the box of the robot relative to the chassis and the position of the detection frame, a set of vertices V of the detection frame is obtained: {v1, v2, ....vn}.

[0012] Calculate the position [ui,vi] of each vertex set V of the detection frame in the camera pixel coordinate system:

[0013] [ui,vi]=1 / z×K×Twb×vi

[0014] Wherein, K is the intrinsic parameter of the camera, z is the depth value of the vertex set V of the detection frame in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment.

[0015] The positions [ui,vi] of the vertex set V of the detection frame in the camera pixel coordinate system are traversed and compared to obtain the highest point coordinates [umax,vmax] and the lowest point coordinates [umin,vmin].

[0016] Label the data.

[0017] The position of the robot upper warehouse relative to the chassis is calculated.

[0018] Furthermore, the robot warehouse visual recognition method labels the data, including:

[0019] The coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection frame are marked, and the orientation of the box is obtained in the posture Tbc of the chassis camera coordinate system.

[0020] or,

[0021] The highest point coordinates [umax, vmax] and the lowest point coordinates [umin, vmin] of the detection box and the visual key points are marked.

[0022] Furthermore, the robot upper warehouse visual recognition method, marking visual key points, includes:

[0023] Assume the visual key points F of the upper warehouse: {f1,...fm}.

[0024] Calculate the position [uf, vf] of the upper warehouse visual key point F in the camera pixel coordinate system:

[0025] [uf,vf]=1 / z×K×Twb×vi

[0026] Wherein, K is the camera internal parameter, z is the depth value of the upper warehouse visual key point F in the camera coordinate system, and Twb is the position of the chassis camera at a certain moment.

[0027] Furthermore, the robot upper warehouse visual recognition method calculates the position and posture of the robot upper warehouse relative to the chassis, including:

[0028] A target detection network is used to obtain the position of the robot's upper warehouse relative to the chassis.

[0029] or,

[0030] The position and posture of the robot upper warehouse relative to the chassis is calculated based on the visual key point coordinate set.

[0031] Furthermore, in the robot upper warehouse visual recognition method, the position of the upper warehouse in the environment includes:

[0032] The position and posture of the upper warehouse in the environment is obtained through a motion capture camera.

[0033] Furthermore, in the robot upper warehouse visual recognition method, the position and posture of the chassis in the environment includes:

[0034] The position and posture of the chassis in the environment is obtained through a motion capture camera or chassis positioning.

[0035] Furthermore, in the robot upper warehouse visual recognition method, the position and posture of the chassis in the environment includes:

[0036] The position and posture of the chassis in the environment are obtained by chassis positioning.

[0037] Furthermore, in the robot warehouse visual recognition method, the target detection network includes:

[0038] The target detection network is yolo or SSD or mobilenet.

[0039] Furthermore, in the robot warehouse visual recognition method, the target detection network includes:

[0040] The target detection network is SSD.

[0041] Furthermore, the robot upper warehouse visual recognition method uses a target detection network to obtain the position of the robot upper warehouse relative to the chassis, including:

[0042] The target detection network is selected and combined with the L2 loss of the angle error to obtain the position of the robot's upper warehouse relative to the chassis.

[0043] A second aspect of the present disclosure provides a robot upper warehouse visual recognition device, comprising:

[0044] The first obtaining unit is used to obtain the posture of the robot box relative to the chassis and the position of the detection frame according to the posture of the upper warehouse in the environment and the posture of the chassis in the environment.

[0045] The first calculation unit is used to calculate the position and posture Tbc of the box in the chassis camera coordinate system:

[0046] Tbc=Twb -1 ×Twc

[0047] Wherein, Twb is the position of the chassis camera at a certain moment, and Twc is the position of the box in the environment.

[0048] The second acquisition unit is used to acquire a set V of vertices of the detection frame: {v1, v2, ....vn} according to the posture of the robot box relative to the chassis and the position of the detection frame.

[0049] The second calculation unit is used to calculate the position [ui,vi] of each vertex set V of the detection frame in the camera pixel coordinate system:

[0050] [ui,vi]=1 / z×K×Twb×vi

[0051] Wherein, K is the intrinsic parameter of the camera, z is the depth value of the vertex set V of the detection frame in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment.

[0052] The third acquisition unit is used to traverse and compare the positions [ui, vi] of each vertex set V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinates [umax, vmax] and the lowest point coordinates [umin, vmin].

[0053] The labeling unit is used to label the data.

[0054] The third calculation unit is used to calculate the position of the robot upper warehouse relative to the chassis.

[0055] Furthermore, the robot upper warehouse visual recognition device, the labeling unit includes:

[0056] The first marking module is used to mark the coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection frame and the orientation of the box in the posture Tbc of the chassis camera coordinate system.

[0057] or,

[0058] The second marking module is used to mark the highest point coordinates [umax, vmax] and the lowest point coordinates [umin, vmin] of the detection frame and the visual key points.

[0059] Furthermore, in the robot upper warehouse visual recognition device, the third computing unit includes:

[0060] The first computing module is used to select a target detection network to obtain the position of the robot upper warehouse relative to the chassis.

[0061] or,

[0062] The second calculation module is used to calculate the position and posture of the robot upper warehouse relative to the chassis according to the visual key point coordinate set.

[0063] A third aspect of the present disclosure provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0064] A fourth aspect of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0065] The present invention provides a robot upper warehouse visual recognition method, device, electronic device and medium, including: obtaining the posture and detection frame position of the robot box relative to the chassis according to the posture of the upper warehouse in the environment and the posture of the chassis in the environment; calculating the posture Tbc of the box in the chassis camera coordinate system; obtaining the vertex set V of the detection frame according to the posture of the robot box relative to the chassis and the detection frame position; calculating the position [ui,vi] of the vertex set V of the detection frame in the camera pixel coordinate system; traversing and comparing the position [ui,vi] of the vertex set V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinates [umax,vmax] and the lowest point coordinates [umin,vmin]; marking the data; calculating the posture of the robot upper warehouse relative to the chassis. Compared with the prior art, the present invention obtains the posture of the robot upper warehouse relative to the chassis by calculation, without the need to externally paste a QR code, which is beautiful and convenient; at the same time, it is not affected by the external environment, has a wide range of applications and high recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1 This is a schematic diagram of a robot warehouse visual recognition method flow in an embodiment of the present disclosure;

[0068] Figure 2 This is a flow chart of another robot warehouse visual recognition method in an embodiment of the present disclosure;

[0069] Figure 3 This is a schematic diagram of the structure of a robot upper warehouse visual recognition device in an embodiment of the present disclosure;

[0070] Figure 4 It is a schematic diagram of the structure of another robot upper warehouse visual recognition device in an embodiment of the present disclosure;

[0071] Figure 5 It is a schematic diagram of the structure of a robot upper warehouse visual recognition electronic equipment in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0073] Unless otherwise defined, all technical and scientific terms used in the present disclosure have the same meanings as those commonly understood by technicians in the field of the present disclosure; the terms used in the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the terms "including" and "having" and any variations thereof in the specification and claims of the present disclosure and the above-mentioned drawings, and are intended to cover non-exclusive inclusions.

[0074] In the description of the embodiments of the present disclosure, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present disclosure, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0075] In the description of the embodiments of the present disclosure, the term "and / or" is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0076] In the description of the embodiments of the present disclosure, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0077] In the description of the embodiments of the present disclosure, the orientations or positional relationships indicated by technical terms such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", and "circumferential" are based on the orientations or positional relationships shown in the accompanying drawings and are only for the convenience of describing the embodiments of the present disclosure and simplifying the description. They do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the embodiments of the present disclosure.

[0078] In the description of the embodiments of the present disclosure, unless otherwise clearly specified and limited, technical terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present disclosure can be understood according to the specific circumstances.

[0079] Example 1

[0080] The present disclosure provides a robot upper warehouse visual recognition method, such as Figure 1 As shown, including:

[0081] 101. According to the posture of the upper warehouse in the environment and the posture of the chassis in the environment, the posture of the robot box relative to the chassis and the position of the detection frame are obtained.

[0082] Robot: refers to an intelligent robot that can automatically plan a route to the target location and move automatically along the route.

[0083] Upper warehouse: It is used to store the goods that users purchase at the merchandise vending counter.

[0084] Chassis: A robot is a combination of multiple technologies and functions. In addition to software functions, other important parts are concentrated on the robot chassis hardware, which carries multiple functions of the robot such as positioning, navigation and obstacle avoidance.

[0085] 102. Calculate the position and posture Tbc of the box in the chassis camera coordinate system:

[0086] Tbc=Twb -1 ×Twc

[0087] Wherein, Twb is the position of the chassis camera at a certain moment, and Twc is the position of the box in the environment.

[0088] 103. According to the posture of the box of the robot relative to the chassis and the position of the detection frame, obtain a set of vertices V of the detection frame: {v1, v2, ....vn}.

[0089] 104. Calculate the position [ui,vi] of each vertex set V of the detection frame in the camera pixel coordinate system:

[0090] [ui,vi]=1 / z×K×Twb×vi

[0091] Wherein, K is the intrinsic parameter of the camera, z is the depth value of the vertex set V of the detection frame in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment.

[0092] 105. Traverse and compare the positions [ui, vi] of each vertex set V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinates [umax, vmax] and the lowest point coordinates [umin, vmin].

[0093] 106. Label the data.

[0094] 107. Calculate the position of the robot upper warehouse relative to the chassis.

[0095] The present disclosure provides a robot upper warehouse visual recognition method, comprising: obtaining the posture and detection frame position of the robot box relative to the chassis according to the posture of the upper warehouse in the environment and the posture of the chassis in the environment; calculating the posture Tbc of the box in the chassis camera coordinate system; obtaining the vertex set V of the detection frame according to the posture of the robot box relative to the chassis and the detection frame position; calculating the position [ui, vi] of the vertex set V of the detection frame in the camera pixel coordinate system; traversing and comparing the position [ui, vi] of the vertex set V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinates [umax, vmax] and the lowest point coordinates [umin, vmin]; marking the data; calculating the posture of the robot upper warehouse relative to the chassis. Compared with the prior art, the present disclosure obtains the posture of the robot upper warehouse relative to the chassis by calculation, without the need to externally paste a QR code, which is beautiful and convenient; at the same time, it is not affected by the external environment, has a wide range of applications, and has high recognition accuracy.

[0096] Example 2

[0097] The present disclosure provides a robot upper warehouse visual recognition method, such as Figure 2 As shown, including:

[0098] 201. According to the posture of the upper warehouse in the environment and the posture of the chassis in the environment, the posture of the robot box relative to the chassis and the position of the detection frame are obtained.

[0099] The position and posture of the upper warehouse in the environment can be obtained by a motion capture camera, and the position and posture of the chassis in the environment can be obtained by a motion capture camera or chassis positioning. In this embodiment, the position and posture of the chassis in the environment is obtained by chassis positioning. In other embodiments, the implementer can determine the way to obtain the position and posture of the upper warehouse in the environment and the position and posture of the chassis in the environment according to their own needs.

[0100] 202. Calculate the position and posture Tbc of the box in the chassis camera coordinate system:

[0101] Tbc=Twb -1 ×Twc

[0102] Wherein, Twb is the position of the chassis camera at a certain moment, and Twc is the position of the box in the environment.

[0103] 203. According to the posture of the box of the robot relative to the chassis and the position of the detection frame, obtain a set of vertices V of the detection frame: {v1, v2, ....vn}.

[0104] 204. Calculate the position [ui,vi] of each vertex set V of the detection frame in the camera pixel coordinate system:

[0105] [ui,vi]=1 / z×K×Twb×vi

[0106] Wherein, K is the intrinsic parameter of the camera, z is the depth value of the vertex set V of the detection frame in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment.

[0107] 205. Traverse and compare the positions [ui, vi] of each vertex set V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinates [umax, vmax] and the lowest point coordinates [umin, vmin].

[0108] 206. Label the data.

[0109] The coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection frame are marked, and the orientation of the box is obtained in the posture Tbc of the chassis camera coordinate system.

[0110] or,

[0111] The highest point coordinates [umax, vmax] and the lowest point coordinates [umin, vmin] of the detection box and the visual key points are marked.

[0112] 207. Calculate the position of the robot upper warehouse relative to the chassis.

[0113] The target detection network is selected and combined with the L2 loss of the angle error to obtain the position of the robot's upper warehouse relative to the chassis.

[0114] The target detection network is yolo or SSD or mobilenet, and SSD is selected as the target detection network in this embodiment. In other embodiments, the implementer may also select other target detection networks, which is not limited in this embodiment.

[0115] This embodiment uses a target detection network, combined with the L2 loss of the angle error, to obtain the position of the robot's upper warehouse relative to the chassis. The position of the upper warehouse relative to the chassis is set to (x, y). It is generally assumed that the lower base of the detection frame is on the ground, and the midpoint of the lower base is approximately considered to be the position of the upper warehouse. After the projection equation limits the upper warehouse coordinate z to 0, x and y can be obtained.

[0116] or,

[0117] The position and posture of the robot upper warehouse relative to the chassis is calculated based on the visual key point coordinate set.

[0118] Assume the visual key points F of the upper warehouse: {f1,...fm};

[0119] Calculate the position [uf, vf] of the upper warehouse visual key point F in the camera pixel coordinate system:

[0120] [uf,vf]=1 / z×K×Twb×vi

[0121] Wherein, K is the camera internal parameter, z is the depth value of the upper warehouse visual key point F in the camera coordinate system, and Twb is the position of the chassis camera at a certain moment.

[0122] Calculating the position of the robot's upper warehouse relative to the chassis is divided into three steps: the first step is ordinary target detection, which can reuse the existing target detection network and results in the chassis; the second step is visual key point detection, similar to HRNet, outputting the predicted position [uf,vf] of the visual key point F in the image coordinate system; the third step is to calculate the position of the upper warehouse relative to the chassis through the set of visual key point coordinates, using the projection formula:

[0123] [ui,vi]=1 / z×K×Tbc×fi,

[0124] Where K is the camera intrinsic parameter matrix, fi is the position of the i-th feature point in the upper warehouse coordinate system. According to the known structure, ui\vi is the predicted position of the visual key point, and Tbc is calculated. At the same time, since the upper warehouse and the chassis are in the same plane, there are 6 unknowns in Tbc. The above projection formula can provide two constraints, so more than 3 feature points can constitute an overdetermined equation, and the position and posture of the upper warehouse relative to the chassis can be solved.

[0125] The present disclosure provides a robot upper warehouse visual recognition method, comprising: obtaining the posture and detection frame position of the robot box relative to the chassis according to the posture of the upper warehouse in the environment and the posture of the chassis in the environment; calculating the posture Tbc of the box in the chassis camera coordinate system; obtaining the vertex set V of the detection frame according to the posture of the robot box relative to the chassis and the detection frame position; calculating the position [ui, vi] of the vertex set V of the detection frame in the camera pixel coordinate system; traversing and comparing the position [ui, vi] of the vertex set V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinates [umax, vmax] and the lowest point coordinates [umin, vmin]; marking the data; calculating the posture of the robot upper warehouse relative to the chassis. Compared with the prior art, the present disclosure obtains the posture of the robot upper warehouse relative to the chassis by calculation, without the need to externally paste a QR code, which is beautiful and convenient; at the same time, it is not affected by the external environment, has a wide range of applications, and has high recognition accuracy.

[0126] Example 3

[0127] The present disclosure provides a robot upper warehouse visual recognition device, such as Figure 3 As shown, including:

[0128] The first obtaining unit 31 is used to obtain the posture of the robot box relative to the chassis and the position of the detection frame according to the posture of the upper warehouse in the environment and the posture of the chassis in the environment.

[0129] The first calculation unit 32 is used to calculate the position and posture Tbc of the box in the chassis camera coordinate system:

[0130] Tbc=Twb -1 ×Twc

[0131] Wherein, Twb is the position of the chassis camera at a certain moment, and Twc is the position of the box in the environment.

[0132] The second acquisition unit 33 is used to acquire a set of vertices V of the detection frame: {v1, v2, ....vn} according to the posture of the robot box relative to the chassis and the position of the detection frame.

[0133] The second calculation unit 34 is used to calculate the position [ui,vi] of each vertex set V of the detection frame in the camera pixel coordinate system:

[0134] [ui,vi]=1 / z×K×Twb×vi

[0135] Wherein, K is the intrinsic parameter of the camera, z is the depth value of the vertex set V of the detection frame in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment.

[0136] The third acquisition unit 35 is used to traverse and compare the positions [ui, vi] of each vertex set V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinates [umax, vmax] and the lowest point coordinates [umin, vmin].

[0137] The labeling unit 36 ​​is used to label the data.

[0138] The third calculation unit 37 is used to calculate the position and posture of the robot upper warehouse relative to the chassis.

[0139] It should be noted here that the detailed description of each component of this embodiment can refer to other embodiments and will not be repeated here.

[0140] The present invention provides a robot upper warehouse visual recognition device, comprising: a first obtaining unit obtains the posture and detection frame position of the robot box relative to the chassis according to the posture of the upper warehouse in the environment and the posture of the chassis in the environment; a first calculation unit calculates the posture Tbc of the box in the chassis camera coordinate system, and a second obtaining unit obtains the vertex set V of the detection frame according to the posture of the robot box relative to the chassis and the detection frame position; a second calculation unit calculates the position [ui, vi] of the vertex set V of the detection frame in the camera pixel coordinate system; a third obtaining unit traverses and compares the position [ui, vi] of the vertex set V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinates [umax, vmax] and the lowest point coordinates [umin, vmin]; a labeling unit labels the data; a third calculation unit calculates the posture of the robot upper warehouse relative to the chassis. Compared with the prior art, the present invention obtains the posture of the robot upper warehouse relative to the chassis by calculation, without the need to externally paste a QR code, which is beautiful and convenient; at the same time, it is not affected by the external environment, has a wide range of applications, and has high recognition accuracy.

[0141] Example 4

[0142] The present disclosure provides a robot upper warehouse visual recognition device, such as Figure 4 As shown, including:

[0143] The first obtaining unit 41 is used to obtain the posture of the robot box relative to the chassis and the position of the detection frame according to the posture of the upper warehouse in the environment and the posture of the chassis in the environment.

[0144] The first calculation unit 42 is used to calculate the position and posture Tbc of the box in the chassis camera coordinate system:

[0145] Tbc=Twb -1 ×Twc

[0146] Wherein, Twb is the position of the chassis camera at a certain moment, and Twc is the position of the box in the environment.

[0147] The second acquisition unit 43 is used to acquire a set of vertices V of the detection frame: {v1, v2, ....vn} according to the posture of the robot box relative to the chassis and the position of the detection frame.

[0148] The second calculation unit 44 is used to calculate the position [ui,vi] of each vertex set V of the detection frame in the camera pixel coordinate system:

[0149] [ui,vi]=1 / z×K×Twb×vi

[0150] Wherein, K is the intrinsic parameter of the camera, z is the depth value of the vertex set V of the detection frame in the camera coordinate system, and Twb is the pose of the chassis camera at a certain moment.

[0151] The third acquisition unit 45 is used to traverse and compare the positions [ui, vi] of each vertex set V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinates [umax, vmax] and the lowest point coordinates [umin, vmin].

[0152] The labeling unit 46 is used to label the data.

[0153] The first marking module 461 is used to mark the coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection frame and the orientation of the box in the posture Tbc of the chassis camera coordinate system.

[0154] or,

[0155] The second labeling module 462 is used to label the highest point coordinates [umax, vmax] and the lowest point coordinates [umin, vmin] of the detection frame and the visual key points.

[0156] The third calculation unit 47 is used to calculate the position of the robot upper warehouse relative to the chassis.

[0157] The first calculation module 471 is used to select a target detection network to obtain the position of the robot upper warehouse relative to the chassis.

[0158] or,

[0159] The second calculation module 472 is used to calculate the position of the robot upper warehouse relative to the chassis according to the visual key point coordinate set.

[0160] It should be noted here that the detailed description of each component of this embodiment can refer to other embodiments and will not be repeated here.

[0161] The present invention provides a robot upper warehouse visual recognition device, comprising: a first obtaining unit obtains the posture and detection frame position of the robot box relative to the chassis according to the posture of the upper warehouse in the environment and the posture of the chassis in the environment; a first calculation unit calculates the posture Tbc of the box in the chassis camera coordinate system, and a second obtaining unit obtains the vertex set V of the detection frame according to the posture of the robot box relative to the chassis and the detection frame position; a second calculation unit calculates the position [ui, vi] of the vertex set V of the detection frame in the camera pixel coordinate system; a third obtaining unit traverses and compares the position [ui, vi] of the vertex set V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinates [umax, vmax] and the lowest point coordinates [umin, vmin]; a labeling unit labels the data; a third calculation unit calculates the posture of the robot upper warehouse relative to the chassis. Compared with the prior art, the present invention obtains the posture of the robot upper warehouse relative to the chassis by calculation, without the need to externally paste a QR code, which is beautiful and convenient; at the same time, it is not affected by the external environment, has a wide range of applications, and has high recognition accuracy.

[0162] Example 5

[0163] The present disclosure provides a robot upper warehouse visual recognition electronic device, such as Figure 5 As shown, it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned robot upper warehouse visual recognition steps are implemented.

[0164] A processing device (e.g., a central processing unit, a graphics processing unit, etc.) 51 can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 52 or a program loaded from a storage device 58 into a random access memory (RAM) 53. Various programs and data required for the operation of the electronic device are also stored in the RAM 53. The processing device 51, the ROM 52, and the RAM 53 are connected to each other via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0165] Typically, the following devices may be connected to the I / O interface 55: an input device 56 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 57 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 58 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 59. The communication device 59 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 5 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 5 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0166] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 59, or installed from the storage device 58, or installed from the ROM 52. When the computer program is executed by the processing device 51, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0167] The present disclosure provides a robot upper warehouse visual recognition electronic device, comprising: obtaining the posture and detection frame position of the robot box relative to the chassis according to the posture of the upper warehouse in the environment and the posture of the chassis in the environment; calculating the posture Tbc of the box in the chassis camera coordinate system; obtaining the set of vertices V of the detection frame according to the posture of the robot box relative to the chassis and the detection frame position; calculating the position [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system; traversing and comparing the position [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinates [umax,vmax] and the lowest point coordinates [umin,vmin]; marking the data; calculating the posture of the robot upper warehouse relative to the chassis. Compared with the prior art, the present disclosure obtains the posture of the robot upper warehouse relative to the chassis by calculation, without the need to externally paste a QR code, which is beautiful and convenient; at the same time, it is not affected by the external environment, has a wide range of applications, and has high recognition accuracy.

[0168] Example 6

[0169] An embodiment of the present disclosure provides a computer-readable storage medium for robot warehouse visual recognition, wherein a computer program is stored on the computer-readable storage medium. When the computer program is executed by one or more processors, the above-mentioned robot warehouse visual recognition method is implemented.

[0170] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0171] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0172] The computer-readable medium may be included in the device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the posture and detection frame position of the robot box relative to the chassis according to the posture of the upper warehouse in the environment and the posture of the chassis in the environment; calculates the posture Tbc of the box in the camera coordinate system of the chassis; obtains the vertex set V of the detection frame according to the posture of the robot box relative to the chassis and the detection frame position; calculates the position [ui,vi] of the vertex set V of the detection frame in the camera pixel coordinate system; traverses and compares the position [ui,vi] of the vertex set V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinates [umax,vmax] and the lowest point coordinates [umin,vmin]; labels the data; calculates the posture of the robot upper warehouse relative to the chassis.

[0173] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0174] The present disclosure provides a computer-readable storage medium for robot upper warehouse visual recognition, including: obtaining the posture and detection frame position of the robot box relative to the chassis according to the posture of the upper warehouse in the environment and the posture of the chassis in the environment; calculating the posture Tbc of the box in the chassis camera coordinate system; obtaining the set of vertices V of the detection frame according to the posture of the robot box relative to the chassis and the detection frame position; calculating the position [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system; traversing and comparing the position [ui,vi] of the set of vertices V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinates [umax,vmax] and the lowest point coordinates [umin,vmin]; annotating the data; calculating the posture of the robot upper warehouse relative to the chassis. Compared with the prior art, the present disclosure obtains the posture of the robot upper warehouse relative to the chassis by calculation, without the need to externally paste a QR code, which is beautiful and convenient; at the same time, it is not affected by the external environment, has a wide range of applications, and has high recognition accuracy.

[0175] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0176] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes a receiving unit, an information acquisition unit, a target determination unit, and a distribution unit. The names of these units do not constitute limitations on the units themselves in some cases.

[0177] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0178] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit them; although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention. In particular, as long as there is no structural conflict, the various technical features mentioned in each embodiment can be combined in any way. The present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions that fall within the scope of the claims.

Claims

1. A robot warehouse visual recognition method, characterized in that: include: According to the posture of the upper warehouse and the posture of the chassis in the environment, the posture of the robot box relative to the chassis and the position of the detection frame are obtained; Calculate the position Tbc of the box in the chassis camera coordinate system: Tbc=Twb -1 ×Twc Wherein, Twb is the position of the chassis camera at a certain moment, and Twc is the position of the box in the environment; According to the posture of the robot box relative to the chassis and the position of the detection frame, obtain a set of vertices V of the detection frame: {v1, v2, ....vn}; Calculate the position [ui,vi] of each vertex set V of the detection frame in the camera pixel coordinate system: [ui,vi]=1 / z×K×Twb×vi Wherein, K is the intrinsic parameter of the camera, z is the depth value of the vertex set V of the detection frame in the camera coordinate system, and Twb is the position of the chassis camera at a certain moment; Traverse and compare the positions [ui,vi] of each vertex set V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinates [umax,vmax] and the lowest point coordinates [umin,vmin]; Label the data; The position of the robot upper warehouse relative to the chassis is calculated.

2. The robot upper warehouse visual recognition method according to claim 1, characterized in that: Label the data, including: Mark the coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection frame and the orientation of the box in the position Tbc of the chassis camera coordinate system; or, The highest point coordinates [umax, vmax] and the lowest point coordinates [umin, vmin] of the detection box and the visual key points are marked.

3. The robot upper warehouse visual recognition method according to claim 2, characterized in that: Annotate visual key points, including: Assume the visual key points F of the upper warehouse: {f1,...fm}; Calculate the position [uf, vf] of the upper warehouse visual key point F in the camera pixel coordinate system: [uf,vf]=1 / z×K×Twb×vi Wherein, K is the camera internal parameter, z is the depth value of the upper warehouse visual key point F in the camera coordinate system, and Twb is the position of the chassis camera at a certain moment.

4. The robot upper warehouse visual recognition method according to claim 1, characterized in that: Calculating the position of the robot upper warehouse relative to the chassis, including: Using a target detection network to obtain the position of the robot's upper warehouse relative to the chassis; or, The position of the robot upper warehouse relative to the chassis is calculated based on the visual key point coordinate set.

5. The robot upper warehouse visual recognition method according to claim 1, characterized in that: The position of the upper warehouse in the environment includes: The position and posture of the upper warehouse in the environment is obtained through a motion capture camera.

6. The robot upper warehouse visual recognition method according to claim 1, characterized in that: The position and posture of the chassis in the environment include: The position and posture of the chassis in the environment is obtained through a motion capture camera or chassis positioning.

7. The robot upper warehouse visual recognition method according to claim 1, characterized in that: The position and posture of the chassis in the environment include: The position and posture of the chassis in the environment are obtained by chassis positioning.

8. The robot upper warehouse visual recognition method according to claim 4, characterized in that: The target detection network includes: The target detection network is yolo or SSD or mobilenet.

9. The robot upper warehouse visual recognition method according to claim 8, characterized in that: The target detection network includes: The target detection network is SSD.

10. The robot upper warehouse visual recognition method according to claim 4, characterized in that: Select the target detection network to obtain the position of the robot's upper warehouse relative to the chassis, including: The target detection network is selected and combined with the L2 loss of the angle error to obtain the position of the robot's upper warehouse relative to the chassis.

11. A robot warehouse visual recognition device, characterized in that: include: A first obtaining unit is used to obtain the posture of the robot box relative to the chassis and the position of the detection frame according to the posture of the upper warehouse in the environment and the posture of the chassis in the environment; The first calculation unit is used to calculate the position and posture Tbc of the box in the chassis camera coordinate system: Tbc=Twb -1 ×Twc Wherein, Twb is the position of the chassis camera at a certain moment, and Twc is the position of the box in the environment; A second acquisition unit is used to acquire a set V of vertices of the detection frame: {v1, v2, ....vn} according to the posture of the robot box relative to the chassis and the position of the detection frame; The second calculation unit is used to calculate the position [ui,vi] of each vertex set V of the detection frame in the camera pixel coordinate system: [ui,vi]=1 / z×K×Twb×vi Wherein, K is the intrinsic parameter of the camera, z is the depth value of the vertex set V of the detection frame in the camera coordinate system, and Twb is the position of the chassis camera at a certain moment; The third acquisition unit is used to traverse and compare the positions [ui, vi] of each vertex set V of the detection frame in the camera pixel coordinate system to obtain the highest point coordinates [umax, vmax] and the lowest point coordinates [umin, vmin]; A labeling unit, used to label data; The third calculation unit is used to calculate the position of the robot upper warehouse relative to the chassis.

12. The robot upper warehouse visual recognition device according to claim 11, characterized in that: The annotation units include: A first marking module is used to mark the coordinates of the highest point [umax, vmax] and the lowest point [umin, vmin] of the detection frame and the orientation of the box in the posture Tbc of the chassis camera coordinate system; or, The second marking module is used to mark the highest point coordinates [umax, vmax] and the lowest point coordinates [umin, vmin] of the detection frame and the visual key points.

13. The robot upper warehouse visual recognition device according to claim 11, characterized in that: The third computing unit includes: A first computing module is used to select a target detection network to obtain the position of the robot upper warehouse relative to the chassis; or, The second calculation module is used to calculate the position and posture of the robot upper warehouse relative to the chassis according to the visual key point coordinate set.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images