Method for forking goods by AGV (Automatic Guided Vehicle) and AGV

Abstract

The invention discloses an AGV goods forking method and an AGV, the AGV is provided with a depth camera, and the AGV goods forking method comprises the steps that a first RGB image and a first depth image collected by the depth camera are acquired; identifying a predetermined target according to the first RGB image and the first depth image; obtaining a second RGB image and a second depth image collected by a depth camera; according to the second RGB image and the second depth image, respectively identifying a target action and a pallet area of the predetermined target; and according to the target action and the pallet area, a target pallet to be forked is determined. According to the method for forking the goods by the AGV, the predetermined target, the target action and the pallet area can be identified according to the images collected by the depth camera, and the target pallet to be forked is determined, so that the goods are forked automatically.

Description

Technical Field

[0001] The present application relates to the field of image detection technology, and more specifically to an AGV fork-picking method and an AGV. Background Art

[0002] In the non-mapped area of ​​the warehouse or outside the warehouse, it is difficult for AGV (Automated Guided Vehicle) to complete automatic navigation and forklift cargo, and manual guidance is required. In the related technology, the AGV is guided to the designated area by following the AGV, and then the cargo is moved to the AGV manually. This method is time-consuming and labor-intensive, and the AGV is not fully developed. Summary of the invention

[0003] The present invention is proposed to solve at least one of the above problems. According to the first aspect of the present application, a method for AGV to pick up goods is provided, wherein the AGV is provided with a depth camera, and the method comprises: obtaining a first RGB image and a first depth image acquired by the depth camera; identifying a predetermined target according to the first RGB image and the first depth image; obtaining a second RGB image and a second depth image acquired by the depth camera; identifying a target action and a pallet area of ​​the predetermined target according to the second RGB image and the second depth image, respectively; and determining a target pallet to be picked up according to the target action and the pallet area.

[0004] In one embodiment of the present application, the step of identifying a predetermined target based on the first RGB image and the first depth image includes: identifying the first RGB image using a recognition detection model trained with a first training set, and outputting a target recognition result; and obtaining the predetermined target based on the target recognition result and the first depth image.

[0005] In one embodiment of the present application, the step of respectively identifying the target action and the pallet area of ​​the predetermined target according to the second RGB image and the second depth image includes: detecting the second RGB image using a recognition detection model trained with a second training set, and outputting an action recognition result; obtaining the target action according to the action recognition result and the second depth image; detecting the second RGB image using a recognition detection model trained with a third training set, and outputting a pallet recognition result; and obtaining the pallet area according to the pallet recognition result and the second depth image.

[0006] In one embodiment of the present application, the recognition detection model includes: a backbone network for feature extraction, a neck network for feature fusion, and a head network for target detection, which are connected in sequence; the neck network includes a multi-scale weighted fusion attention module and a feature fusion module; the multi-scale weighted fusion attention module is connected to the backbone network, and is used to perform information fusion on the features output by the backbone network to obtain information fusion features of different scales; the feature fusion module is respectively connected to the multi-scale weighted fusion attention module and the backbone network, and is used to perform feature fusion on the features output by the backbone network and the information fusion features of different scales.

[0007] In one embodiment of the present application, the multi-scale weighted fusion attention module includes: a first attention module, connected to the backbone network, for processing the first path of features output by the backbone network and outputting the first feature; a second attention module, connected to the backbone network, for processing the second path of features output by the backbone network and outputting the second feature; a third attention module, connected to the backbone network, for processing the third path of features output by the backbone network and outputting the third feature; a weight allocation module, respectively connected to the first attention module, the second attention module and the third attention module, for performing weighted sum processing on the first feature, the second feature and the third feature, and outputting a fourth feature; a fourth attention module, connected to the weight allocation module, for processing the fourth feature and outputting the information fusion features of different scales.

[0008] In one embodiment of the present application, the first attention module includes a first convolutional layer and a first channel attention module; the second attention module includes a second channel attention module; the third attention module includes a third channel attention module and an upsampling module; and the fourth attention module includes a first spatial attention module.

[0009] In one embodiment of the present application, the output of the first attention module is expressed by the following formula:

[0010] X11=ChannelAttention(Conv(X1))

[0011] The output of the second attention module is expressed as follows:

[0012] X21=ChannelAttention(X2)

[0013] The output of the third attention module is expressed by the following formula:

[0014] X31=ChannelAttention(Upsample(X3))

[0015] The output of the weight distribution module is expressed by the following formula:

[0016]

[0017] The output of the fourth attention module is expressed by the following formula:

[0018] Y2=SpatialAttention(Y1)

[0019] Among them, X1 represents the first path feature, X2 represents the second path feature, X3 represents the third path feature, Wi represents the weight coefficient, and i is represented by 1, 2, and 3.

[0020] In one embodiment of the present application, the first channel attention module, the second channel attention module and the third channel attention module are each composed of a first average pooling layer, a second convolutional layer, a first activation layer and a first Hadamard product module connected in sequence, wherein the first Hadamard product module is also connected to the second convolutional layer.

[0021] In one embodiment of the present application, the outputs of the first channel attention module, the second channel attention module, and the third channel attention module are expressed by the following formula:

[0022]

[0023] Among them, X represents the input features.

[0024] In one embodiment of the present application, the first spatial attention module includes a second average pooling layer, a maximum pooling layer, a connection layer, a second activation layer and a second Hadamard product module; wherein the maximum pooling layer, the connection layer, the second activation layer and the second Hadamard product module are connected in sequence, the second Hadamard product module is also connected to the weight allocation module, the input end of the second average pooling layer is connected to the weight allocation module, and the output end of the second average pooling layer is connected to the connection layer.

[0025] In one embodiment of the present application, the output of the first spatial attention module is expressed by the following formula:

[0026]

[0027] Among them, X represents the input features.

[0028] In one embodiment of the present application, the step of obtaining the predetermined target based on the target recognition result and the first depth image includes: determining the predetermined target based on pixel coordinates, an intrinsic parameter matrix of the depth camera and a depth value in the first depth image; wherein the recognition result includes the pixel coordinates of the predetermined target, the position coordinates of the predetermined target are in a predetermined relationship with the pixel coordinates, the parameters in the intrinsic parameter matrix and the depth value, and the parameters in the intrinsic parameter matrix include the horizontal coordinate of the focal length, the vertical coordinate of the focal length, the horizontal coordinate of the optical center and the vertical coordinate of the optical center.

[0029] In one embodiment of the present application, the position coordinates of the predetermined target are in a predetermined relationship with the pixel point coordinates, the parameters in the intrinsic parameter matrix and the depth value, specifically: the horizontal coordinate of the position coordinate is in a predetermined relationship with the vertical coordinate of the position coordinate, the horizontal coordinate of the pixel point coordinate, the horizontal coordinate of the optical center and the horizontal coordinate of the focal length; the vertical coordinate of the position coordinate is in a predetermined relationship with the vertical coordinate of the position coordinate, the vertical coordinate of the pixel point coordinate, the vertical coordinate of the optical center and the vertical coordinate of the focal length; the vertical coordinate of the position coordinate is in a predetermined relationship with the depth value.

[0030] In one embodiment of the present application, the position coordinates of the predetermined target are in a predetermined relationship with the pixel point coordinates, the parameters in the intrinsic parameter matrix and the depth value, specifically: the horizontal coordinate of the position coordinate is in a predetermined relationship with the vertical coordinate of the position coordinate, the difference between the horizontal coordinate of the pixel point coordinate and the horizontal coordinate of the optical center and the quotient of the horizontal coordinate of the focal length; the vertical coordinate of the position coordinate is in a predetermined relationship with the vertical coordinate of the position coordinate, the difference between the vertical coordinate of the pixel point coordinate and the vertical coordinate of the optical center and the quotient of the vertical coordinate of the focal length; the vertical coordinate of the position coordinate is equal to the depth value.

[0031] In one embodiment of the present application, the abscissa of the position coordinate is expressed by the following formula:

[0032] X=Z(uc x ) / f x

[0033] The ordinate of the position coordinate is expressed by the following formula:

[0034] Y=Z(vc y ) / f y

[0035] The vertical coordinate of the position coordinate is expressed by the following formula:

[0036] Z=d

[0037] Wherein, X represents the horizontal coordinate of the position coordinate, Y represents the vertical coordinate of the position coordinate, Z represents the vertical coordinate of the position coordinate, u represents the horizontal coordinate of the pixel point coordinate, v represents the vertical coordinate of the pixel point coordinate, c represents the horizontal coordinate of the pixel point coordinate, x represents the abscissa of the optical center, c y represents the ordinate of the optical center, f x represents the focal length abscissa, f y represents the focal length ordinate, and d represents the depth value.

[0038] In one embodiment of the present application, the step of determining the target pallet to be forked based on the target action and the pallet area includes: determining the action direction based on the target action; when the action direction passes through the pallet area, the pallet that overlaps with the action direction in the pallet area is used as the target pallet to be forked.

[0039] In one embodiment of the present application, after identifying the predetermined target based on the RGB image and the depth image, the method further includes: maintaining a predetermined distance from the predetermined target and following the predetermined target to reach a designated position; at the designated position, the depth camera acquires the second RGB image and the second depth image.

[0040] In one embodiment of the present application, after the step of determining the target pallet to be forked according to the target action and the pallet area, the method further includes: the AGV moves to the target pallet position to fork the goods.

[0041] In one embodiment of the present application, the target action is a gesture.

[0042] According to a second aspect of the present application, an AGV is provided, on which a depth camera is provided, and the AGV includes: an image acquisition module, used to acquire a first RGB image and a first depth image acquired by the depth camera; a target detection module, used to identify a predetermined target based on the first RGB image and the first depth image; the image acquisition module is also used to acquire a second RGB image and a second depth image acquired by the depth camera; a recognition module, used to respectively identify a target action and a pallet area of ​​the predetermined target based on the second RGB image and the second depth image; a pallet determination module, used to determine a target pallet to be forked based on the target action and the pallet area.

[0043] According to the AGV forking cargo method and AGV provided in the embodiments of the present application, it is possible to identify the predetermined target, target action and pallet area respectively based on the image captured by the depth camera, and determine the target pallet to be forked to automatically fork the cargo. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0045] Figure 1 A schematic diagram of a flow chart of a method for AGV picking up goods provided in one embodiment of the present invention;

[0046] Figure 2 A schematic diagram of identifying a staff member provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of identifying a pallet area provided by an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of the structure of an identification and detection model provided by an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of determining a target pallet to be forked provided by an embodiment of the present invention;

[0050] Figure 6 A schematic diagram of the structure of an AGV provided in one embodiment of the present invention.

[0051] Description of reference numerals:

[0052] 1: AGV; 2: depth camera; 3: AGV fork; 4: staff; 5: pedestrian detection rectangle; 6: goods; 7: pallet; 8: pallet recognition rectangle; 9: gesture pointing direction. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0054] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features well known in the art are not described.

[0055] It should be understood that the present invention can be implemented in different forms and should not be interpreted as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to make the disclosure thorough and complete and to fully convey the scope of the present invention to those skilled in the art.

[0056] In order to fully understand the present invention, a detailed structure will be proposed in the following description to illustrate the technical solution proposed by the present invention. The optional embodiments of the present invention are described in detail as follows, but in addition to these detailed descriptions, the present invention may also have other implementations.

[0057] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0058] The first aspect of the present application provides an AGV fork-picking method, such as Figure 1 As shown, the AGV is equipped with a depth camera, and the AGV fork-picking method includes:

[0059] S1, obtaining a first RGB image and a first depth image captured by a depth camera.

[0060] As an example, a depth camera can be set at the front or top of the AGV. The RGB image collected by the depth camera can provide color information of the object, and the collected depth image can provide distance information between the object and the depth camera.

[0061] S2: Identify a predetermined target according to the first RGB image and the first depth image.

[0062] It should be noted that the AGV may utilize image processing and machine learning techniques, such as a target detection algorithm, to identify a predetermined target in the first RGB image and the first depth image.

[0063] As an example, the predetermined target may be a worker. After the worker issues a work instruction, the AGV identifies the worker based on the acquired image.

[0064] S3, obtaining a second RGB image and a second depth image captured by the depth camera.

[0065] S4, identifying a target action and a pallet area of ​​a predetermined target respectively according to the second RGB image and the second depth image.

[0066] Specifically, the target action of the predetermined target is a gesture.

[0067] S5, determining the target pallet to be forked according to the target action and the pallet area.

[0068] It should be noted that the pallet area should include one or more pallets.

[0069] The AGV cargo picking method of the embodiment of the present invention can identify the predetermined target, target action and pallet area respectively according to the image captured by the depth camera, and determine the target pallet to be picked up, so as to automatically pick up the goods.

[0070] In some embodiments, the step of identifying a predetermined target according to the first RGB image and the first depth image includes:

[0071] S211, using the recognition detection model trained by the first training set to recognize the first RGB image, and output a target recognition result.

[0072] It should be noted that the first training set may include RGB images with labels, each target is marked or annotated with a corresponding label, and the recognition detection model may be a lightweight target detection model.

[0073] S212: Obtain a predetermined target according to the target recognition result and the first depth image.

[0074] like Figure 2 As shown, the staff can be accurately identified by the depth camera.

[0075] In this embodiment, by combining the target recognition result obtained from the first RGB image with the first depth image, the position of the predetermined target can be determined more accurately.

[0076] In some embodiments, the step of respectively identifying the target action and the pallet area of ​​the predetermined target according to the second RGB image and the second depth image includes:

[0077] S41, using the recognition detection model trained with the second training set to detect the second RGB image, and outputting an action recognition result.

[0078] It should be noted that the second training set may include RGB images with labels, the action of each target is marked or annotated with a corresponding label, and the recognition detection model may be a lightweight target detection model.

[0079] S42, obtaining a target action according to the action recognition result and the second depth image.

[0080] S43, using the recognition detection model trained with the third training set to detect the second RGB image, and outputting a pallet recognition result.

[0081] It should be noted that the third training set may include RGB images with labels, each stack is marked or annotated with a corresponding label, and the recognition detection model may be a lightweight target detection model.

[0082] S44, obtaining a pallet area according to the pallet recognition result and the second depth image.

[0083] like Figure 3 As shown, the pallet area can be accurately identified by the depth camera.

[0084] In this embodiment, by combining the action recognition result and the pallet recognition result respectively obtained from the second RGB image with the first depth image, the target action and the pallet area can be determined more accurately.

[0085] The recognition detection model may adopt YOLOv8n, or other models may be selected according to the actual application effect, such as YOLOv7, YOLOv6, etc. Next, the recognition detection model includes YOLOv8n for explanation.

[0086] In some embodiments, Figure 4 As shown, the recognition and detection model includes: a backbone network for feature extraction, a neck network for feature fusion, and a head network for target detection, which are connected in sequence.

[0087] The neck network includes a multi-scale weighted fusion attention module (MWFA) and a feature fusion module.

[0088] The multi-scale weighted fusion attention module is connected to the backbone network and is used to fuse the features output by the backbone network to obtain information fusion features of different scales.

[0089] The feature fusion module is connected to the multi-scale weighted fusion attention module and the backbone network respectively, and is used to perform feature fusion on the features output by the backbone network and the information fusion features of different scales.

[0090] Specifically, the multi-scale weighted fusion attention module includes:

[0091] The first attention module is connected to the backbone network, and is used to process the first path of features output by the backbone network and output the first feature.

[0092] The second attention module is connected to the backbone network and is used to process the second path of features output by the backbone network and output the second features.

[0093] The third attention module is connected to the backbone network and is used to process the third feature output by the backbone network and output the third feature.

[0094] The weight allocation module is connected to the first attention module, the second attention module and the third attention module respectively, and is used to perform weighted sum processing on the first feature, the second feature and the third feature, and output the fourth feature.

[0095] The fourth attention module is connected to the weight allocation module, and is used to process the fourth feature and output information fusion features of different scales.

[0096] More specifically, the first attention module includes a first convolutional layer and a first channel attention module; the second attention module includes a second channel attention module; the third attention module includes a third channel attention module and an upsampling module; and the fourth attention module includes a first spatial attention module.

[0097] As an example, the output of the first attention module is expressed as follows:

[0098] X11=ChannelAttention(Conv(X1))

[0099] The output of the second attention module is expressed as follows:

[0100] X21=ChannelAttention(X2)

[0101] The output of the third attention module is expressed as follows:

[0102] X31=ChannelAttention(Upsample(X3))

[0103] The output of the weight distribution module is expressed as follows:

[0104]

[0105] The output of the fourth attention module is expressed as follows:

[0106] Y2=SpatialAttention(Y1)

[0107] Among them, X1 represents the first path feature, X2 represents the second path feature, X3 represents the third path feature, Wi represents the weight coefficient, and i is represented by 1, 2, and 3.

[0108] More specifically, the first channel attention module, the second channel attention module and the third channel attention module are each composed of a first average pooling layer, a second convolutional layer, a first activation layer and a first Hadamard product module connected in sequence, wherein the first Hadamard product module is also connected to the second convolutional layer.

[0109] As an example, the outputs of the first channel attention module, the second channel attention module, and the third channel attention module are expressed by the following formula:

[0110]

[0111] Among them, X represents the input features.

[0112] More specifically, the first spatial attention module includes a second average pooling layer, a maximum pooling layer, a connection layer, a second activation layer, and a second Hadamard product module. Among them, the maximum pooling layer, the connection layer, the second activation layer, and the second Hadamard product module are connected in sequence, the second Hadamard product module is also connected to the weight distribution module, the input end of the second average pooling layer is connected to the weight distribution module, and the output end of the second average pooling layer is connected to the connection layer.

[0113] As an example, the output of the first spatial attention module is expressed as follows:

[0114]

[0115] Among them, X represents the input features.

[0116] Accordingly, the YOLOv8n network containing the multi-scale weighted fusion attention module can be denoted as MWFA-YOLOv8n.

[0117] Next, the advantages of the present invention are explained by comparing the experimental results of YOLOv8n and MWFA-YOLOv8n. When the number of parameters in the network is slightly increased, the experimental results are as follows:

[0118] As shown in Tables 1, 2, and 3, when the number of parameters in the network is slightly increased, the recognition accuracy is greatly improved and the recognition effect is significantly improved.

[0119] Table 1 Experimental results of target detection

[0120] Model <![CDATA[Parameter quantity (×10 6 )]]> mAP(0.5) / % YOLOv8n 3.0 97.7 MWFA-YOLOv8n 3.4 98.8

[0121] Table 2 Experimental results of target action recognition

[0122] Model <![CDATA[Parameter quantity (×10 6 )]]> mAP(0.5) / % YOLOv8n 3.0 97.1 MWFA-YOLOv8n 3.4 99.0

[0123] Table 3 Experimental results of pallet area recognition

[0124] Model <![CDATA[Parameter quantity (×10 6 )]]> mAP(0.5) / % YOLOv8n 3.0 96.1 MWFA-YOLOv8n 3.4 97.5

[0125] In this embodiment, since the traditional YOLOv8n network uses a simple splicing layer, it fails to distinguish the importance of different features, and the importance of feature channels to recognition is different. By adding a multi-scale weighted fusion attention module, it is possible to obtain information fusion features of different scales with weights, and then continue to perform feature fusion in the neck network and finally output it to the head network.

[0126] In some embodiments, the step of obtaining a predetermined target according to the target recognition result and the first depth image includes:

[0127] The predetermined target is determined according to the pixel coordinates, the intrinsic parameter matrix of the depth camera and the depth value in the first depth image.

[0128] Among them, the recognition result includes the pixel point coordinates of the predetermined target, the position coordinates of the predetermined target are in a predetermined relationship with the pixel point coordinates, the parameters in the intrinsic parameter matrix and the depth value, and the parameters in the intrinsic parameter matrix include the focal length horizontal coordinate, the focal length vertical coordinate, the optical center horizontal coordinate and the optical center vertical coordinate.

[0129] Specifically, the position coordinates of the predetermined target are in a predetermined relationship with the pixel coordinates, the parameters in the intrinsic parameter matrix and the depth value, specifically:

[0130] The horizontal coordinate of the position coordinate has a predetermined relationship with the vertical coordinate of the position coordinate, the horizontal coordinate of the pixel point coordinate, the horizontal coordinate of the optical center and the horizontal coordinate of the focal length; the vertical coordinate of the position coordinate has a predetermined relationship with the vertical coordinate of the position coordinate, the vertical coordinate of the pixel point coordinate, the vertical coordinate of the optical center and the vertical coordinate of the focal length; the vertical coordinate of the position coordinate has a predetermined relationship with the depth value.

[0131] More specifically, the position coordinates of the predetermined target are in a predetermined relationship with the pixel coordinates, the parameters in the intrinsic parameter matrix and the depth value, specifically:

[0132] The horizontal coordinate of the position coordinate and the vertical coordinate of the position coordinate, the difference between the horizontal coordinate of the pixel point coordinate and the horizontal coordinate of the optical center and the horizontal coordinate of the focal length form a predetermined relationship; the vertical coordinate of the position coordinate and the vertical coordinate of the position coordinate, the difference between the vertical coordinate of the pixel point coordinate and the vertical coordinate of the optical center and the horizontal coordinate of the focal length form a predetermined relationship; the vertical coordinate of the position coordinate is equal to the depth value.

[0133] More specifically, the abscissa of the position coordinates is expressed by the following formula:

[0134] X=Z(uc x ) / f x

[0135] The ordinate of the position coordinate is expressed by the following formula:

[0136] Y=Z(vc y ) / f y

[0137] The vertical coordinate of the position coordinate is expressed by the following formula:

[0138] Z=d

[0139] Among them, X represents the horizontal coordinate of the position coordinate, Y represents the vertical coordinate of the position coordinate, Z represents the vertical coordinate of the position coordinate, u represents the horizontal coordinate of the pixel point coordinate, v represents the vertical coordinate of the pixel point coordinate, and c x represents the abscissa of the optical center, c y represents the ordinate of the optical center, f x Represents the focal length horizontal axis, f y represents the focal length ordinate, and d represents the depth value.

[0140] As an example, the calculation formula for combining the target recognition result and the first depth image can be expressed as:

[0141]

[0142] Among them, P uv represents the pixel coordinates of the RGB image, K represents the intrinsic parameter matrix of the depth camera, and P represents the coordinates of the depth image.

[0143] In this embodiment, the position coordinates of the target are calculated by combining the intrinsic parameter matrix and the depth value of the depth camera, so that the accuracy, real-time and robustness of target recognition can be improved.

[0144] In some embodiments, the step of determining the target pallet to be forked according to the target action and the pallet area includes:

[0145] S51, determining the action direction according to the target action.

[0146] As an example, the action pointing may be a gesture pointing by a staff member.

[0147] S52, when the action is directed through the pallet area, the action is directed to the pallet that overlaps with the pallet area as the target pallet to be forked.

[0148] As an example, according to the identified target action and the point cloud information of the pallet area, the action direction is fitted into a ray, such as Figure 5 As shown, the angle between the action direction and the depth camera plane is θ, and it is determined whether the ray passes through the pallet area, and the pallet passing through the pallet area is regarded as the target pallet to be forked.

[0149] In some embodiments, after identifying the predetermined target according to the RGB image and the depth image, the AGV fork-to-cargo method further includes:

[0150] S221, maintaining a predetermined distance from a predetermined target and following the predetermined target to reach a designated location.

[0151] As an example, the predetermined distance may be set to 2.5-3.5 meters.

[0152] S222: At the specified position, the depth camera acquires a second RGB image and a second depth image.

[0153] In this embodiment, by maintaining a predetermined distance from a predetermined target, the safety of the staff can be ensured and the imaging quality of the depth camera is good.

[0154] In some embodiments, after the step of determining the target pallet to be forked according to the target action and the pallet area, the AGV forking goods method further includes: the AGV moves to the target pallet position to fork the goods.

[0155] As an example, after the AGV has finished picking up the goods, it returns to its original position and continues to follow the staff. After reaching the destination, it re-performs the target action recognition, places the goods in the designated area, and controls the AGV to complete the unloading.

[0156] As another example, when cargo unloading is required, the target action is obtained and the action direction is determined. The position where the action direction coincides with the ground is used as the placement area. After the coordinate information of the placement area is calculated, the path is generated through short-term path planning, and the AGV is controlled to travel to the placement area according to the planned path to complete the unloading.

[0157] In addition, the present application also provides an AGV, such as Figure 6 As shown, the AGV is equipped with a depth camera, and the AGV includes:

[0158] The image acquisition module 100 is used to acquire a first RGB image and a first depth image captured by a depth camera.

[0159] The target detection module 200 is used to identify a predetermined target according to the first RGB image and the first depth image.

[0160] The image acquisition module 100 is also used to acquire a second RGB image and a second depth image captured by the depth camera.

[0161] The recognition module 300 is used to recognize the target action and the pallet area of ​​the predetermined target according to the second RGB image and the second depth image.

[0162] The pallet determination module 400 is used to determine the target pallet to be forked according to the target action and the pallet area.

[0163] The AGV of the embodiment of the present invention can respectively identify the predetermined target, the target action and the pallet area according to the image collected by the depth camera, and determine the target pallet to be forked, so as to automatically fork the goods.

[0164] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present application to this. Those of ordinary skill in the art may make various changes and modifications therein without departing from the scope and spirit of the present application. All these changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0165] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0166] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0167] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0168] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present application should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with features less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0169] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0170] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0171] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some modules according to the embodiments of the present application. The application can also be implemented as a device program (e.g., computer program and computer program product) for executing a part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0172] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0173] The above is only a specific implementation or description of a specific implementation of the present application, and the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An AGV fork-picking method, characterized in that: The AGV is provided with a depth camera, and the method comprises: Acquire a first RGB image and a first depth image captured by the depth camera; Identify a predetermined target according to the first RGB image and the first depth image; Acquire a second RGB image and a second depth image captured by the depth camera; Respectively identifying a target action and a pallet area of ​​the predetermined target according to the second RGB image and the second depth image; A target pallet to be forked is determined according to the target action and the pallet area.

2. The AGV fork-picking method of claim 1, characterized in that: The step of identifying a predetermined target according to the first RGB image and the first depth image comprises: Using the recognition detection model trained with the first training set to recognize the first RGB image, and output a target recognition result; The predetermined target is obtained according to the target recognition result and the first depth image.

3. The AGV fork-picking method of claim 1, characterized in that: The step of respectively identifying the target action and the pallet area of ​​the predetermined target according to the second RGB image and the second depth image comprises: Using the recognition detection model trained with the second training set to detect the second RGB image, and output an action recognition result; Obtaining the target action according to the action recognition result and the second depth image; Using the recognition detection model trained with the third training set to detect the second RGB image, and outputting a pallet recognition result; The pallet area is obtained according to the pallet recognition result and the second depth image.

4. The AGV fork-picking method as claimed in claim 2 or 3, characterized in that: The recognition detection model includes: a backbone network for feature extraction, a neck network for feature fusion, and a head network for target detection, which are connected in sequence; The neck network includes a multi-scale weighted fusion attention module and a feature fusion module; The multi-scale weighted fusion attention module is connected to the backbone network and is used to fuse the features output by the backbone network to obtain information fusion features of different scales; The feature fusion module is connected to the multi-scale weighted fusion attention module and the backbone network respectively, and is used to perform feature fusion on the features output by the backbone network and the information fusion features of different scales.

5. The AGV fork-picking method of claim 4, characterized in that: The multi-scale weighted fusion attention module comprises: A first attention module is connected to the backbone network and is used to process the first path of features output by the backbone network and output a first feature; A second attention module is connected to the backbone network and is used to process the second path of features output by the backbone network and output a second feature; A third attention module is connected to the backbone network and is used to process the third path of features output by the backbone network and output a third feature; a weight allocation module, connected to the first attention module, the second attention module and the third attention module respectively, and configured to perform weighted sum processing on the first feature, the second feature and the third feature, and output a fourth feature; The fourth attention module is connected to the weight allocation module, and is used to process the fourth feature and output the information fusion features of different scales.

6. The AGV fork-picking method of claim 5, characterized in that: The first attention module includes a first convolutional layer and a first channel attention module; The second attention module includes a second channel attention module; The third attention module includes a third channel attention module and an upsampling module; The fourth attention module includes the first spatial attention module.

7. The AGV fork-picking method of claim 6, characterized in that: The output of the first attention module is expressed by the following formula: X11=ChannelAttention(Conv(X1)) The output of the second attention module is expressed as follows: X21=ChannelAttention(X2) The output of the third attention module is expressed by the following formula: X31=ChannelAttention(Upsample(X3)) The output of the weight distribution module is expressed by the following formula: The output of the fourth attention module is expressed by the following formula: Y2=SpatialAttention(Y1) Among them, X1 represents the first path feature, X2 represents the second path feature, X3 represents the third path feature, Wi represents the weight coefficient, and i is represented by 1, 2, and 3.

8. The AGV fork-picking method of claim 6, characterized in that: The first channel attention module, the second channel attention module and the third channel attention module are each composed of a first average pooling layer, a second convolutional layer, a first activation layer and a first Hadamard product module connected in sequence, wherein the first Hadamard product module is also connected to the second convolutional layer.

9. The AGV fork-picking method of claim 8, characterized in that: The outputs of the first channel attention module, the second channel attention module, and the third channel attention module are expressed by the following formula: Among them, X represents the input features.

10. The AGV fork-picking method of claim 6, characterized in that: The first spatial attention module includes a second average pooling layer, a maximum pooling layer, a connection layer, a second activation layer and a second Hadamard product module; Among them, the maximum pooling layer, the connection layer, the second activation layer and the second Hadamard product module are connected in sequence, the second Hadamard product module is also connected to the weight distribution module, the input end of the second average pooling layer is connected to the weight distribution module, and the output end of the second average pooling layer is connected to the connection layer.

11. The AGV fork-picking method of claim 10, wherein: The output of the first spatial attention module is expressed as follows: Among them, X represents the input features.

12. The AGV fork-picking method of claim 2, characterized in that: The step of obtaining the predetermined target according to the target recognition result and the first depth image comprises: Determining the predetermined target according to pixel coordinates, an intrinsic parameter matrix of the depth camera, and a depth value in the first depth image; Among them, the recognition result includes the pixel point coordinates of the predetermined target, the position coordinates of the predetermined target are in a predetermined relationship with the pixel point coordinates, the parameters in the intrinsic parameter matrix and the depth value, and the parameters in the intrinsic parameter matrix include the horizontal coordinate of the focal length, the vertical coordinate of the focal length, the horizontal coordinate of the optical center and the vertical coordinate of the optical center.

13. The AGV fork-picking method of claim 12, wherein: The position coordinates of the predetermined target are in a predetermined relationship with the pixel point coordinates, the parameters in the intrinsic parameter matrix and the depth value, specifically: The horizontal coordinate of the position coordinate is in a predetermined relationship with the vertical coordinate of the position coordinate, the horizontal coordinate of the pixel point coordinate, the horizontal coordinate of the optical center and the horizontal coordinate of the focal length; The ordinate of the position coordinate is in a predetermined relationship with the vertical coordinate of the position coordinate, the ordinate of the pixel point coordinate, the ordinate of the optical center and the ordinate of the focal length; The vertical coordinate of the position coordinate is in a predetermined relationship with the depth value.

14. The AGV fork-picking method of claim 13, wherein: The position coordinates of the predetermined target are in a predetermined relationship with the pixel point coordinates, the parameters in the intrinsic parameter matrix and the depth value, specifically: The horizontal coordinate of the position coordinate and the vertical coordinate of the position coordinate, the difference between the horizontal coordinate of the pixel point coordinate and the horizontal coordinate of the optical center and the quotient of the horizontal coordinate of the focal length form a predetermined relationship; The ordinate of the position coordinate and the vertical coordinate of the position coordinate, the difference between the ordinate of the pixel point coordinate and the ordinate of the optical center and the quotient of the ordinate of the focal length form a predetermined relationship; The vertical coordinate of the position coordinate is equal to the depth value.

15. The AGV fork-picking method of claim 12, wherein: The abscissa of the position coordinates is expressed by the following formula: X=Z(u-c x ) / f x The ordinate of the position coordinate is expressed by the following formula: Y=Z(v-c y ) / f y The vertical coordinate of the position coordinate is expressed by the following formula: Z=d Wherein, X represents the horizontal coordinate of the position coordinate, Y represents the vertical coordinate of the position coordinate, Z represents the vertical coordinate of the position coordinate, u represents the horizontal coordinate of the pixel point coordinate, v represents the vertical coordinate of the pixel point coordinate, c represents the horizontal coordinate of the pixel point coordinate, x represents the abscissa of the optical center, c y represents the ordinate of the optical center, f x represents the focal length abscissa, f y represents the focal length ordinate, and d represents the depth value.

16. The AGV fork-picking method of claim 1, characterized in that: The step of determining the target pallet to be forked according to the target action and the pallet area comprises: According to the target action, determining the action direction; When the action is directed through the pallet area, the action is directed to the pallet that overlaps with the pallet area as the target pallet to be forked.

17. The AGV fork-picking method of claim 1, characterized in that: After identifying the predetermined target according to the RGB image and the depth image, the method further includes: maintaining a predetermined distance from the predetermined target and following the predetermined target to reach a designated location; At the designated position, the depth camera acquires the second RGB image and the second depth image.

18. The AGV fork-picking method of claim 1, characterized in that: After the step of determining the target pallet to be forked according to the target action and the pallet area, the method further includes: The AGV moves to the target pallet position to fork the goods.

19. The AGV fork-picking method of claim 1, characterized in that: The target action is a gesture.

20. An AGV, characterized in that: The AGV is provided with a depth camera, and the AGV comprises: An image acquisition module, used to acquire a first RGB image and a first depth image acquired by the depth camera; A target detection module, configured to identify a predetermined target based on the first RGB image and the first depth image; The image acquisition module is further used to acquire a second RGB image and a second depth image captured by the depth camera; an identification module, configured to identify a target action and a pallet area of ​​the predetermined target, respectively, according to the second RGB image and the second depth image; The pallet determination module is used to determine the target pallet to be forked according to the target action and the pallet area.

Images