A fork AGV target pose detection method based on two-dimensional and three-dimensional imaging

Abstract

The application discloses a fork AGV target pose detection method based on two-dimensional and three-dimensional imaging, which is divided into training and inference detection. The training comprises the following steps: collecting two-dimensional images of all object containers in the scene; performing image labeling on the collected two-dimensional images to complete dataset construction; building a deep convolutional neural network model, initializing model parameters, and using the dataset and optimization algorithm to train and optimize the model. The inference detection comprises the following steps: collecting two-dimensional and three-dimensional images of the target; detecting the object container through the deployed target detection model; fusing the two-dimensional and three-dimensional images to complete target pose information detection through three-dimensional image noise reduction, point cloud plane mapping, target pose calculation and other steps; and sending the detected information to the fork AGV control system. The method is based on a deep convolutional neural network and three-dimensional point cloud plane mapping, realizes the fork taking of different types and different specifications of object containers by the fork AGV, and avoids the occurrence of risks such as collision and overturning.

Description

Technical Field

[0001] This invention relates to the fields of intelligent logistics and robot vision recognition, specifically to a method for detecting the pose of a fork-type AGV based on two-dimensional and three-dimensional imaging. Background Technology

[0002] Trackless forklift AGVs have no fixed route restrictions, boast large carrying capacity, flexible movement, high intelligence, and wide application scenarios, making them commonly used equipment in intelligent logistics and unmanned warehouses. However, the global positioning of trackless forklift AGVs suffers from random and cumulative errors, which can impact the realization of automated handling. Especially in scenarios where forklift AGVs are used in conjunction with manual forklifts, the forklifts' positioning, angle, and spacing of the containers deviate from the AGV system's planned map, making it impossible for the forklift AGV to ensure the forklift forks align with the picking holes on the containers. Furthermore, forklift AGVs cannot autonomously identify the type and size of containers; in current applications, a single forklift AGV typically only handles containers of the same type and size. Even so, because the AGV cannot detect the position and orientation of the containers and goods, nor can it assess the surrounding environment, there is a risk of collision.

[0003] Existing solutions for target pose detection in forklift AGVs only identify the same type of cargo container, determining partial pose information of the container through its edge contour. For example, patent CN113267180A, a method for AGV forklift pallet positioning and picking based on 3D depth vision, uses morphological templates to identify pallets; patent CN114820391B, a warehouse pallet detection and positioning method and system based on point cloud processing, uses point cloud template matching to identify and locate pallets. In complex environments, this morphological recognition method is prone to misidentification. With the development of artificial intelligence technology, solutions using deep learning algorithms to assist in cargo container identification have emerged, such as patent CN113537096A, a method and system for AGV forklift parking pallet recognition and assisted positioning based on ROS; and patent CN112017240A, a pallet recognition and positioning method for unmanned forklifts. However, these solutions only identify one type of cargo container—the pallet—and cannot be applied to scenarios where different types and sizes of cargo containers are used simultaneously. They also only detect part of the cargo container's pose, without detecting the pose of the AGV fork teeth, the cargo, or the surrounding environment, thus failing to meet the needs of actual use scenarios for forklift AGVs. Summary of the Invention

[0004] The purpose of this invention is to address existing problems by providing a target pose detection method for forklift AGVs based on two-dimensional and three-dimensional imaging. This method involves acquiring two-dimensional and three-dimensional image data of the target, detecting the pose information of objects such as the AGV forks, the target container and goods, adjacent containers and goods, and shelves, and then sending the detected pose information to the forklift AGV. Based on the detection results, the AGV can determine whether the conditions for picking up the goods are met and adjust its posture, fork position, and spacing to pick up containers of different types and sizes, avoiding risks such as collisions and tipping over.

[0005] The technical solution of the present invention is as follows:

[0006] A target pose detection method for a fork-type AGV based on 2D and 3D imaging is proposed. This method utilizes a deep convolutional neural network to train a target detection model for the cargo container and achieves planar mapping of the 3D point cloud based on the 3D and 2D coordinate mapping relationship. The method includes the following steps:

[0007] Collect two-dimensional images of all containers that need to be identified in the scene from multiple angles and under various ambient light intensities, annotate the two-dimensional images, and construct a dataset;

[0008] Build a deep convolutional neural network object detection model, initialize model parameters, and train and optimize the model using datasets and optimization algorithms;

[0009] An imaging device composed of two-dimensional and three-dimensional imaging devices is used to acquire two-dimensional and three-dimensional images of the target, and the target detection model of the deployed cargo container is used for detection.

[0010] When using the target detection model, if the target container exists, the two-dimensional and three-dimensional images are fused to detect the target pose information; if the target container does not exist, the two-dimensional and three-dimensional images are no longer fused to detect the target pose information.

[0011] The detection of target pose information by fusing two-dimensional and three-dimensional images is accomplished through three-dimensional image denoising, two-dimensional and three-dimensional image alignment, point cloud plane mapping, mapping map filtering, target instance segmentation, instance key information extraction, and target pose calculation.

[0012] The detected target pose information is sent to the onboard control system of the forklift AGV via the communication interface. The AGV can determine whether the conditions for picking up the goods are met based on the detection results, and then adjust its own posture, fork position and spacing to pick up different types and sizes of cargo containers, avoiding risks such as collisions and tipping over.

[0013] Furthermore, the target detection model for the cargo container can identify cargo containers of different types and specifications, determine whether a target cargo container exists, and accurately locate its position in a two-dimensional image.

[0014] Furthermore, the point cloud planar mapping will search for the required target edges and surfaces from the target point cloud acquired by the 3D imaging device and map them into a 2D planar mapping point map with self-determined resolution, thereby reducing the amount of point cloud processing data and improving the rate of instance segmentation and key information extraction; including the following steps:

[0015] Using the imaging plane of the imaging device as the reference coordinate plane, based on the center of the target cargo container and the aligned two-dimensional and three-dimensional images, a portion of the three-dimensional image containing the target cargo container, the cargo, and other objects adjacent to the target is selected as the first target detection area.

[0016] Using the relative position of the AGV fork tooth and the imaging device as a reference, a second target detection area containing the AGV fork tooth is selected from the three-dimensional image;

[0017] Obtain point cloud maps P of the regions covered by the first and second target detection areas in the 3D image. The point cloud of the first target detection area is P1, and the point cloud of the second target detection area is P2. P = P1 + P2.

[0018] The initial point cloud planar mapping map corresponding to the target detection area is generated based on the size of the range of P in the X and Y directions, and the resolution is set according to the actual situation.

[0019] Divide P1 and P2 into several subsets. Each subset has the following characteristics: points within the subsets have the same x and y values, but different z values; that is:

[0020] P1={P1 i |P1 i ={(x i ,y i ,z1),(x i ,y i ,z2),…,(x i ,y i ,z p )},i∈[1,m]},

[0021] P2={P2 j |P2 j ={(x j ,y j ,z1),(x j ,y j ,z2),…,(x j ,y j ,z q)},j∈[1,n]},

[0022] Where m and n are the number of subsets of P1 and P2, respectively, and p and q are the number of points in the subsets;

[0023] Perform the following operation on all subsets of P1: Iterate through P1 i For all points, select the smallest z-value and fill that value into (x). i ,y i In the corresponding coordinates of the point cloud planar mapping;

[0024] Perform the following operation on all subsets of P2: Iterate through P2 i For all points, select the largest z-value and fill it into (x). j ,y j In the corresponding coordinates of the point cloud plane mapping.

[0025] Furthermore, the 3D image denoising involves capturing n consecutive 3D images from a 3D imaging device and processing them using a temporal denoising algorithm to obtain a 3D image with high confidence. The calculation formula is as follows:

[0026] H(x,y,z)=f(G i (x,y,z)),i∈[1,n],

[0027] Among them, G i (x,y,z) is the i-th frame of the 3D image used for noise reduction, H(x,y,z) is the 3D image obtained after noise reduction, and f is the temporal noise reduction algorithm; the temporal noise reduction algorithm includes temporal mean filtering, temporal median filtering, and motion adaptive noise reduction.

[0028] Furthermore, the specific operation of aligning the two-dimensional and three-dimensional images is as follows: based on the extrinsic parameters calibrated by the two-dimensional and three-dimensional imaging devices and the intrinsic parameters of the imaging devices, the centers of the two-dimensional and three-dimensional images are aligned, and the mapping relationship between the points of the two-dimensional image and the points of the three-dimensional image is determined.

[0029] Furthermore, the mapping graph filtering removes noise points introduced by the influence of equipment accuracy and environmental factors, removes outliers caused by occlusion, smooths irregular data density, and restores the true state of the target cargo container. The mapping graph filtering methods include bilateral filtering, statistical filtering, conditional filtering, and random sampling consistency filtering.

[0030] Furthermore, the specific operation of target instance segmentation is as follows: on the filtered point cloud map, the target cargo container, target cargo, AGV fork teeth and other adjacent target objects are segmented by a segmentation algorithm; the segmentation algorithm includes threshold-based segmentation, region-growing segmentation, edge-based segmentation, clustering-based segmentation, and alignment-based two-dimensional and three-dimensional image segmentation.

[0031] Furthermore, the specific operation of extracting the instance key information is as follows: extract the instance key information through the instance key information extraction algorithm; the instance key information includes the center point, corner point, centroid, edge point, center point of the contour data feature region and feature region size data of each instance; the instance key information extraction algorithm includes ISS3D, Harris3D, NARF and SIFT3D.

[0032] Furthermore, the specific operation of the target pose calculation is as follows: based on the target detection model detection results, target instance segmentation results and extracted instance key information, various target pose information is determined and calculated; the target pose information is the relative pose of the target cargo container and its goods, as well as the relative pose between the target cargo container and its goods and the AGV, adjacent cargo containers and goods, and shelf objects.

[0033] Furthermore, the cargo container is a cargo carrier that needs to be picked up in the application scenario of forklift AGV, including cargo pallets, cargo cages, cargo frames and cargo boxes of different specifications, types and materials.

[0034] Compared with existing technologies, the advantages of this invention are:

[0035] A target pose detection method for forklift AGVs based on 2D and 3D imaging is proposed. This method utilizes a deep neural network target detection model to identify and locate various types of cargo containers. Through an improved point cloud plane mapping algorithm, combined with 3D image denoising, 2D and 3D image alignment, mapping filtering, target instance segmentation, instance key information extraction, and target pose information calculation, the method detects the pose of cargo containers, goods, AGV forks, and the surrounding environment. This enables the forklift AGV to autonomously determine picking conditions and adjust its own posture to pick up different types and specifications of cargo containers, avoiding risks such as collisions and overturning. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of a scene structure for a fork-type AGV target pose detection method based on two-dimensional and three-dimensional imaging.

[0037] Figure 2 This is a schematic diagram illustrating the working steps of a fork-type AGV target pose detection method based on two-dimensional and three-dimensional imaging.

[0038] Figure 3 This is a schematic diagram of the detection results of a target container detection model based on a fork-type AGV target pose detection method using two-dimensional and three-dimensional imaging.

[0039] Figure 4 This is a point cloud planar mapping of the target storage pallet, goods, AGV fork teeth, and other adjacent objects in a target pose detection method for fork-type AGVs based on two-dimensional and three-dimensional imaging.

[0040] Figure 5 This is a schematic diagram of key points and size data of each instance in a point cloud planar mapping map, and a schematic diagram of pose calculation, for a target pose detection method for a fork-type AGV based on two-dimensional and three-dimensional imaging.

[0041] Figure 6 This is a schematic diagram illustrating a method for calculating the yaw angle of the target container and the AGV fork teeth in a fork-type AGV target pose detection method based on two-dimensional and three-dimensional imaging.

[0042] Attached label: 1-Goods, 2-Shelf, 3-AGV forklift, 4-RGBD camera, 5-Target storage pallet, 6-Non-target storage pallet. Detailed Implementation

[0043] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0044] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0045] Please see Figure 1-6 A target pose detection method for a fork-type AGV based on 2D and 3D imaging is proposed. This method utilizes a deep convolutional neural network to train a target detection model for the cargo container and achieves planar mapping of the 3D point cloud based on the 3D and 2D coordinate mapping relationship. The method includes the following steps:

[0046] Collect two-dimensional images of all containers that need to be identified in the scene from multiple angles and under various ambient light intensities, annotate the two-dimensional images, and construct a dataset;

[0047] Build a deep convolutional neural network object detection model, initialize model parameters, and train and optimize the model using datasets and optimization algorithms;

[0048] An imaging device composed of two-dimensional and three-dimensional imaging devices is used to acquire two-dimensional and three-dimensional images of the target, and the target detection model of the deployed cargo container is used for detection.

[0049] When using the target detection model, if the target container exists, the two-dimensional and three-dimensional images are fused to detect the target pose information; if the target container does not exist, the two-dimensional and three-dimensional images are no longer fused to detect the target pose information.

[0050] The detection of target pose information by fusing two-dimensional and three-dimensional images is accomplished through three-dimensional image denoising, two-dimensional and three-dimensional image alignment, point cloud plane mapping, mapping map filtering, target instance segmentation, instance key information extraction, and target pose calculation.

[0051] The detected target pose information is sent to the onboard control system of the forklift AGV via the communication interface. The AGV can determine whether the conditions for picking up the goods are met based on the detection results, and then adjust its own posture, fork position and spacing to pick up different types and sizes of cargo containers, avoiding risks such as collisions and tipping over.

[0052] The target detection model for the cargo container can identify cargo containers of different types and specifications, determine whether the target cargo container exists, and accurately locate its position in the two-dimensional image.

[0053] The point cloud planar mapping method searches for the required target edges and surfaces from the target point cloud acquired by the 3D imaging device and maps them into a 2D planar mapping point map with self-determined resolution, thereby reducing the amount of point cloud processing data and improving the rate of instance segmentation and key information extraction; it includes the following steps:

[0054] Using the imaging plane of the imaging device as the reference coordinate plane, based on the center of the target cargo container and the aligned two-dimensional and three-dimensional images, a portion of the three-dimensional image containing the target cargo container, the cargo, and other objects adjacent to the target is selected as the first target detection area.

[0055] Using the relative position of the AGV fork tooth and the imaging device as a reference, a second target detection area containing the AGV fork tooth is selected from the three-dimensional image;

[0056] Obtain point cloud maps P of the regions covered by the first and second target detection areas in the 3D image. The point cloud of the first target detection area is P1, and the point cloud of the second target detection area is P2. P = P1 + P2.

[0057] The initial point cloud planar mapping map corresponding to the target detection area is generated based on the size of the range of P in the X and Y directions, and the resolution is set according to the actual situation.

[0058] Divide P1 and P2 into several subsets. Each subset has the following characteristics: points within the subsets have the same x and y values, but different z values; that is:

[0059] P1={P1 i |P1 i ={(x i ,y i ,z1),(x i ,y i ,z2),…,(x i ,y i ,z p )},i∈[1,m]},

[0060] P2={P2 j |P2 j ={(x j ,y j ,z1),(x j ,y j ,z2),…,(x j ,y j ,z q )},j∈[1,n]},

[0061] Where m and n are the number of subsets of P1 and P2, respectively, and p and q are the number of points in the subsets;

[0062] Perform the following operation on all subsets of P1: Iterate through P1 i For all points, select the smallest z-value and fill that value into (x). i ,y i In the corresponding coordinates of the point cloud planar mapping;

[0063] Perform the following operation on all subsets of P2: Iterate through P2 i For all points, select the largest z-value and fill it into (x). j ,y j In the corresponding coordinates of the point cloud plane mapping.

[0064] The 3D image denoising method involves capturing n consecutive frames of 3D images from a 3D imaging device and processing them using a temporal denoising algorithm to obtain a 3D image with high confidence. The calculation formula is as follows:

[0065] H(x,y,z)=f(G i (x,y,z)),i∈[1,n],

[0066] Among them, G i (x,y,z) is the i-th frame of the 3D image used for noise reduction, H(x,y,z) is the 3D image obtained after noise reduction, and f is the temporal noise reduction algorithm; the temporal noise reduction algorithm includes temporal mean filtering, temporal median filtering, and motion adaptive noise reduction.

[0067] The specific operation of aligning the two-dimensional and three-dimensional images is as follows: based on the extrinsic parameters calibrated by the two-dimensional and three-dimensional imaging devices and the intrinsic parameters of the imaging devices, the centers of the two-dimensional and three-dimensional images are aligned, and the mapping relationship between the points of the two-dimensional image and the points of the three-dimensional image is determined.

[0068] The mapping graph filtering method removes noise points introduced by the influence of equipment accuracy and environmental factors, removes outliers caused by occlusion, smooths irregular data density, and restores the true state of the target container. The mapping graph filtering method includes bilateral filtering, statistical filtering, conditional filtering, and random sample consistency filtering.

[0069] The specific operation of target instance segmentation is as follows: on the filtered point cloud map, the target cargo container, target cargo, AGV fork teeth and other adjacent target objects are segmented by the segmentation algorithm; the segmentation algorithm includes threshold-based segmentation, region-growing segmentation, edge-based segmentation, clustering segmentation, and alignment-based two-dimensional and three-dimensional image segmentation.

[0070] The specific operation for extracting key information of instances is as follows: extract key information of instances through instance key information extraction algorithms; key information of instances includes the center point, corner points, centroid, edge points, center point of contour data feature region and feature region size data of each instance; instance key information extraction algorithms include ISS3D, Harris3D, NARF and SIFT3D.

[0071] The specific operation of the target pose calculation is as follows: based on the target detection model detection results, target instance segmentation results and extracted instance key information, various target pose information is determined and calculated; the target pose information is the relative pose of the target cargo container and its goods, as well as the relative pose between the target cargo container and its goods and the AGV, adjacent cargo containers and goods, and shelf objects.

[0072] The cargo container is a cargo carrier that needs to be picked up in the application scenario of forklift AGV, including cargo pallets, cargo cages, cargo frames and cargo boxes of different specifications, types and materials.

[0073] In a specific application scenario, a method for detecting the target pose of a forklift AGV based on two-dimensional and three-dimensional imaging is described. The specific scenario involves a forklift AGV picking up and storing containers and goods in an automated warehouse. Figure 1As shown. The two-dimensional and three-dimensional imaging devices used are RGBD cameras, mounted on the lifting mechanism of the forklift AGV. After installation, the cameras were calibrated to unify the camera coordinate system and the AGV coordinate system. The detected containers are storage pallets of various sizes and materials, placed on the automated storage and retrieval system (AS / RS) shelves or on the ground. The communication interface for sending the detected target pose information to the forklift AGV's onboard control system is a serial communication RS232 interface.

[0074] The detected target pose information includes whether the target storage pallet exists, the offset between the front center of the target storage pallet and the AGV center, the yaw angle of the target storage pallet and AGV fork teeth, the offset between the AGV fork tooth center and the target storage pallet fork hole center, whether the target goods are too wide, the height of the target goods, and whether the target storage pallet and goods will collide. Collisions involving the target storage pallet and goods include the following situations: collisions between the target storage pallet and goods and adjacent storage pallets and goods; collisions between the target storage pallet and goods and adjacent shelves; the front face of the target goods exceeding the target storage pallet, causing a collision between the AGV shelf and the target goods during forklift AGV retrieval operations; and the goods stacked on the target storage pallet being too high, causing a collision between the target goods and the upper shelf during forklift AGV retrieval operations.

[0075] The detected target pose information is sent to the onboard control system of the forklift AGV in the form of a message. The message consists of a message header, message content, and CRC checksum. The message header consists of a header byte, message sequence number, slave address, message content length (excluding CRC bytes), and tail byte; the message content consists of the detected target pose information.

[0076] An industrial control computer is used to deploy a target detection model for recognizing and locating warehouse pallets. The training and deployment process of the target detection model for warehouse pallets is as follows: Figure 2 As shown, the process includes the following steps: using an RGBD camera to acquire RGB images of storage pallets from different angles under different lighting intensities at the automated warehouse site; labeling the acquired RGB images to complete the dataset construction; building a suitable deep network model, initializing model parameters, and training and optimizing the model using the dataset and optimization algorithms; and deploying the trained and optimized target detection model of the storage pallets on an industrial control computer.

[0077] The target pose information includes the relative pose of the target container and its cargo, as well as the relative pose between the target container and cargo and objects such as the AGV, adjacent containers and cargo, and shelves. The target pose information includes whether the target container exists, its type, whether the target cargo exists, the outline and dimensions of the target cargo, the deviation between the center of the target container and the center of the AGV, the positional deviation and yaw angle between the target container and the AGV fork teeth, whether the target cargo exceeds width or height limits, whether the target cargo will collide with other adjacent objects, and whether the target cargo has a risk of tipping over. Specifically, the positional deviation between the target container and the AGV fork teeth refers to the positional deviation between the center of the tips of the two fork teeth of the forklift AGV and the center of the corresponding fork hole of the target container.

[0078] The specific operation of the target pose calculation is as follows: based on the target detection model detection results, target instance segmentation results, and extracted instance key information, various target pose information is determined and calculated:

[0079] The presence and type of the target container are determined by the detection results of the target detection model.

[0080] Determine whether the target goods exist based on the target instance segmentation results;

[0081] The outline of the target cargo is obtained through the outline data of the target cargo instance, and the size of the target cargo can be calculated through the edge points of the target cargo instance.

[0082] Calculate the deviation between the center of the target cargo container and the center of the AGV using the center point of the target cargo container instance;

[0083] Calculate the positional deviation between the target cargo container and the AGV fork tooth by using the center point of the fork hole of the target cargo container and the center point of the tip of the AGV fork tooth;

[0084] The angles between the target cargo container and the imaging device plane, and between the AGV fork tooth and the imaging device plane are calculated by the edge point of the target cargo container and the center point of the AGV fork tooth tip. Then, the yaw angle between the target cargo container and the AGV fork tooth is calculated by the angles between the target cargo container and the imaging device plane, and between the AGV fork tooth and the imaging device plane.

[0085] The distance between the edge of the target cargo and the edge of the target container is calculated by using the edge points of the target container and the edge points of the target cargo, and then used as the basis for determining whether the target cargo exceeds the width limit.

[0086] The height of the target cargo is calculated by the edge points of the target cargo, and combined with the actual cargo height limit, it serves as the basis for determining whether the target cargo exceeds the height limit.

[0087] The distances between the target container and cargo and other adjacent objects are calculated by using the edge points of the target container, the target cargo, and other adjacent objects, and then used as a basis for judging whether there is a risk of collision during the forklift operation.

[0088] The shape of the target cargo is calculated by taking its center of gravity, edge points, and outline data, and then used as a basis for judging whether the target cargo has a risk of overturning.

[0089] Work steps are as follows Figure 2 As shown, it includes the following steps:

[0090] S1 starts the industrial control computer and RGBD camera, waiting for the detection trigger command from the forklift AGV vehicle control system.

[0091] When the S2 forklift AGV arrives at the detection point, the onboard control system issues a target pose detection command.

[0092] After receiving the detection command, the S3 industrial computer processes the RGB and depth images captured by the RGBD camera:

[0093] S3.1 uses a deployed pallet target detection model to identify and locate pallets in RGB images, determine the presence of target pallets, and pinpoint their location within the RGB image. Figure 3 As shown;

[0094] S3.2 uses a temporal denoising algorithm to denoise the depth image. The temporal denoising algorithm used in this embodiment is temporal mean filtering, and the calculation formula is as follows:

[0095]

[0096] Among them, G i (x,y,z) is the depth image of the i-th frame used for noise reduction, and H(x,y,z) is the depth image obtained after noise reduction. In this embodiment, n=25.

[0097] Based on the recognition results of the target detection model, if the target storage pallet exists, S4 fuses the RGB image and the denoised depth image to detect the target pose information; if it does not exist, it sends the result to the forklift AGV on-board control system and waits for the AGV's next detection instruction.

[0098] The S5 acquires the intrinsic and extrinsic parameters of the RGB and depth cameras, iterates through the coordinates of each pixel in the temporally denoised depth image, performs rotation and translation transformations using the intrinsic and extrinsic parameters of the two cameras, calculates the coordinates of the pixels in the depth image corresponding to the RGB image, and completes the alignment of the depth image and the RGB image.

[0099] S6 divides the target detection region, performs point cloud planar mapping on the target detection region, and generates a point cloud planar mapping map of the target detection region. The specific steps are as follows:

[0100] S6.1 Based on the pose of the target storage pallet located on the RGB image by the storage pallet target detection model, select a first target detection area on the aligned depth image that includes a portion of the target cargo container and goods as well as other objects adjacent to the target.

[0101] S6.2 Based on the relative position of the AGV fork teeth and the RGBD camera, select the second target detection area containing the AGV fork teeth on the aligned depth image;

[0102] S6.3 converts the first and second target detection regions in the depth image into a point cloud image P. The conversion formula is as follows:

[0103]

[0104] (u,v) represents the coordinates of a point within the target detection area in the aligned depth image, D is the depth value of that point, K is the intrinsic parameter of the depth camera, (x,y,z) is the corresponding coordinate of that point in the point cloud image, and f x f y These are the focal lengths of the depth camera along the X and Y axes, respectively. o ,v o () represents the actual location of the depth camera's master point.

[0105] S6.4 Generates an initial point cloud planar mapping map corresponding to the target detection area based on the size of P in the X and Y directions, with the resolution set to 1 mm / pix.

[0106] S6.5 divides the point cloud P1 of the first target detection area and the point cloud P2 of the second target detection area into several subsets. These subsets have the following characteristics: the coordinates of points in the subsets have the same x and y values, but different z values. That is:

[0107] P1={P1 i |P1 i ={(x i ,y i ,z1),(x i ,y i ,z2),…,(x i ,y i ,z p )}, i∈[1,m]},

[0108] P2={P2 j |P2 j ={(x j ,y j,z1),(x j ,y j ,z2),…,(x j ,y j ,z q )},j∈[1,n]},

[0109] Where m and n are the number of subsets of P1 and P2, respectively, and p and q are the number of points in the subsets.

[0110] S6.6 performs the following operation on all subsets of P1: Iterate through P1 i For all points, select the smallest z-value and fill that value into (x). i ,y i In the corresponding coordinates of the point cloud planar mapping;

[0111] S6.7 performs the following operation on all subsets of P2: Iterate through P2 j For all points, select the largest z-value and fill it into (x). j ,y j In the corresponding coordinates of the point cloud plane mapping.

[0112] S7 uses statistical filtering to remove discrete points from the point cloud planar mapping and then performs small-scale hole filling to restore the true shape of the target detection area.

[0113] S8 segments each target instance in the filtered point cloud planar mapping, such as Figure 4 As shown, the segmentation method is as follows:

[0114] S8.1 Based on the target pallet target detection model, the region coordinates of the target pallet in the RGB image are mapped to the aligned depth image, and then mapped to the point cloud plane mapping map, thereby segmenting the target pallet instance.

[0115] S8.2 Based on the relative positions of the AGV fork teeth and the RGBD camera, a threshold-based segmentation method is used to segment the AGV fork tooth instances;

[0116] S8.3 Based on the target warehouse pallet instance, the target cargo instance is segmented using a region-growing segmentation method;

[0117] S8.4 uses the target storage pallet instance, target cargo instance, and AGV fork instance as a basis, and adopts an edge-based segmentation method to complete the segmentation of other adjacent object instances.

[0118] S9 extracts key information for each target instance in the point cloud planar mapping map. A schematic diagram of the key information is shown below. Figure 5 As shown, it includes the following

[0119] data:

[0120] S9.1 AGV fork teeth: Coordinates and distance values ​​of the center point of the upper surface of the fork tooth tip. Single fork tooth width W;

[0121] S9.2 Target cargo: Coordinates and distance values ​​of cargo edge points Coordinates and distance values ​​of the point closest to the RGBD camera plane

[0122] S9.3 Target Storage Pallet: Coordinates and Distance Values ​​of Pallet Edge Points Tray front face and G f Coordinates and distance values ​​of points with the same x-value Coordinates of the center point of the pallet fork hole area Length l, width w;

[0123] S9.4 Adjacent Objects: Coordinates and distance values ​​of the nearest points of the other objects adjacent to the target storage pallet and goods on both the left and right sides.

[0124] S10 calculates the target pose information using the extracted key information data:

[0125] S10.1 as Figure 6 As shown, the formulas for calculating the offset R between the front center of the target storage pallet and the center of the AGV, and the yaw angle θ of the target storage pallet and the AGV fork are as follows:

[0126]

[0127]

[0128] S10.2 as Figure 5 As shown, the offset R between the center of the fork teeth and the center of the fork hole on the storage pallet l and R r The cargo's overwidth ΔX l ΔX r ,

[0129] The formula for calculating ΔZ is as follows:

[0130]

[0131]

[0132]

[0133] S10.3 as Figure 5As shown, the method for detecting whether cargo will collide is as follows:

[0134] S10.3.1 Calculate the distance between the side of the cargo and the adjacent objects. and the height of the goods

[0135]

[0136] S10.3.2 will D l and D r Compare with the minimum allowed spacing value, if D l Or D r If the distance is less than the minimum allowable distance, there is a risk of collision for the goods;

[0137] S10.3.3 Compare h with the maximum permissible stacking height of goods. If h is greater than the maximum permissible stacking height of goods, there is a risk of collision of goods.

[0138] S11 constructs a message based on the calculated target pose information, in the following order: whether the target storage pallet exists, the offset between the front center of the target storage pallet and the AGV center, the yaw angle between the target storage pallet and the AGV fork teeth, the offset between the center of the AGV fork teeth and the center of the storage pallet fork hole, the cargo's width, the cargo's height, and whether a collision is likely. It then completes the corresponding message header and CRC checksum, and sends the complete message to the AGV onboard control system via the RS232 communication interface.

[0139] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed and specific, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.

Claims

1. A method for target pose detection in a fork-type AGV based on two-dimensional and three-dimensional imaging, characterized in that, The following steps are included: training a target detection model for a container using a deep convolutional neural network and realizing planar mapping of the 3D point cloud based on the mapping relationship between 3D and 2D coordinates: Collect two-dimensional images of all containers that need to be identified in the scene from multiple angles and under various ambient light intensities, annotate the two-dimensional images, and construct a dataset; Build a deep convolutional neural network object detection model, initialize model parameters, and train and optimize the model using datasets and optimization algorithms; An imaging device composed of two-dimensional and three-dimensional imaging devices is used to acquire two-dimensional and three-dimensional images of the target, and the target detection model of the deployed cargo container is used for detection. When using the target detection model, if the target container exists, the two-dimensional and three-dimensional images are fused to detect the target pose information; if the target container does not exist, the two-dimensional and three-dimensional images are no longer fused to detect the target pose information. The detection of target pose information by fusing two-dimensional and three-dimensional images is accomplished through three-dimensional image denoising, two-dimensional and three-dimensional image alignment, point cloud plane mapping, mapping map filtering, target instance segmentation, instance key information extraction, and target pose calculation. The detected target pose information is sent to the onboard control system of the forklift AGV via the communication interface; The point cloud planar mapping process involves searching for the desired target edges and surfaces from the target point cloud acquired by the 3D imaging device and mapping them into a 2D planar mapping point map with a self-defined resolution; it includes the following steps: Using the imaging plane of the imaging device as the reference coordinate plane, based on the center of the target cargo container and the aligned two-dimensional and three-dimensional images, a portion of the three-dimensional image containing the target cargo container, the cargo, and other objects adjacent to the target is selected as the first target detection area. Using the relative position of the AGV fork tooth and the imaging device as a reference, a second target detection area containing the AGV fork tooth is selected from the three-dimensional image; Obtain point cloud maps of the areas covered by the first and second target detection regions in the 3D image. The point cloud of the first target detection area is The point cloud of the second target detection area is , ; according to exist and The area size in the direction generates the initial point cloud planar mapping map corresponding to the target detection area, and the resolution is set according to the actual situation; Will , Divide into several subsets, and the subsets have the following characteristics: the coordinates of the points in the subsets are the same. Value and Value, Different Value; that is: , , in and They are , The number of subsets and It is the number of points in the subset; right Perform the following operation on all subsets: traverse Of all points, select the smallest one. Value, fill in this value. In the corresponding coordinates of the point cloud planar mapping; right Perform the following operation on all subsets: traverse Of all the points, select the largest one. Value, fill in this value. In the corresponding coordinates of the point cloud planar mapping.

2. The method for target pose detection of a fork-type AGV based on two-dimensional and three-dimensional imaging according to claim 1, characterized in that, The target detection model for the cargo container can identify cargo containers of different types and specifications, determine whether the target cargo container exists, and accurately locate its position in the two-dimensional image.

3. The method for target pose detection of a fork-type AGV based on two-dimensional and three-dimensional imaging according to claim 1, characterized in that, The 3D image denoising method involves capturing n consecutive frames of 3D images from a 3D imaging device and processing them using a temporal denoising algorithm to obtain a 3D image with high confidence. The calculation formula is as follows: in, For noise reduction Frame 3D image, The 3D image obtained after noise reduction processing. This is a time-domain denoising algorithm; the time-domain denoising algorithm includes time-domain mean filtering, time-domain median filtering, and motion adaptive denoising.

4. The target pose detection method for a fork-type AGV based on two-dimensional and three-dimensional imaging according to claim 1, characterized in that, The specific operation of aligning the two-dimensional and three-dimensional images is as follows: based on the extrinsic parameters calibrated by the two-dimensional and three-dimensional imaging devices and the intrinsic parameters of the imaging devices, the centers of the two-dimensional and three-dimensional images are aligned, and the mapping relationship between the points of the two-dimensional image and the points of the three-dimensional image is determined.

5. The method for target pose detection of a fork-type AGV based on two-dimensional and three-dimensional imaging according to claim 1, characterized in that, The mapping graph filtering method removes noise points introduced by the influence of equipment accuracy and environmental factors, removes outliers caused by occlusion, smooths irregular data density, and restores the true state of the target container. The mapping graph filtering method includes bilateral filtering, statistical filtering, conditional filtering, and random sample consistency filtering.

6. The method for target pose detection of a fork-type AGV based on two-dimensional and three-dimensional imaging according to claim 1, characterized in that, The specific operation of target instance segmentation is as follows: on the filtered point cloud map, the target cargo container, target cargo, AGV fork teeth and other adjacent target objects are segmented by the segmentation algorithm; the segmentation algorithm includes threshold-based segmentation, region-growing segmentation, edge-based segmentation, clustering segmentation, and alignment-based two-dimensional and three-dimensional image segmentation.

7. The target pose detection method for a fork-type AGV based on two-dimensional and three-dimensional imaging according to claim 1, characterized in that, The specific operation for extracting key information of instances is as follows: extract key information of instances through instance key information extraction algorithms; key information of instances includes the center point, corner points, centroid, edge points, center point of contour data feature region and feature region size data of each instance; instance key information extraction algorithms include ISS3D, Harris3D, NARF and SIFT3D.

8. The method for target pose detection of a fork-type AGV based on two-dimensional and three-dimensional imaging according to claim 1, characterized in that, The specific operation of the target pose calculation is as follows: based on the target detection model detection results, target instance segmentation results and extracted instance key information, various target pose information is determined and calculated; the target pose information is the relative pose of the target cargo container and its goods, as well as the relative pose between the target cargo container and its goods and the AGV, adjacent cargo containers and goods, and shelf objects.

9. The method for target pose detection of a fork-type AGV based on two-dimensional and three-dimensional imaging according to claim 1, characterized in that, The cargo container is a cargo carrier that needs to be picked up in the application scenario of forklift AGV, including cargo pallets, cargo cages, cargo frames and cargo boxes of different specifications, types and materials.

Images