Tray pose recognition method, controller and forklift

Abstract

The embodiment of the invention provides a tray pose recognition method, a controller and a forklift, and relates to the technical field of image recognition. The tray pose recognition method comprises the steps that when a forklift is located in front of a to-be-forked tray, a sensing image and a first point cloud in front of a fork arm of the forklift are acquired; based on an image detection result obtained by detecting the sensing image, obtaining a tray image of the tray in the sensing image and a tray point cloud of the tray in the first point cloud; fusing the tray image and the tray point cloud to obtain a target point cloud of the tray; and determining pose information of the tray based on the target point cloud of the tray. According to the method, the tray pose recognition precision is improved, the tray forking success rate can be increased through the more accurate tray pose, manual intervention is reduced, and the overall carrying efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a pallet posture recognition method, a controller and a forklift. Background Art

[0002] With the development of intelligent driving technology, unmanned forklifts have also been applied in the field of industrial automation. In modern warehousing and logistics, the use of unmanned forklifts has greatly improved the efficiency of cargo handling and reduced labor costs.

[0003] Unmanned forklifts usually carry goods by picking up pallets with forks. During the transportation process, the sensors installed on the forklift collect images and other data to identify the position of the pallet, based on which the forklift is controlled to complete the automatic transportation. It can be seen that pallet position recognition is related to the automatic transportation efficiency of the forklift; however, the traditional pallet position recognition through sensor collection of images and other data is easily affected by environmental interference such as lighting changes and occlusions, which will seriously affect the accuracy of pallet position recognition and cannot cope with complex and changing working environments. Summary of the invention

[0004] The purpose of the present invention is to provide a pallet posture recognition method, a controller and a forklift. When the forklift is located in front of a pallet to be forked, a sensor image and a first point cloud in front of the fork arm of the forklift are first obtained, and the sensor image and the first point cloud both contain the pallet. The sensor image is then detected to obtain an image detection result, and based on the image detection result, a pallet image corresponding to the pallet in the sensor image and a pallet point cloud corresponding to the first point cloud and the pallet are obtained. The sensor image and the pallet point cloud are thereby fused to obtain a target point cloud of the pallet. The target point cloud simultaneously contains features of the image and point cloud of the pallet. Therefore, the posture information of the pallet subsequently determined based on the target point cloud of the pallet is more accurate. A more accurate pallet posture can improve the success rate of pallet forking, reduce manual intervention, and improve overall handling efficiency.

[0005] To achieve the above-mentioned objectives, the present invention provides a pallet posture recognition method, comprising: when a forklift is located in front of a pallet to be forked, obtaining a sensor image and a first point cloud in front of a fork arm of the forklift; based on an image detection result obtained by detecting the sensor image, obtaining a pallet image of the pallet in the sensor image and a pallet point cloud of the pallet in the first point cloud; fusing the pallet image with the pallet point cloud to obtain a target point cloud of the pallet; and determining the posture information of the pallet based on the target point cloud of the pallet.

[0006] The present invention also provides a controller for executing the above-mentioned pallet posture recognition method.

[0007] The present invention also provides a forklift, comprising: a forklift body, a data acquisition device and the above-mentioned controller; the data acquisition device is mounted on the forklift body, and is used to collect sensor images and point clouds in front of the fork arm of the forklift and send them to the controller.

[0008] The present invention also provides a computer-readable storage medium, which is a non-volatile storage medium or a non-transient storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the tray posture recognition method as described above are executed.

[0009] In one embodiment, the image detection result includes the pallet type of the pallet; and based on the target point cloud of the pallet, determining the position information of the pallet includes:

[0010] Based on the target point cloud of the pallet, obtaining preliminary position information of the pallet;

[0011] Based on the preliminary pose information of the pallet, the target point cloud and the pallet template point cloud corresponding to the pallet type of the pallet, the final pose information of the pallet is obtained.

[0012] In one embodiment, obtaining the final posture information of the pallet based on the preliminary posture information of the pallet, the target point cloud, and the pallet template point cloud corresponding to the pallet type of the pallet includes:

[0013] Registering the pallet template point cloud with the target point cloud to obtain conversion information of the pallet template point cloud to the coordinate system of the target point cloud;

[0014] Based on the conversion information and the preliminary posture information, final posture information of the tray is obtained.

[0015] In one embodiment, obtaining preliminary position information of the pallet based on the target point cloud of the pallet includes:

[0016] Selecting a plurality of feature points located within a set feature area of ​​the pallet from the target point cloud;

[0017] Calculating the center point of the plurality of feature points as the reference center point of the pallet;

[0018] Obtaining a vertical vector of the pallet toward a target plane of the forklift based on the coordinates of the plurality of feature points;

[0019] Determining a reference heading angle of the target plane relative to the forklift based on a vertical vector of the target plane;

[0020] The preliminary position information includes the reference center point and the reference heading angle.

[0021] In one embodiment, obtaining a vertical vector of the pallet toward a target plane of the forklift based on the coordinates of the plurality of feature points includes:

[0022] The plurality of feature points are combined into a feature data matrix, and the coordinate values ​​of each dimension contained in the feature data matrix are standardized;

[0023] The covariance matrix of the characteristic data matrix after standardization is obtained, and the eigenvector corresponding to the maximum eigenvalue among all eigenvalues ​​of the covariance matrix is ​​used as the vertical vector.

[0024] In one embodiment, the image detection result includes: field of view range information when the sensor image is collected; the method of obtaining the pallet point cloud of the pallet in the first point cloud is:

[0025] A second point cloud segmented from the first point cloud based on the field of view range information;

[0026] A pallet point cloud of the pallet is acquired from the second point cloud.

[0027] In one embodiment, fusing the pallet image with the pallet point cloud to obtain a target point cloud of the pallet includes:

[0028] Acquire an image weight of the sensor image and a point cloud weight of the first point cloud;

[0029] Based on the image weight and the point cloud weight, the coordinates of each point in the pallet image and the coordinates of each point in the pallet point cloud are fused to obtain the target point cloud of the pallet.

[0030] In one embodiment, obtaining the image weight of the sensor image and the point cloud weight of the first point cloud includes:

[0031] Performing quality assessment on the sensor image and the first point cloud respectively to obtain image quality parameters of the sensor image and point cloud quality parameters of the first point cloud;

[0032] Based on the image quality parameter and the point cloud quality parameter, an image weight of the sensor image and a point cloud weight of the first point cloud are obtained.

[0033] In one embodiment, after obtaining the image weight of the sensor image and the point cloud weight of the first point cloud, the method further includes:

[0034] Adjusting the image weight and the point cloud weight based on the light intensity value of the scene currently located by the forklift and / or the integrity assessment value of the first point cloud;

[0035] Based on the image weight and the point cloud weight, the coordinates of each point in the pallet image are fused with the coordinates of each point in the pallet point cloud to obtain the target point cloud of the pallet, including:

[0036] Based on the adjusted image weight and the point cloud weight, the coordinates of each point in the pallet image and the coordinates of each point in the pallet point cloud are fused to obtain the target point cloud of the pallet.

[0037] In one embodiment, the quality assessment method of the sensor image is:

[0038] Acquire at least one image quality evaluation value of the sensor image, wherein the image quality evaluation value is any one of the following: a clarity evaluation value, an exposure evaluation value, and a motion blur evaluation value;

[0039] Based on the at least one image quality evaluation value of the sensor image, an image quality parameter of the sensor image is obtained.

[0040] In one embodiment, the quality assessment method of the first point cloud is:

[0041] Acquire at least one point cloud quality assessment value of the first point cloud, the point cloud quality assessment value being any one of the following: a point cloud density assessment value, a point cloud integrity assessment value, and a noise level assessment value;

[0042] Based on the at least one point cloud quality assessment value of the first point cloud, a point cloud quality parameter of the first point cloud is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a specific flow chart of the pallet posture recognition method according to the first embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of a sensor image after detection according to the first embodiment of the present invention;

[0045] Figure 3 yes Figure 1 A specific flow chart of step 104 of the pallet posture recognition method;

[0046] Figure 4 is a schematic diagram of the registration of the pallet template point cloud and the target point cloud according to the first embodiment of the present invention;

[0047] Figure 5 4 is a specific flow chart of the pallet posture recognition method according to the second embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be described in detail with reference to the accompanying drawings to provide a clearer understanding of the purpose, features and advantages of the present invention. It should be understood that the embodiments shown in the accompanying drawings are not intended to limit the scope of the present invention, but are only intended to illustrate the essential spirit of the technical solution of the present invention.

[0049] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the relevant art will recognize that the embodiments may be practiced without one or more of these specific details. In other cases, well-known devices, structures, and techniques associated with the present application may not be shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0050] Unless the context requires otherwise, throughout the specification and claims, the word "comprise" and variations such as "include" and "have" should be construed in an open, inclusive sense, ie, should be interpreted as "including, but not limited to."

[0051] References throughout the specification to "one embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of "in one embodiment" or "in an embodiment" in various places throughout the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any manner in one or more embodiments.

[0052] As used in this specification and the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. It should be noted that the term "or" is generally employed in its sense including "or / and" unless the context clearly dictates otherwise.

[0053] In the following description, in order to clearly show the structure and working mode of the present invention, many directional words will be used for description, but the words "front", "back", "left", "right", "outside", "inside", "outward", "inward", "up", "down", etc. should be understood as convenient terms and should not be understood as restrictive terms.

[0054] A first embodiment of the present invention relates to a pallet posture recognition method, which is applied to a forklift. The forklift is equipped with a controller, which can be a controller dedicated to pallet posture recognition or a main controller in the forklift.

[0055] The forklift body of the forklift is equipped with a data acquisition device, which is used to collect images and point cloud data in front of the forklift's fork arm and send them to the controller. The data acquisition device may include: a radar sensor (such as a laser radar) and an image sensor (such as a camera, a video camera); the radar sensor is installed on the front of the forklift body to obtain point cloud data in front of the forklift's fork arm; the image sensor is installed on the forklift's fork arm (for example, installed at the fork arm fork tip) to collect images in front of the forklift's fork arm. It should be noted that this application only describes in detail the part of the forklift's pallet posture recognition, and does not describe in detail the working principle of the forklift itself. You can refer to the existing forklifts and will not repeat them here.

[0056] The specific process of the pallet posture recognition method in this embodiment is as follows: Figure 1 shown.

[0057] Step 101 , when a forklift is located in front of a pallet to be forked, a sensor image and a first point cloud in front of a fork arm of the forklift are obtained.

[0058] Specifically, the forklift can move toward the location of the goods to be transported based on the received control command or autonomously. During the movement of the forklift, the forklift can determine whether it has reached the goods to be transported by its relative position with the goods to be transported and in combination with the collected images. The goods to be transported are placed on the pallet. After the forklift moves to the goods to be transported, that is, the forklift is located in front of the pallet to be forked (specifically, the forklift fork arm is facing the pallet to be forked), at this time, the image sensor is used to collect the sensor image of the forklift's fork arm including the pallet, and the radar sensor is used to collect the first point cloud of the forklift's fork arm including the pallet.

[0059] In some embodiments, the first point cloud may be obtained by fusing several frames of radar point clouds collected by a radar sensor. Furthermore, the fused point cloud may be first subjected to ground removal processing, and the point cloud may be filtered according to the ground range; and the filtered point cloud may be subjected to adaptive voxel filtering, wherein the voxel size is set according to the local point cloud density, with smaller voxels used in high-density point cloud areas and larger voxels used in low-density point cloud areas; in each voxel, the centroid coordinates of the points within the voxel are calculated, and the centroid is used to represent all the original points within the voxel, thereby reducing the amount of point cloud data while retaining the point cloud features; the point cloud obtained after ground removal processing and adaptive voxel filtering is then used as the final first point cloud, thereby achieving a denoising effect.

[0060] Step 102 : Based on the image detection result obtained by detecting the sensor image, a pallet image of the pallet in the sensor image and a pallet point cloud of the pallet in the first point cloud are acquired.

[0061] Specifically, the sensor image captured by the image sensor can be processed first, such as performing image dedistortion, lighting balance, and size normalization on the sensor image. The processed sensor image is then detected, and the detection content includes but is not limited to: pallet type detection, pallet overall detection, and pallet leg position identification, thereby obtaining an image detection result of the sensor image, and the image detection result includes: pallet type, position information of the entire pallet in the sensor image, and position information of the pallet legs in the sensor image. The position information of the entire pallet may include the coordinate range of the entire pallet in the sensor image, and the position information of the pallet legs may include the coordinate range of the pallet legs in the sensor image; then, based on the position information of the entire pallet in the sensor image indicated in the image detection result obtained after the sensor image is detected, a pallet image containing the pallet in the sensor image is obtained, and the pallet image shows the area of ​​the surface of the pallet in the sensor image that faces the forklift. As Figure 2 As shown, the entire area of ​​the pallet and the area of ​​the pallet legs are marked after the sensor image is detected.

[0062] Exemplarily, image detection can be achieved by a preset image recognition algorithm or a pallet detection network model trained specifically for pallet detection; wherein the pallet detection network model can be a network architecture using a YOLO network architecture, such as a network architecture of YOLOv5S; the collected training data of the pallet detection network model forms data in the form of (X, Y), where X represents the input image, and Y represents the output image detection result. The input image X can cover images of different scenes, such as: different lighting conditions (such as sunny days, cloudy days, and indoor photos, etc., which can be represented by the light intensity of the environment), different angles of the forklift relative to the pallet (1 to 180 degrees, the angle interval can be selected as needed, such as 15 degrees, 10 degrees, 5 degrees, etc.), different distances of the forklift from the pallet (for example, 0.5m to 3m), and different degrees of occlusion of the pallet (occlusion rate 20% to 50%), and different types of pallets (such as field pallets, Sichuan pallets, wooden pallets, plastic pallets, European standard pallets); the image detection result Y can include: pallet type, position information of the pallet as a whole in the image, and position information of the pallet legs in the image. By training the pallet detection network model using the collected training data, a pallet detection network model for pallet detection can be obtained; the specific model training process can refer to the existing technology and will not be repeated here.

[0063] In the detection of the sensor image, the FOV (Field of View) range of the image sensor is also calculated; the FOV range of the image sensor is calculated as follows: based on the overall position information of the pallet obtained after the sensor image is detected and the internal parameters of the image sensor, the FOV range of the image sensor can be calculated, specifically: based on the overall position information of the pallet in the image, the actual size of the pallet, and the focal length of the image sensor, the FOV range of the image sensor is calculated using geometric relationships or existing image processing algorithms; for example, based on the relationship between the imaging size of the surface of the pallet facing the forklift in the image and the focal length of the camera, combined with the known actual size of the surface of the pallet facing the forklift (which can be obtained based on the identified pallet type), the horizontal and vertical FOV angles or ranges of the camera and other parameters are calculated using methods such as the principle of similar triangles, thereby obtaining the FOV range of the image sensor.

[0064] Based on the external parameter calibration of the image sensor and the external parameter calibration of the radar sensor, the first point cloud is transformed into the coordinate system of the image sensor, and then the first point cloud in the image sensor coordinate system is segmented using the FOV range of the image sensor to obtain the second point cloud, which is in the same FOV range as the sensor image; then the second point cloud is clustered, and the largest cluster is selected, and the largest cluster is the pallet point cloud in the second point cloud. Among them, before clustering after obtaining the second point cloud, the second point cloud can also be statistically filtered to filter out the noise in the second point cloud, and then the filtered second point cloud is clustered, and the largest cluster is selected to obtain the pallet point cloud.

[0065] Step 103: Fusing the pallet image with the pallet point cloud to obtain a target point cloud of the pallet.

[0066] Specifically, the coordinates of each point on the pallet in the pallet image are fused with the coordinates of the corresponding points in the pallet point cloud to obtain the coordinates of each point on the pallet. The coordinates of each point on the pallet calculated here form a target point cloud of the pallet, which integrates the useful features carried in the sensor image collected by the image sensor (such as auxiliary information for pallet type identification) and the precise spatial position information contained in the first point cloud collected by the lidar (even when the light is weak). The fusion calculation is, for example, to find the mean of the two coordinates, or to select one of the coordinates, or to assign different weights to the coordinates of the points on the pallet in the pallet image and the coordinates of the points on the pallet in the pallet point cloud in a weighted manner, and then calculate the coordinates of each point on the pallet.

[0067] Step 104 : determining the position and posture information of the pallet based on the target point cloud of the pallet.

[0068] Specifically, the target point cloud of the pallet can be understood as representing the target plane of the pallet facing the forklift, and then based on the target plane, the position information of the pallet can be obtained. The position information may include the coordinates of the center point of the pallet and the heading angle of the pallet relative to the forklift.

[0069] In some examples, the target point cloud of the pallet may have some deviations due to the fusion of image and point cloud information, so it can be further optimized to obtain more accurate pallet posture information. The specific method is as follows:

[0070] Please refer to Figure 3 , step 104 includes the following sub-steps:

[0071] Sub-step 1041 , based on the target point cloud of the pallet, obtaining preliminary position and posture information of the pallet.

[0072] Specifically, the target point cloud obtained by coordinate fusion can be used to analyze and obtain the preliminary pose information of the pallet, and the preliminary pose information can be used as reference information for subsequent calculation to obtain accurate pallet pose information. The specific method of obtaining the preliminary pose information of the pallet is as follows:

[0073] Select multiple feature points located in the set feature area of ​​the pallet from the target point cloud; that is, select the set feature area from the pallet to represent the entire pallet. The set feature area is, for example, the pallet leg. The pallet leg of the pallet is generally also centrally symmetrical, so all feature points on the pallet leg are selected from the target point cloud. Then calculate the center point of the multiple feature points as the reference center point of the pallet.

[0074] Based on the coordinates of multiple feature points, the vertical vector of the target plane of the pallet toward the forklift is obtained. Specifically, the coordinates of multiple feature points are combined into a feature data matrix, and the coordinate values ​​of each dimension contained in the feature data matrix are standardized, that is, the coordinates of each feature point are standardized, and the mean and standard deviation of the coordinates of all feature points on the horizontal axis, vertical axis, and vertical axis are calculated respectively, and then the coordinates of each feature point are converted into standardized coordinates with a mean of 0 and a standard deviation of 1, thereby obtaining a standardized feature data matrix. Subsequently, the covariance matrix of the standardized feature data matrix is ​​obtained. Assuming that the standardized feature data matrix is ​​A, which includes the coordinates of n feature points that have been standardized, the feature data matrix is ​​the covariance matrix B of A = (1 / n-1)A T A.

[0075] Solve the eigenvalues ​​of the covariance matrix B and the eigenvectors corresponding to each eigenvalue, sort all the eigenvalues ​​from large to small, and use the eigenvector corresponding to the largest eigenvalue among all the eigenvalues ​​of the covariance matrix B as the vertical vector of the pallet toward the target plane of the forklift.

[0076] The target plane is in the forklift coordinate system, so the reference heading angle of the target plane relative to the forklift can be determined based on the vertical vector of the target plane. The angle between the vertical vector of the target plane and the positive direction of the horizontal axis (x-axis) of the forklift coordinate system is the reference heading angle of the target plane relative to the forklift; the reference center point and the reference heading angle calculated above constitute the preliminary posture information of the pallet.

[0077] Sub-step 1042 , based on the preliminary posture information of the pallet, the target point cloud, and the pallet template point cloud corresponding to the pallet type of the pallet, obtain the final posture information of the pallet.

[0078] Specifically, the pallet template point cloud and the target point cloud are registered to obtain the conversion information of the pallet template point cloud to the coordinate system of the target point cloud. For example: the ICP (Iterative Closest Point) nearest point iteration algorithm is used to register the pallet template point cloud and the target point cloud. First, the corresponding point pairs between the pallet template point cloud and the target point cloud are obtained, and a rotation and translation matrix is ​​constructed based on the corresponding point pairs. Then, the pallet template point cloud is transformed to the coordinate system of the target point cloud using the constructed rotation and translation matrix, and the error function between the transformed pallet template point cloud and the target point cloud is calculated. If the value of the error function is greater than the preset error threshold, the above iterative operation is continued, the corresponding point pairs between the pallet template point cloud and the target point cloud are adjusted, the rotation and translation matrix is ​​reconstructed, and the value of the error function between the transformed pallet template point cloud and the target point cloud is calculated until the value of the error function meets the set error requirement (i.e., less than or equal to the preset error threshold). At this time, the error between the pallet template point cloud and the target point cloud is minimized; Figure 4 The figure shows the schematic diagram of the registration of the pallet template point cloud and the target point cloud, where the blue point cloud is the pallet template point cloud, the yellow point cloud is the target point cloud, and the red column is the vertical vector of the pallet template point cloud.

[0079] The rotation and translation matrix used when the error between the pallet template point cloud and the target point cloud is the minimum is the transformation information of the pallet template point cloud to the coordinate system of the target point cloud.

[0080] Then, based on the conversion information and the preliminary pose information, the final pose information of the pallet is obtained. For example, the preliminary pose information including the reference center point and the vertical vector of the target plane of the pallet facing the forklift is transformed using the rotation and translation matrix of the pallet template point cloud transformed to the target point cloud coordinate system, and the pallet center point of the transformed pallet template point cloud and the vertical vector of the transformed pallet template point cloud are obtained. Then, based on the vertical vector of the transformed pallet template point cloud, the final heading angle of the pallet can be obtained; the pallet center point of the pallet template point cloud and the final heading angle constitute the final pose information of the pallet.

[0081] Subsequently, the position of the forklift and the angle facing the pallet can be adjusted based on the final posture information of the pallet. For example, based on the center point of the pallet in the final posture information of the pallet, the position of the forklift can be adjusted so that the center point of the forklift is aligned with the center point of the pallet; based on the final heading angle in the final posture information of the pallet, the angle between the forklift and the pallet can be adjusted so that the forklift and the pallet are parallel, that is, the vertical vector of the pallet coincides with the horizontal axis of the forklift coordinate system; after the forklift and the pallet are aligned, the forklift can be directly controlled to move forward to pick up the pallet. In addition, based on the identified pallet type, the socket spacing corresponding to the pallet type can be obtained, and then the spacing of the fork arms can be adjusted.

[0082] Based on the above process, the preliminary posture information of the pallet is first obtained based on the target point cloud, and the target point cloud is a fusion of the pallet image and point cloud information, so the preliminary posture information is also the characteristics of the fusion of the image and point cloud information. Furthermore, the pallet template point cloud corresponding to the pallet type of the pallet is registered with the target point cloud to obtain the transformation information of the pallet template point cloud to the target point cloud coordinate system. At this time, the error between the transformed pallet template point cloud and the target point cloud is the smallest. The transformed pallet template point cloud is a more standard pallet point cloud, avoiding the deviation caused by the fusion of the pallet image and point cloud. Therefore, based on the transformation information and the preliminary posture information, more accurate pallet posture information can be obtained, which improves the accuracy of pallet posture recognition. The horizontal position accuracy of the pallet reaches ±10mm and the posture angle accuracy reaches ±1°; it directly shortens the pallet posture recognition time and improves the success rate of forklifts picking up pallets.

[0083] The second embodiment of the present invention relates to a method for recognizing the position and posture of a pallet. Compared with the first embodiment, in this embodiment, a target point cloud of the pallet is obtained by fusing the image quality and the point cloud quality.

[0084] The specific process of the pallet posture recognition method in this embodiment is as follows: Figure 5 shown.

[0085] Step 201, when the forklift is located in front of the pallet to be forked, a sensor image and a first point cloud in front of the fork arm of the forklift are obtained. This is substantially the same as step 101 in the first embodiment and will not be described in detail here.

[0086] Step 202, based on the image detection result obtained by detecting the sensor image, obtain the pallet image of the pallet in the sensor image and the pallet point cloud of the pallet in the first point cloud. This is substantially the same as step 102 in the first embodiment and will not be described in detail here.

[0087] Step 203 includes the following sub-steps:

[0088] Sub-step 2031, obtaining the image weight of the sensor image and the point cloud weight of the first point cloud.

[0089] Sub-step 2032 , based on the image weight and the point cloud weight, the coordinates of each point in the pallet image are merged with the coordinates of each point in the pallet point cloud to obtain a target point cloud of the pallet.

[0090] Specifically, different weights are set for the image and point cloud to fuse the coordinates of the corresponding points in the pallet image and the pallet point cloud. The specific process is as follows:

[0091] In one example, the image weight and the point cloud weight may be preset in advance.

[0092] In another example, the image weight and the point cloud weight are related to the quality of the acquired sensor image and the first point cloud. Specifically, the quality of the sensor image and the first point cloud are evaluated respectively to obtain the image quality parameter of the sensor image and the point cloud quality parameter of the first point cloud; based on the image quality parameter and the point cloud quality parameter, the image weight of the sensor image and the point cloud weight of the first point cloud are obtained. Specifically, as follows:

[0093] First, the quality of the sensor image is evaluated to obtain an image quality parameter of the sensor image, including: obtaining at least one image quality evaluation value of the sensor image, where the image quality evaluation value is any one of the following: a clarity evaluation value, an exposure evaluation value, and a motion blur evaluation value;

[0094] The clarity evaluation value is used to characterize the clarity of the sensor image. The calculation method is as follows: first, the gradient of the sensor image is calculated using the Laplace operator, the gradient distribution characteristics of the sensor image are statistically analyzed, and then the clarity evaluation value is generated based on the gradient distribution characteristics. The clarity evaluation value S q The calculation formula is as follows:

[0095]

[0096] Where: (x, y) represents the coordinate position of the pixel in the sensor image, represents the Laplace operator result of the sensor image at the pixel point (x, y), N represents the total number of pixels in the sensor image, α represents a preset weight coefficient (for example, 1.2), T_min represents a preset lower threshold value of image clarity (for example, 20), and T_max represents a preset upper threshold value of image clarity (for example, 150).

[0097] The exposure evaluation value is used to characterize the exposure of the sensor image. The calculation method is as follows: first analyze the sensor image brightness histogram, calculate the ratio of the overexposed area to the underexposed area in the sensor image, and then generate the exposure evaluation value based on the ratio of the overexposed area to the underexposed area in the sensor image. The exposure evaluation value S b The calculation formula is as follows:

[0098] Sb =min(100,max(0,100*(1-β*max(|μ-μ_target| / σ_target,P_over+P_u

[0099] nder))));

[0100] Wherein: μ represents the average brightness of pixels in the sensor image; μ_target represents a preset target brightness value (for example, 128); σ_target represents a preset target standard deviation (for example, 45); P_over represents the proportion of overexposed pixels in the sensor image, and overexposed pixels are pixels whose brightness values ​​are greater than the overexposure brightness threshold (for example, 220); P_under represents the proportion of underexposed pixels in the sensor image, and underexposed pixels are pixels whose brightness values ​​are less than the underexposure brightness threshold (for example, 30); β represents a preset weight coefficient (for example, 1.5).

[0101] The motion blur evaluation value is used to characterize the degree of motion blur of the sensor image. The calculation method is as follows: calculate the difference between the sensor image and the adjacent frame image, analyze the edge blur degree of the sensor image, and generate the motion blur evaluation value of the sensor image based on the edge blur degree of the sensor image. The motion blur evaluation value S m The calculation formula is as follows:

[0102] S m =min(100,max(0,100*(1-γ*Σ|I_t(x,y)-I_{t-1}(x,y)| / (N*255))));

[0103] Wherein: I_t(x,y) represents the pixel value of the sensor image of the current frame; I_{t-1}(x,y) represents the previous frame image of the sensor image; N is the total number of pixels of the sensor image; γ is a preset weight coefficient (for example, 2.0).

[0104] Based on at least one image quality evaluation value of the sensor image, an image quality parameter of the sensor image is obtained. Specifically: if the selected image quality evaluation value of the sensor image is one of the clarity evaluation value, the exposure evaluation value and the motion blur evaluation value, the image quality evaluation value can be directly used as the image quality parameter of the sensor image; if the selected image quality evaluation value of the sensor image is multiple of the clarity evaluation value, the exposure evaluation value and the motion blur evaluation value, the image quality parameter of the sensor image can be calculated by calculating the average or in a weighted manner.

[0105] For example, the clarity evaluation value, exposure evaluation value, and motion blur evaluation value of the sensor image are obtained at the same time, and then the weights of the clarity evaluation value, exposure evaluation value, and motion blur evaluation value are obtained respectively, so that the image quality parameter of the sensor image can be calculated by weighted summation. The expression of the image quality parameter Kt of the sensor image is as follows:

[0106] Kt = w_c*clarity evaluation value + w_e*exposure evaluation value + w_m*motion blur evaluation value;

[0107] Among them: w_c is the clarity weight, for example, 0.4; w_e is the exposure weight, for example, 0.3; w_m is the motion blur weight, for example, 0.3.

[0108] Performing a quality assessment on the first point cloud to obtain a quality assessment parameter of the first point cloud includes: obtaining at least one point cloud quality assessment value of the first point cloud, where the point cloud quality assessment value is any one of the following: a point cloud density assessment value, a point cloud integrity assessment value, and a noise level assessment value;

[0109] The point cloud density evaluation value is used to characterize the uniformity of the first point cloud. The calculation method is as follows: first calculate the number of points in a unit volume in the first point cloud, evaluate the uniformity of the first point cloud based on the number of points in each unit volume, and obtain the point cloud density evaluation value of the first point cloud. The point cloud density evaluation value S d The calculation formula is as follows:

[0110] S d =min(100,max(0,100*(1-δ*|ρ_actual-ρ_target| / ρ_target)));

[0111] Wherein: ρ_actual represents the actual point cloud density of the first point cloud, which is the number of points of the first point cloud divided by the volume; ρ_target is the preset target point cloud density, which can be set according to the current scene, the characteristics of the radar sensor and the accuracy requirements, for example, a value in the range of 100-500 points / cubic meter; δ is a preset weight coefficient (for example, 1.5).

[0112] The point cloud integrity evaluation value is used to characterize the integrity of the first point cloud, and its calculation method is: first compare the first point cloud with the pallet template point cloud corresponding to the pallet type of the pallet, calculate the coverage rate of the first point cloud covering the key feature area of ​​the pallet template point cloud (for example, the surface of the pallet facing the forklift), and obtain the number of matching points between the first point cloud and the pallet template point cloud; the specific calculation method of the coverage rate is: use the external parameter calibration matrix of the radar sensor to convert the pallet template point cloud to the coordinate system of the radar sensor, and then use the matching algorithm (such as the nearest neighbor algorithm) to find the closest point pair between the pallet template point cloud and the first point cloud in the same coordinate system. The closest point pair is the matching point pair between the first point cloud and the pallet template point cloud. The number of points in the pallet template point cloud that match the first point cloud is compared with the total number of points in the pallet template point cloud to obtain the coverage rate of the first point cloud covering the key feature area of ​​the pallet template point cloud (for example, the surface of the pallet facing the forklift); then the integrity evaluation value of the first point cloud can be obtained, and the integrity evaluation value S w The calculation formula is as follows:

[0113] S w =min(100,max(0,100*(N_match / N_template)*(1-ε*D_avg / D_max)));

[0114] Wherein: N_match represents the number of matching points in the first point cloud with the pallet template point cloud; N_template represents the number of points in the pallet template point cloud; D_avg represents the average point distance of the point pairs in the first point cloud that match the pallet template point cloud, specifically: for the matching point pairs between the first point cloud and the pallet template point cloud, the Euclidean distance between each point pair is calculated, and the average of the Euclidean distances of these point pairs is calculated as the average point distance; D_max represents the preset maximum allowable distance, which is set based on the geometric features of the pallet. Taking the standard pallet as an example, its value is within 20-50mm; ε represents the preset weight coefficient (for example, 0.8).

[0115] The noise level assessment value is used to characterize the noise situation in the first point cloud. The calculation method is as follows: first count the proportion of abnormal points in the first point cloud in the first point cloud. The method for determining abnormal points is as follows: first cluster the first point cloud to obtain a point cloud cluster in the first point cloud. For each point, calculate the distance from the point to the center of the local point cloud cluster, and determine whether the distance exceeds a preset threshold. If it exceeds the preset threshold, the point is determined to be an abnormal point; otherwise, it is a normal point; wherein the preset threshold is 1.5-2 times the mean of the distance from the point in the point cloud cluster to the center of the point cloud cluster. Perform a smoothness analysis on the first point cloud, and use the standard deviation of the local points to reflect the smoothness of the point cloud. The local points can be selected from the area near the first point cloud, that is, the points on the pallet facing the forklift surface (the area radius is, for example, 5-10cm). Then, the standard deviation of the points in the area near the first point cloud is calculated as the standard deviation of the local points; then generate a noise level assessment value, the noise level assessment value S z The calculation formula is as follows:

[0116] S z =min(100,max(0,,100*(1-η*(N_outlier / N_total+σ_local / σ_max))));

[0117] Wherein: N_outlier represents the number of outliers in the first point cloud; N_total represents the total number of points in the first point cloud; σ_local represents the local standard deviation, that is, the standard deviation of the local points mentioned above; σ_max is the maximum allowable standard deviation, which is set based on the accuracy of the radar sensor, for example, a value within 5-10mm or 10-20mm; η is the weight coefficient (for example, 1.2).

[0118] Based on at least one point cloud quality assessment value of the first point cloud, a point cloud quality assessment value of the first point cloud is obtained. Specifically: if the selected point cloud quality assessment value of the first point cloud is one of the point cloud density assessment value, the point cloud integrity assessment value, and the noise level assessment value, the image quality assessment value can be directly used as the image quality parameter of the sensor image; if the selected point cloud quality assessment value of the first point cloud is multiple of the point cloud density assessment value, the point cloud integrity assessment value, and the noise level assessment value, the point cloud quality assessment value of the first point cloud can be calculated by calculating the mean or in a weighted manner.

[0119] For example, the point cloud density evaluation value, the point cloud integrity evaluation value and the noise level evaluation value of the first point cloud are obtained at the same time, and then the weights of the point cloud density evaluation value, the point cloud integrity evaluation value and the noise level evaluation value are obtained respectively, so that the point cloud quality parameter of the first point cloud can be calculated by weighted summation. The expression of the point cloud quality parameter Kd of the first point cloud is as follows:

[0120] Kd = w_d*point cloud density assessment value + w_b*point cloud integrity assessment value + w_n*noise level assessment value;

[0121] Among them: w_d is the clarity weight, for example, 0.3; w_b is the exposure weight, for example, 0.4; w_n is the motion blur weight, for example, 0.3.

[0122] After the above calculation process, the image quality parameter Kt of the sensor image and the point cloud quality parameter Kd of the first point cloud are calculated, and then the image weight of the sensor image and the point cloud weight of the first point cloud are calculated. The specific calculation formula is as follows:

[0123] The image weight Wt of the sensor image is calculated as: Wc = Kt / (Kt+Kd);

[0124] The calculation formula of the point cloud weight Wd of the first point cloud is: Wd=Kd / (Kt+Kd).

[0125] Subsequently, based on the image weight and the point cloud weight, the coordinates of each point in the pallet image and the coordinates of each point in the pallet point cloud can be fused to obtain the target point cloud of the pallet.

[0126] For each point on the pallet, it has corresponding coordinates on the pallet image and the pallet point cloud. Then, the two coordinates corresponding to a point are fused to obtain the coordinates of the point. Taking a point P as an example, the fused coordinates of point P (P xm , P ym , P zm ) is calculated as:

[0127] P xm =Wc*P xt +Wd*P xc ;

[0128] P ym =Wc*P yt +Wd*P yc ;

[0129] P zm =Wc*P zt +Wd*P zc ;

[0130] Among them, (P xt , P yt , P zt ) represents the coordinates of point P on the tray image, (P xc , P yc , P zc ) represents the coordinates of point P on the pallet point cloud.

[0131] In some embodiments, the image weight and the point cloud weight can also be adjusted based on the light intensity value of the scene currently located by the forklift and / or the integrity assessment value of the first point cloud. For example, the light intensity upper limit threshold and the light intensity lower limit threshold, as well as the integrity upper limit threshold and the integrity lower limit threshold are preset in advance. If the light intensity value of the scene currently located by the forklift is greater than the light intensity upper limit threshold, and / or the integrity assessment value of the first point cloud is less than the integrity lower limit threshold, the image weight is increased and the point cloud weight is reduced accordingly. For example, if the current image weight is 0.4 and the point cloud weight is 0.6, the image weight can be increased to 0.5 and the point cloud weight can be reduced to 0.5. If the light intensity value of the scene currently located by the forklift is less than the light intensity lower limit threshold, and / or the integrity assessment value of the first point cloud is greater than the integrity upper limit threshold, the image weight is reduced and the point cloud weight is increased accordingly. For example, if the current image weight is 0.4 and the point cloud weight is 0.6, the image weight can be reduced to 0.3 and the point cloud weight can be increased to 0.7.

[0132] Based on the adjusted image weight and point cloud weight, the coordinates of each point in the pallet image and the coordinates of each point in the pallet point cloud are fused to obtain the target point cloud of the pallet. The calculation method is similar to the above method and will not be repeated here.

[0133] Step 204, determining the position and posture information of the pallet based on the target point cloud of the pallet, is substantially the same as step 104 in the first embodiment and will not be described in detail herein.

[0134] Based on the above process, the weights can be adjusted and point cloud fusion can be achieved based on the current scene of the forklift to adapt to different lighting conditions and multiple pallet types, thereby increasing the anti-interference ability of pallet posture recognition in the point cloud fusion stage and improving the accuracy of the target point cloud of the pallet obtained by fusion.

[0135] The third embodiment of the present invention relates to a controller, which is applied to a forklift, and can be a controller dedicated to pallet posture recognition, or a main controller in the forklift. The controller is used to execute the steps of the pallet posture recognition method in the first or second embodiment.

[0136] Since the first and second embodiments correspond to this embodiment, this embodiment can be implemented in conjunction with the first and second embodiments. The relevant technical details mentioned in the first and second embodiments are still valid in this embodiment, and the technical effects that can be achieved in the first and second embodiments can also be achieved in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the first and second embodiments.

[0137] A fourth embodiment of the present invention relates to a forklift, which comprises a forklift body, a data acquisition device and the controller in the third embodiment.

[0138] The data acquisition device is mounted on the forklift body, and is used to collect images and point cloud data in front of the forklift's fork arm and send them to the controller. The data acquisition device may include: a radar sensor (such as a laser radar) and an image sensor (such as a camera, a video camera); the radar sensor is mounted on the front of the forklift body, and is used to obtain point cloud data in front of the forklift's fork arm; the image sensor is installed on the forklift's fork arm (for example, installed at the fork arm fork tip), and is used to collect images in front of the forklift's fork arm. It should be noted that this application only describes in detail the part of the forklift's pallet posture recognition, and does not describe in detail the working principle of the forklift itself. You can refer to the existing forklifts, and will not repeat them here.

[0139] The controller may execute the steps of the tray posture recognition method in the first or second embodiment.

[0140] A fifth embodiment of the present invention relates to a computer-readable storage medium, which is a non-volatile storage medium or a non-transient storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the tray posture recognition method in the first or second embodiment are executed.

[0141] Preferred embodiments of the present invention have been described above in detail, but it should be understood that aspects of the embodiments can be modified, if necessary, to employ aspects, features and concepts of the various patents, applications and publications to provide further embodiments.

[0142] These and other changes can be made to the embodiments in light of the above detailed description.In general, in the claims, the terms used should not be considered limited to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which these claims are entitled.

Claims

1. A pallet posture recognition method, characterized in that: include: When the forklift is located in front of the pallet to be forked, a sensor image and a first point cloud in front of the fork arm of the forklift are obtained; Based on an image detection result obtained by detecting the sensor image, acquiring a pallet image of the pallet in the sensor image and a pallet point cloud of the pallet in the first point cloud; Fusion of the pallet image with the pallet point cloud to obtain a target point cloud of the pallet; Based on the target point cloud of the pallet, the position and posture information of the pallet is determined.

2. The method for recognizing the position and posture of a pallet according to claim 1, characterized in that: The image detection result includes the pallet type of the pallet; and based on the target point cloud of the pallet, determining the position information of the pallet includes: Based on the target point cloud of the pallet, obtaining preliminary position information of the pallet; Based on the preliminary pose information of the pallet, the target point cloud and the pallet template point cloud corresponding to the pallet type of the pallet, the final pose information of the pallet is obtained.

3. The method for recognizing the position and posture of a pallet according to claim 2, characterized in that: Based on the target point cloud of the pallet, preliminary position information of the pallet is obtained, including: Selecting a plurality of feature points located within a set feature area of ​​the pallet from the target point cloud; Calculating the center point of the plurality of feature points as the reference center point of the pallet; Obtaining a vertical vector of the pallet toward a target plane of the forklift based on the coordinates of the plurality of feature points; Determining a reference heading angle of the target plane relative to the forklift based on a vertical vector of the target plane; The preliminary position information includes the reference center point and the reference heading angle.

4. The method for recognizing the position and posture of a pallet according to claim 3, characterized in that: Obtaining a vertical vector of the pallet toward a target plane of the forklift based on the coordinates of the plurality of feature points includes: The plurality of feature points are combined into a feature data matrix, and the coordinate values ​​of each dimension contained in the feature data matrix are standardized; The covariance matrix of the characteristic data matrix after standardization is obtained, and the eigenvector corresponding to the maximum eigenvalue among all eigenvalues ​​of the covariance matrix is ​​used as the vertical vector.

5. The method for recognizing the position and posture of a pallet according to claim 1, characterized in that: The pallet image is fused with the pallet point cloud to obtain a target point cloud of the pallet, including: Acquire an image weight of the sensor image and a point cloud weight of the first point cloud; Based on the image weight and the point cloud weight, the coordinates of each point in the pallet image and the coordinates of each point in the pallet point cloud are fused to obtain the target point cloud of the pallet.

6. The method for recognizing the position and posture of a pallet according to claim 5, characterized in that: Acquiring an image weight of the sensor image and a point cloud weight of the first point cloud, including: Performing quality assessment on the sensor image and the first point cloud respectively to obtain image quality parameters of the sensor image and point cloud quality parameters of the first point cloud; Based on the image quality parameter and the point cloud quality parameter, an image weight of the sensor image and a point cloud weight of the first point cloud are obtained.

7. The method for recognizing the position and posture of a pallet according to claim 5, characterized in that: After acquiring the image weight of the sensor image and the point cloud weight of the first point cloud, the method further includes: Adjusting the image weight and the point cloud weight based on the light intensity value of the scene currently located by the forklift and / or the integrity assessment value of the first point cloud; Based on the image weight and the point cloud weight, the coordinates of each point in the pallet image are fused with the coordinates of each point in the pallet point cloud to obtain the target point cloud of the pallet, including: Based on the adjusted image weight and the point cloud weight, the coordinates of each point in the pallet image and the coordinates of each point in the pallet point cloud are fused to obtain the target point cloud of the pallet.

8. The method for recognizing the position and posture of a pallet according to claim 6, characterized in that: The quality assessment method of the sensor image is: Acquire at least one image quality evaluation value of the sensor image, wherein the image quality evaluation value is any one of the following: a clarity evaluation value, an exposure evaluation value, and a motion blur evaluation value; Obtaining an image quality parameter of the sensor image based on the at least one image quality assessment value of the sensor image; The quality assessment method of the first point cloud is: Acquire at least one point cloud quality assessment value of the first point cloud, the point cloud quality assessment value being any one of the following: a point cloud density assessment value, a point cloud integrity assessment value, and a noise level assessment value; Based on the at least one point cloud quality assessment value of the first point cloud, a point cloud quality parameter of the first point cloud is obtained.

9. A controller, characterized in that: Used to execute the steps of the pallet posture recognition method described in any one of claims 1-8.

10. A forklift, characterized in that: include: A forklift body, a data acquisition device, and a controller as claimed in claim 9; The data acquisition device is mounted on the forklift body and is used to collect sensor images and point clouds in front of the fork arm of the forklift and send them to the controller.

Images