Goods overloading detection method

Abstract

The invention discloses a cargo overloading detection method. The method comprises the following steps: acquiring a real-time tray image and depth data; inputting a real-time tray image and depth data into the trained tray identification model to obtain a region of interest; segmenting the region of interest to obtain a tray jack segmentation region; according to the tray jack segmentation area and the tray physical structure characteristics, calculating to obtain the pixel coordinates of the center point of the tray pillar; calculating a yaw angle of the tray relative to the center of the vehicle body according to the pixel coordinates of the central point of the tray strut and the depth data; according to the pixel coordinates of the center point of the tray supporting column, the yaw angle of the tray relative to the center of the vehicle body and a preset over-loading threshold value, the cargo over-loading detection range is calculated; and comparing the preset minimum range threshold value with the cargo over-loading detection range, and judging whether the tray is over-loaded or not. According to the method, the AGV tail end operation safety application in the man-vehicle hybrid scene is effectively improved, and the AGV application scene is expanded.

Description

Technical Field

[0001] The present application belongs to the technical field of industrial vehicle automatic driving, and in particular, to a method for detecting over-cargo. Background Art

[0002] For complex mixed human and vehicle operation scenarios, there are various types of intricate environments such as human-dragged carts, manually operated forklifts, AGV automatic forklifts, etc., and all kinds of pallet loads may be offset, blocked, and over-carried. In mixed human and vehicle operation scenarios, the traditional method uses auxiliary tags to identify pallets and installs cargo sensors to determine whether there is cargo on the forks. This method requires manual participation and a large amount of additional work, and cannot adapt to the needs of large scenarios. In addition, the cargo sensor cannot detect whether the pallet cargo is over-carried, and the end operation objects are subject to offset and over-carrying, and there is a risk of cargo falling off. Summary of the invention

[0003] In order to solve the deficiencies in the prior art, the present application proposes a method for detecting over-consignment of goods, comprising:

[0004] Get real-time pallet images and depth data;

[0005] Input the real-time pallet image and depth data into the trained pallet recognition model to obtain the region of interest;

[0006] Segment the region of interest to obtain the tray jack segmentation region;

[0007] According to the segmentation area of ​​the pallet socket and the physical structure characteristics of the pallet, the pixel coordinates of the center point of the pallet support are calculated;

[0008] The yaw angle of the pallet relative to the center of the vehicle body is calculated based on the pixel coordinates of the center point of the pallet support and the depth data;

[0009] The cargo over-pallet detection range is calculated based on the pixel coordinates of the center point of the pallet pillar, the yaw angle of the pallet relative to the center of the vehicle body, and the preset over-pallet threshold;

[0010] Compare the preset minimum range threshold with the cargo over-pallet detection range to determine whether the pallet is over-palletized.

[0011] Optionally, the calculating the pixel coordinates of the center point of the pallet support pillar according to the segmented area of ​​the pallet insertion hole and the physical structure characteristics of the pallet includes:

[0012] Performing filtering processing on the segmented pallet image to obtain the pallet end surface area, the pallet support area and the insertion hole area;

[0013] According to the circumscribed rectangle of the socket area, the pixel coordinates of the four corner points of the middle pillar area in the pillar area of ​​the tray and the pixel coordinates of the center point of the middle pillar are obtained;

[0014] According to the physical dimensions of the left and right pillars of the tray, the pixel coordinates of the center points of the left pillar and the right pillar are obtained; the pixel coordinates of the center point of the middle pillar, the pixel coordinates of the center point of the left pillar and the pixel coordinates of the center point of the right pillar constitute the pixel coordinates of the center point of the right pillar.

[0015] Optionally, the calculating of the yaw angle of the pallet relative to the center of the vehicle body according to the pixel coordinates of the center point of the pallet support and the depth data includes:

[0016] The yaw angle of the end face of the pallet in the camera coordinate system is calculated according to the pixel coordinates of the center point of the pallet support and the depth data;

[0017] The yaw angle of the end surface of the pallet in the camera coordinate system is converted into the yaw angle of the end surface of the pallet in the vehicle body coordinate system through the calibrated rotation matrix of the camera coordinate system relative to the vehicle body coordinate system.

[0018] Optionally, the yaw angle of the end surface of the pallet in the camera coordinate system is calculated according to the pixel coordinates of the center point of the pallet support and the depth data, and the calculation formula is as follows:

[0019]

[0020] Where, θ' is the yaw angle of the tray relative to the camera, d p3 is the depth value of the left pillar center point P3, d p4 is the depth value of the center point P4 of the right pillar; p xP3 is the pixel horizontal coordinate of the center point P3 of the left pillar, p xP4 is the pixel horizontal coordinate of the center point P4 of the right pillar, f x is the horizontal pixel focal length of the camera, c x is the horizontal coordinate of the camera optical center in the pixel coordinate system.

[0021] Optionally, the yaw angle of the end face of the pallet in the camera coordinate system is converted into the yaw angle of the end face of the pallet in the vehicle coordinate system through the rotation matrix of the calibrated camera coordinate system relative to the vehicle coordinate system, and the calculation formula is:

[0022] θ=arctan2(r21,r11)+θ'

[0023] Among them, θ' is the yaw angle of the end surface of the pallet in the camera coordinate system, θ is the yaw angle of the end surface of the pallet in the vehicle coordinate system, r21 is the value of the second row and first column of the rotation matrix of the calibrated camera coordinate system relative to the vehicle coordinate system, and r11 is the value of the first row and first column of the rotation matrix of the calibrated camera coordinate system relative to the vehicle coordinate system.

[0024] Optionally, the cargo over-pallet detection range includes: the front of the pallet, the left side of the pallet, and the right side of the pallet.

[0025] Optionally, the cargo over-carrying detection range is calculated based on the pixel coordinates of the center point of the pallet pillar, the yaw angle of the pallet relative to the center of the vehicle body and a preset over-carrying threshold, including: over-carrying point detection on the front of the pallet, over-carrying point detection on the left side of the pallet, over-carrying point detection on the right side of the pallet and statistical filtering of over-carrying points and image filtering processing.

[0026] Optionally, the statistical filtering of super-support points and image filtering processing includes:

[0027] Traverse the super-support detection range;

[0028] The corrosion post-processing of the over-support detection range is carried out to remove the noise data and filter it into discrete areas using the preset minimum range threshold;

[0029] Expand the small target area in the super-support detection range and restore it to the original cross-section of the super-support object;

[0030] Find the edge information of the hyper-support object, segment the image and extract the object features, and find the circumscribed rectangle of the hyper-support object in the hyper-support detection range;

[0031] Traverse the circumscribed rectangle of the over-palletized goods, and use the preset minimum range threshold to filter again to detect whether there is over-palletized goods on the pallet.

[0032] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0033] The present application provides a method for detecting over-cargo, which performs over-cargo detection on cargo on pallets that are not fixedly placed in mixed human and vehicle operation scenarios of AGV operations, effectively improving the safety application of AGV terminal operations in mixed human and vehicle scenarios and expanding AGV application scenarios.

[0034] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application are realized and obtained by the structures specifically pointed out in the description, claims and drawings.

[0035] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings.

[0036] Advantages of additional aspects of the present application will be given in part in the following description, and in part will become apparent from the following description, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is a flow chart of a method for detecting over-consignment of goods as shown in an embodiment of the present application;

[0039] Figure 2 A flow chart of detection and segmentation of the deep learning method shown in the embodiment of the present application;

[0040] Figure 3 A schematic diagram of the center of a tray support column shown in an embodiment of the present application;

[0041] Figure 4 It is a super-tray scene diagram of the front and left and right sides of the tray shown in the embodiment of the present application;

[0042] Figure 5 It is a top view schematic diagram of the AGV body shown in the embodiment of the present application when it reaches the terminal operation front point;

[0043] Figure 6 This is a schematic diagram of the front over-support detection range when counting over-support points shown in an embodiment of the present application;

[0044] Figure 7 A schematic diagram of detecting the over-support range of the front tray shown in an embodiment of the present application;

[0045] Figure 8 This is a schematic diagram of detecting the over-support range of the right tray shown in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0047] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of methods and devices consistent with some aspects of the present application as detailed in the attached claims.

[0048] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0049] In complex terminal operation scenarios involving mixed human and vehicle operations in the AGV (Automatic Guided Vehicle) forklift industry, when the cargo exceeds a certain range of the pallet, it is generally believed that there is a certain risk of the cargo falling off after the pallet is forked, and it is necessary to wait and warn manual processing. After the manual processing of the over-pallet cargo, the AGV terminal operation can continue to complete the terminal operation;

[0050] At present, the recognition and positioning of pallets in unmanned forklift-related research usually adopts depth vision, laser radar, millimeter wave radar and other depth sensor solutions. AGV forklifts can recognize and locate pallets within a certain range. The current AGV body system installs a load sensor and designs a certain threshold according to the load weight to determine whether the fork is loaded. It is unable to detect the over-carrying of pallet cargo, and there is a safety hazard of cargo falling off in the terminal operation. Therefore, this application provides a method for detecting over-carrying of cargo, which detects whether the pallet cargo to be forked exceeds the upper plane of the pallet. This is further explained below in conjunction with the embodiments and drawings.

[0051] Example

[0052] The following will be combined Figures 1 to 8 , a method for detecting over-consignment of goods provided in an embodiment of the present application is described in detail. Figure 1 As shown, the method of the embodiment of the present application may include the following steps:

[0053] Step S1: Acquire real-time pallet image and depth data;

[0054] During the specific implementation process, a low-cost depth camera is used and installed in the middle position of the root of the forklift fork. The installation height of the depth camera is 20mm-30mm higher than the upper plane of the fork. When the forklift arrives at the front point of the operation, RGB (red, green, blue) and depth data are collected; RGB constitutes a real-time pallet image.

[0055] Step S2: input the real-time pallet image and depth data into the trained pallet recognition model to obtain the region of interest;

[0056] In the specific implementation process, the pallet recognition model is trained by using the historical pallet images collected by the depth camera of the depth camera installation method in step S1; including:

[0057] Calibrate pallet categories for historical pallet images in AGV stacking scenarios;

[0058] The trained pallet recognition model is obtained based on the calibrated historical pallet images and deep learning methods.

[0059] In specific implementation, the trained pallet recognition model can also be converted into a deep learning model of the domestic chip ARM platform through the model conversion tool;

[0060] Specifically, the pallet recognition model is designed by adding a small target detection layer network through the existing open source YOLOv8 backbone network architecture; 10,000 RGB data containing the complete pallet area are collected in the terminal operation scene of the depth camera, and the open source tool LabImg is used to calibrate and segment the jacks; and the calibrated data is subjected to multiple rounds of training in the form of data enhancement; the deep learning model of the domestic chip ARM platform of this application refers to training the model on the domestic Rockchip RK3568 platform server and verifying the running results, converting the model into an RKNN model through the RK Toolkit tool model, and designing quantitative reasoning to obtain the model output results.

[0061] Deploy the pallet recognition model, and obtain the pallet ROI (Region Of Interest) by inputting the real-time pallet image and depth data into the trained pallet recognition model, that is, the location of the pallet in the RGB candidate area; the specific process is as follows:

[0062] When the AGV body reaches the terminal operation front point, the body issues the terminal operation instruction, and the system inputs the real-time collected RGB image and depth data into the pallet recognition model to obtain the pixel coordinates of the pallet recognition, and completes the ROI area of ​​the pallet area on the original image, which specifically includes the area Box pixel coordinates, pixel length and width, and confidence. This application selects a confidence level of more than 85% as a pallet, otherwise it is considered that the pallet cannot be detected.

[0063] Step S3: segment the region of interest to obtain a tray jack segmentation region;

[0064] According to the ROI area, the image of the pallet in the original RGB is cropped out, and the pallet image in the original RGB is segmented; according to image processing morphology, edge detection and image gradient algorithm, the pallet image is filtered to eliminate unnecessary lines and holes, and a pallet end face area including the entire end face of the pallet, the support area of ​​the pallet and the socket area are obtained. The pallet end face area, the support area of ​​the pallet and the socket area constitute the pallet socket segmentation area.

[0065] Step S4: Calculate the pixel coordinates of the center point of the pallet support according to the segmented area of ​​the pallet socket and the physical structure characteristics of the pallet;

[0066] Specifically, find the circumscribed rectangle of the socket segmentation area, obtain the pixel coordinates of the four corner points of the middle pillar area connecting the two socket areas, and thus obtain the pixel coordinates of the center point of the middle pillar;

[0067] According to the physical dimensions of the left and right pillars of the tray, the pixel coordinates of the center points of the left and right pillar areas are calculated.

[0068] Figure 2 and Figure 3 They are respectively the deep learning method detection and segmentation process and the schematic diagram of the pallet support center during steps S2 to S4.

[0069] Step S5: Calculate the yaw angle of the pallet relative to the center of the vehicle body according to the pixel coordinates of the center point of the pallet support and the depth data; the steps are as follows:

[0070] Step S5.1: Calculate the yaw angle of the end surface of the pallet in the camera coordinate system according to the pixel coordinates of the center point of the pallet support and the depth data;

[0071] In the camera coordinate system, based on the pixel coordinates and depth values ​​of the center points of the left and right pillars, the yaw angle θ' of the pallet relative to the camera is measured: the calculation formula is as follows:

[0072]

[0073] Among them, d p3 is the depth value of the left pillar center point P3, d p4 is the depth value of the center point P4 of the right pillar; p xP3 is the pixel horizontal coordinate of the center point P3 of the left pillar, p xP4 is the pixel horizontal coordinate of the center point P4 of the right pillar, f x is the horizontal pixel focal length of the camera, c x is the horizontal coordinate of the camera optical center in the pixel coordinate system.

[0074] Step S5.2: The yaw angle of the end surface of the pallet in the camera coordinate system is converted into the yaw angle of the end surface of the pallet in the vehicle coordinate system through the calibrated rotation matrix of the camera coordinate system relative to the vehicle coordinate system.

[0075] Specifically, the calculation formula is:

[0076] θ=arctan2(r21,r11)+θ′………………(2)

[0077] θ is the yaw angle of the end surface of the pallet in the vehicle coordinate system, r21 is the value of the second row and first column of the rotation matrix of the calibrated camera coordinate system relative to the vehicle coordinate system, and r11 is the value of the first row and first column of the rotation matrix of the calibrated camera coordinate system relative to the vehicle coordinate system.

[0078] The rotation matrix of the camera coordinate system relative to the vehicle coordinate system is obtained by calibrating the external parameters of the depth camera. The process is detailed as follows:

[0079] Considering the differences in the body structures of different AGVs, the camera is installed at a fixed position on the AGV. The center of the AGV body needs to be used as a reference to calibrate the external parameters of the depth camera. External parameter calibration is the process of solving the transformation relationship between the camera coordinate system and the body coordinate system. The transformation between the two three-dimensional coordinate systems is a rigid body motion, which is represented by a rotation matrix and a translation matrix;

[0080] The data in the depth camera contains the image RGB data and the depth data of each pixel. Let P be any point in the three-dimensional space, the pixel coordinates in the image captured by the depth camera are (u, v) and its depth value is d. Since the coordinate position of the object in the image is in pixels, the camera coordinate value of point P needs to be converted into pixel coordinate value. This process is described by the pinhole imaging model.

[0081] In the camera coordinate system, let the three-dimensional space coordinates of point P be (Xc, Yc, Zc), where:

[0082] X c =(uc x )×d / f x …………………………(3)

[0083] Y c =(vc y )×d / f y …………………………(4)

[0084] Z c =d…………………………(5)

[0085] Where u and v are the pixel horizontal coordinate and pixel vertical coordinate of point P in the camera image, respectively, and d is the depth value of point P measured in the camera coordinate system; fx is the horizontal pixel focal length of the depth camera, f y is the vertical pixel focal length of the depth camera; c x and c y are the horizontal and vertical coordinates of the optical center of the depth camera in the pixel coordinate system.

[0086] According to the matrix method:

[0087]

[0088] Wherein K is the camera intrinsic parameter matrix, which is the inherent coefficient of the camera itself. In this embodiment, it can be directly used by the camera manufacturer, or it can be calibrated and obtained through Zhang Zhengyou calibration method.

[0089] The rotation matrix R and translation matrix T of the camera coordinate system relative to the vehicle coordinate system are solved by a nonlinear optimization algorithm. The process is as follows:

[0090] In the vehicle coordinates, let the three-dimensional coordinates of point P be (X A , Y A , Z A ), then the transformation between the two three-dimensional coordinate systems of point P in the camera coordinate system and the vehicle coordinate system is a rigid body motion, which is represented by the rotation matrix and the translation matrix:

[0091]

[0092] After calibration

[0093] R is a 3*3 matrix, which is the rotation matrix of the camera coordinate system relative to the vehicle coordinate system; T is a three-dimensional vector, which is the translation matrix of the camera coordinate system relative to the vehicle coordinate system.

[0094] Step S6: Calculate the cargo over-carrying detection range according to the pixel coordinates of the center point of the pallet pillar, the yaw angle of the pallet relative to the center of the vehicle body, and the preset over-carrying threshold;

[0095] In the mixed scene of AGV, due to the random placement of people, there are deviations and over-carrying phenomena in the end-operation objects, among which over-carrying may exist on all sides of the pallet. However, since the depth camera in the end-operation scene is installed in the range of 200mm-300mm just above the middle of the fork root, the camera cannot capture the opposite side of the front end of the pallet. Therefore, this application specifically performs over-carrying detection calculations in 3 scenarios, namely, the front side, left and right sides of the pallet;

[0096] Specifically, Figure 4 , through the preset over-support threshold L thresh And measure the height range of over-cargo ΔH, calculate the over-cargo detection range; that is, perform over-cargo point detection;

[0097] like Figure 5 In the top view, the corresponding depth value d is obtained from the multi-frame fused depth data according to the pixel coordinates (Mx, My), (Px, Py), (Nx, Ny) of the center points M, P, and N of the three pillars of the tray obtained in step S4 and M =EM,d P =OP,d N =FN; preset the over-support threshold to PQ=L thresh and cargo overload height ΔH(L thresh and ΔH can be set according to the requirements. This application sets L thresh =100mm ， ΔH=200mm), the pallet yaw angle ∠TPN calculated in step 5 is θ, which has positive and negative values, and is an angle between the fork and the vertical direction in the counterclockwise direction; MN is the actual physical width of the left and right pillars of the pallet. In this application, the pallet MN=1100mm.

[0098] When the cargo on the pallet exceeds the pallet size by more than L thresh Value, that is, the goods on the pallet exceed the PQ outer dimensions, from Figure 5 It can be seen that ∠SPQ is equal to ∠TPN, that is, ∠SPQ = θ. Figure 5 It can be seen that MG = PS = NH, and the critical point size of super-support ΔT = PS = PQ / cosθ = L thresh / cosθ, JN=MN*cosθ. For a certain height directly above the pallet, select the JN*ΔH rectangular area directly above the pallet ROI area to count whether all depth value ranges produce over-pallet, and count the cargo depth data.

[0099] Taking the three pillars of the pallet as reference points, the detection range of the pallet front over-support is calculated as follows:

[0100] Estimate the depth value d of the pixel coordinate (u, v) of any point R on the front of the pallet R , L is the distance from point R to the center point of the middle support of the pallet, and the method is as follows:

[0101]

[0102] d R =L*cosθ+d p ………………………… (10)

[0103] ΔT=L thresh / cosθ…………………… (11)

[0104] like Figure 6 In the rectangular area of ​​the statistical super-support area JN*ΔH, the current measured depth d is less than d R When -ΔT, the current depth measurement is considered to be a super-support point.

[0105] Similarly, the calculation method of the over-support detection range on the right side of the pallet is the same as that of the over-support detection range on the front side of the pallet; the specific description is as follows:

[0106] like Figure 8 The red box indicates the detected tray ROI area, the green box indicates the side (right) over-tray detection, and the over-tray threshold L thresh The measurement of the height range ΔH of overloaded cargo is the same as the front overload detection.

[0107] First, obtain the side over-support detection area;

[0108] Specifically, the horizontal distance L to the right is estimated based on the Nx horizontal coordinate of the center point N (Nx, Ny) of the right support of the tray. thresh The pixel coordinates D(Dx,Ny) of are obtained according to formula (12):

[0109]

[0110] Dx is the starting horizontal coordinate of the super-support area, and the width of the super-support area ΔW is set to a fixed value, which is set to 100 pixels in this application;

[0111] After obtaining the over-support detection area, the right support depth value d of the pallet N As the reference standard, traverse each pixel in the super-support area and compare its depth value with d N If the depth value difference is within 200mm, the current point is considered to be a super-support point and is retained. The depth values ​​of other points are negative and are set to 0.

[0112] To calculate the over-support detection range on the left side of the pallet, substitute the coordinates of the center point M of the left support column of the pallet into formula (9) in the same way as above and will not be repeated here.

[0113] Step S7: Compare the preset minimum range threshold with the cargo over-pallet detection range to determine whether the pallet is over-palletized.

[0114] Specifically, by statistically filtering the over-support points and performing image filtering processing, according to the preset minimum range threshold of the over-support of the pallet relative to the front and both sides of the vehicle body, it is detected whether there is over-support of the pallet, and its over-support type and over-support size.

[0115] By counting the over-carrying point detection of the depth data of the area at a set height directly above the pallet, the data without over-carrying points is filtered to obtain the over-carrying image. Since the depth camera measurement itself has errors, and the over-carrying goods generally have a certain shape, objects with particularly slender cross-sections are usually directly filtered. For this reason, this application designs an image filtering algorithm for the cross-section of the over-carrying goods, such as Figure 7 , the red box indicates the detected pallet over-carrying area, and the green box indicates the possible interference point. The processing process is as follows:

[0116] ① Traverse the super-support detection area, remove non-super-support points, that is, isolated points, and assign the depth value to 0;

[0117] ②Using the image morphology algorithm, the super-support area is corroded and processed to remove noise data, and filtered into discrete areas according to the preset minimum range thresholds (minimum area, minimum width, and minimum height thresholds);

[0118] ③Using the expansion algorithm in image morphology, the small target area in the super-support area is expanded to restore it to the original cross-section of the super-support object;

[0119] ④ Through the image processing algorithm, find the edge information of the hyper-support object, segment the image and extract the object features, and find the circumscribed rectangle of the hyper-support object in the hyper-support area;

[0120] ⑤ Traverse the circumscribed rectangle of the over-palletized goods, and use the preset minimum range threshold (minimum area, minimum width, and minimum height thresholds) method to filter again to detect whether there is over-palletized goods on the pallet.

[0121] During the specific implementation process, the terminal operation over-pallet process executes the pallet front over-pallet and side over-pallet processes in sequence. When any over-pallet occurs, the system uploads the AGV body and makes an audible and visual alarm. At the same time, the system periodically detects over-pallet (in the specific implementation, the application cycle is set to 3 seconds, and the user can set the duration by himself), and waits for the manual processing of the over-pallet to be completed before proceeding with the remaining terminal operation process.

[0122] During the end operation process, the coordinates of the center P of the middle pillar of the pallet in the camera coordinate system (Xc, Yc, Zc) are calculated, combined with the calibration external parameters relative to the vehicle body when the camera is installed, the position of the vehicle body in the world coordinate system, and the coordinate conversion between the world coordinate system and the pixel coordinate system. In the actual scene, the R and T values ​​are obtained according to the calibration results, so as to estimate the position and posture of the pallet;

[0123]

[0124] [R|T] represents the extrinsic parameter matrix between the calibrated vehicle body and the camera.

[0125] The pixel coordinates and depth values ​​of point P in the three-dimensional space are calculated by formulas (6) and (8) and the conversion relationship between the vehicle body coordinates is as follows:

[0126]

[0127] Substituting formula (5) into (15), we obtain:

[0128]

[0129] According to formula (16), the coordinate values ​​of different calibration points in the vehicle coordinate system are obtained.

[0130]

[0131] Where [R|T] is the extrinsic matrix between the calibrated vehicle body and the camera, R2 is the rotation matrix of the vehicle body coordinate system relative to the world coordinate system, and T2 is the offset matrix of the vehicle body coordinate system relative to the world coordinate system;

[0132] It can be understood that the values ​​of R2 and T2 are obtained according to the posture information of the vehicle body in the world coordinate system in the actual AGV operation scenario (R2|T2 is usually obtained in real time by AGV navigation control);

[0133] Through rigid body motion calculation, the coordinates O(Xw, Yw, Zw) of the center point O of the pallet leg in the world coordinates are obtained;

[0134] Furthermore, according to the position information of the pallet in the world coordinate system, the Bezier curve between two points is calculated to form a local path planning. The AGV performs local path planning according to the global position of the pallet, and makes adaptive control adjustments according to the new trajectory to achieve accurate forking of the pallet, completing the AGV terminal operation process in the mixed scene of people and vehicles.

[0135] The method for detecting over-cargo transportation disclosed in this embodiment is aimed at complex AGV operation scenarios where people and vehicles are mixed, such as the typical cargo sorting industry (such as express sorting), where pallets and their cargo are placed arbitrarily by humans. After the cargo on the pallet is placed manually, there is a high possibility of over-cargo transportation. To avoid the risk of cargo on the pallet falling, this application designs a cargo over-cargo detection function above the pallet during the pallet identification and positioning process during the AGV terminal operation, effectively improving the safety application of AGV terminal operations in scenarios where people and vehicles are mixed, and expanding the application scenarios of AGV.

[0136] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that any technician familiar with this technical field can still modify the technical solutions recorded in the aforementioned embodiments within the technical scope disclosed in the present application, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be covered by the protection scope of the present application.

Claims

1. A method for detecting over-cargo, characterized in that: include: Get real-time pallet images and depth data; Input the real-time pallet image and depth data into the trained pallet recognition model to obtain the region of interest; Segment the region of interest to obtain the tray jack segmentation region; According to the segmentation area of ​​the pallet socket and the physical structure characteristics of the pallet, the pixel coordinates of the center point of the pallet support are calculated; The yaw angle of the pallet relative to the center of the vehicle body is calculated based on the pixel coordinates of the center point of the pallet support and the depth data; The cargo over-pallet detection range is calculated based on the pixel coordinates of the center point of the pallet pillar, the yaw angle of the pallet relative to the center of the vehicle body, and the preset over-pallet threshold; Compare the preset minimum range threshold with the cargo over-pallet detection range to determine whether the pallet is over-palletized.

2. The method for detecting over-consignment of goods according to claim 1, characterized in that: The method of calculating the pixel coordinates of the center point of the pallet support pillar according to the segmented area of ​​the pallet insertion hole and the physical structure characteristics of the pallet includes: Performing filtering processing on the segmented pallet image to obtain the pallet end surface area, the pallet support area and the insertion hole area; According to the circumscribed rectangle of the socket area, the pixel coordinates of the four corner points of the middle pillar area in the pillar area of ​​the tray and the pixel coordinates of the center point of the middle pillar are obtained; According to the physical dimensions of the left and right pillars of the tray, the pixel coordinates of the center points of the left pillar and the right pillar are obtained; the pixel coordinates of the center point of the middle pillar, the pixel coordinates of the center point of the left pillar and the pixel coordinates of the center point of the right pillar constitute the pixel coordinates of the center point of the right pillar.

3. The method for detecting over-cargo according to claim 2, characterized in that: The yaw angle of the pallet relative to the center of the vehicle body is calculated based on the pixel coordinates of the center point of the pallet support and the depth data, including: The yaw angle of the end face of the pallet in the camera coordinate system is calculated according to the pixel coordinates of the center point of the pallet support and the depth data; The yaw angle of the end surface of the pallet in the camera coordinate system is converted into the yaw angle of the end surface of the pallet in the vehicle body coordinate system through the calibrated rotation matrix of the camera coordinate system relative to the vehicle body coordinate system.

4. The method for detecting over-consignment of goods according to claim 3, characterized in that: The yaw angle of the end face of the pallet in the camera coordinate system is calculated based on the pixel coordinates of the center point of the pallet support and the depth data. The calculation formula is as follows: Where, θ' is the yaw angle of the tray relative to the camera, d p3 is the depth value of the left pillar center point P3, d p4 is the depth value of the center point P4 of the right pillar; p xP3 is the pixel horizontal coordinate of the center point P3 of the left pillar, p xP4 is the pixel horizontal coordinate of the center point P4 of the right pillar, f x is the horizontal pixel focal length of the camera, c x is the horizontal coordinate of the camera optical center in the pixel coordinate system.

5. The method for detecting over-consignment of goods according to claim 3, characterized in that: The yaw angle of the end face of the pallet in the camera coordinate system is converted to the yaw angle of the end face of the pallet in the vehicle coordinate system through the rotation matrix of the calibrated camera coordinate system relative to the vehicle coordinate system. The calculation formula is: θ=arctan2(-r31,r11)+θ' θ' is the yaw angle of the end surface of the pallet in the camera coordinate system, θ is the yaw angle of the end surface of the pallet in the vehicle coordinate system, r31 is the cosine value of the Z axis of the vehicle coordinate system on the X axis of the camera coordinate system, and r11 is the cosine value of the X axis of the vehicle coordinate system on the X axis of the camera coordinate system.

6. The method for detecting over-consignment of goods according to claim 1, characterized in that: The cargo over-pallet detection range includes: the front of the pallet, the left side of the pallet, and the right side of the pallet.

7. The method for detecting over-consignment of goods according to claim 6, characterized in that: The cargo over-carrying detection range is calculated according to the pixel coordinates of the center point of the pallet pillar, the yaw angle of the pallet relative to the center of the vehicle body and the preset over-carrying threshold, including: over-carrying point detection on the front of the pallet, over-carrying point detection on the left side of the pallet, over-carrying point detection on the right side of the pallet, statistical filtering of over-carrying points and image filtering processing.

8. The method for detecting over-consignment of goods according to claim 7, characterized in that: The statistical filtering of super-support points and image filtering processing include: Traverse the over-support detection range; The corrosion post-processing of the over-support detection range is carried out to remove the noise data and filter it into discrete areas using the preset minimum range threshold; Expand the small target area in the super-support detection range and restore it to the original cross-section of the super-support object; Find the edge information of the super-support object, segment the image and extract the object features, and find the super-support detection range. The bounding rectangle of the super-support object; traverse the bounding rectangle of the super-support object and filter it again using the preset minimum range threshold. Check whether there is over-palletizing of goods on the pallet.

Images