An unmanned forklift pallet picking method and system based on center of gravity correction posture

Abstract

The present invention relates to the field of pallet detection and positioning technology, and in particular to a method and system for picking up pallets by an unmanned forklift based on a center of gravity correction posture, comprising S1: acquiring an RGB feature image, defining a pallet detection category template, and establishing a pallet posture estimation model; S2: acquiring a weighing sensor acquisition value in real time; S3: performing a final value consistency analysis; S4: performing a dynamic fluctuation stability analysis; S5: determining the formal picking motion plan of the unmanned forklift based on the picking safety factor and the dynamic fluctuation stability analysis; S6: adjusting the posture of the center of gravity of the pallet picked up object projected on the picking plane coordinate system based on the moment balance principle, and generating the optimal target picking posture of the current pallet scene; S7: the unmanned forklift system generates a planned path for the final picking process based on the optimal target picking posture, and completes the picking; the present application fully considers the center of gravity structure stability problem of pallet object picking to meet the needs of automated unmanned picking operations.

Description

Technical Field

[0001] The present invention relates to the technical field of pallet detection and positioning, and in particular to an unmanned forklift pallet picking method and system based on center of gravity correction posture. Background Art

[0002] Unmanned forklift equipment is used in modern warehousing scenarios to realize the automated transfer of goods in warehouses, and pallet positioning and picking have become important aspects of forklift operations. Unmanned forklift pallet picking methods are usually divided into two categories: one is fixed point picking, which is to perform attachment fork arm picking at a fixed position without guidance by stepping on the point; the other is pallet positioning picking guided by machine vision, which detects and locates pallets by assembling a depth camera on the forklift gantry to collect images.

[0003] Traditional unmanned forklift picking methods use pallets as the only target detection object. Pallet target detection under the guidance of machine vision usually only considers the cross-sectional structure of the pallet to be measured. In the overloaded forklift scenario, the center of gravity structure of the goods carried on the pallet is often not in the central area. There are common picking scenarios with center of gravity offset. The method using traditional pallet posture detection as a reference has defects in application. The traditional picking method fails to consider the impact of the center of gravity structure of the goods carried on the upper layer on the forklift picking. When the upper cargo is stacked asymmetrically or the center of gravity is offset, using the center posture of the pallet cross-section to be measured as the only reference standard for picking has a great safety impact. Especially in the heavy-loaded unmanned forklift scenario, the movement process can easily cause the overall pallet picking posture to be unstable, which may cause the unmanned forklift to collapse during the transfer process. The collapse of tons of heavy loads during the forklift transportation will have a major safety accident impact. Solving the problem of safe and stable pallet picking by unmanned forklifts has become a practical application problem that needs to be overcome. Summary of the invention

[0004] The object of the present invention is to provide an unmanned forklift pallet picking method and system based on center of gravity correction posture to avoid cargo collapse in an unmanned forklift picking scenario.

[0005] In order to achieve the above object, the present invention adopts such technical solution:

[0006] A method for picking up a pallet by an unmanned forklift based on a center of gravity correction posture comprises the following steps performed in sequence:

[0007] S1: The unmanned forklift obtains the RGB feature image of the target pallet cargo, establishes a pallet pose estimation model, and defines a pallet detection category template. The pallet pose estimation model estimates the pallet pose based on the RGB feature image and the pallet detection category template, and outputs an estimated pallet pose. The unmanned forklift obtains the estimated pallet pose and initializes the pre-pickup motion planning. The unmanned forklift is controlled to pre-pick up the pallet cargo at the initial point according to the pre-pickup motion planning.

[0008] S2: The unmanned forklift is equipped with 4 weighing sensors, namely C1, C2, C3 and C4. C1 and C3 are installed one by one on the two fork arms of the unmanned forklift, respectively, near the front end. C2 and C4 are installed one by one on the two fork arms of the unmanned forklift, respectively, near the rear end. In the reference coordinate system of the picking plane, the coordinates of each weighing sensor are (x C1 ,y C1 ,0),(x C1 ,y C2 ,0),(x C3 ,y C1 ,0),(x C3 ,y C2 , 0),

[0009] When the unmanned forklift pre-picks up the pallet cargo at the initial point, the real-time collection values ​​of the four weighing sensors are: M1, M2, M3 and M4 ;

[0010] S3: Perform a final value consistency analysis to determine whether the cargo pallet pre-picked at the initial point meets the safety operation requirements. The specific judgment steps of the final value consistency analysis are as follows:

[0011] S3-1: Use the following formula to calculate the allowable deviation error ratio of the current pre-picked cargo pallet :

[0012] ;

[0013] in, M Indicates the mass value of the pallet cargo, in kg. M=M1+M2+M3+M4, M1 and M3 Collect values ​​for the weighing sensor at the front end of the fork arm of the unmanned forklift ，M2 and M4 Collect values ​​for the weighing sensor at the rear end of the forklift’s fork arm, and M1 and M2 Collect values ​​for the load cells on the same fork arm, M3 and M4 Collect values ​​for the load cells on the same fork arm;

[0014] The constraints for picking the safety factor A are as follows:

[0015] ;

[0016] in, Indicates the value collected by the weighing sensor and , Indicates the value collected by the weighing sensor and ;

[0017] Define the pick-up safety factor A=0, which means that the collected value of the weighing sensor meets the final value consistency;

[0018] The picking safety factor A=1 is defined, indicating that the pallet cargo has a longitudinal center of gravity offset;

[0019] The picking safety factor A=2 is defined, indicating that the pallet cargo has a lateral center of gravity offset;

[0020] S4: Perform dynamic fluctuation stability analysis, obtain n groups of collected data of the 0-point calibration state, dynamic fluctuation state, final value holding state and pallet release stage of the collected values ​​of the four weighing sensors respectively, perform consistency test of ICC intra-group correlation coefficient based on the obtained n groups of collected data, and judge the dynamic fluctuation stability of the collected values ​​of the weighing sensors in the group based on the data performance of ICC;

[0021] The specific steps of dynamic fluctuation stability analysis are as follows:

[0022] S4-1: In a pre-pickup process, n groups of collected data are divided into a 0-point calibration phase, a dynamic fluctuation phase, and a final value data phase;

[0023] S4-2: The sampling data of the 0-point calibration stage, dynamic fluctuation stage and final value data stage were analyzed by ICC intraclass correlation coefficient method;

[0024] S4-3: Select the two-way mixed effects model and use the absolute consistency metric to calculate the ICC intraclass correlation coefficient. The calculation formula is as follows:

[0025] ICC= ;

[0026] Among them, MSr is the mean square of row variables, that is, the mean square of sensor variables, MSe is the mean square of error, MSc is the mean square of column variables, that is, the mean square of sampled weighing data variables, and n is the number of collected data groups;

[0027] Define the pallet picking scenario , indicating that the data consistency is excellent;

[0028] Define the pallet picking scenario , indicating that the data consistency is good;

[0029] Define the pallet picking scenario , indicating that the data consistency is moderate or poor;

[0030] S5: When the pick-up safety factor A=0 and In the case of a pre-pickup motion plan, the pre-pickup motion plan is converted into a formal pickup motion plan, and the unmanned forklift picks up the pallet cargo according to the formal pickup motion plan;

[0031] When the pick-up safety factor A=1 or A=2, and When the unmanned forklift control system obtains the warehouse map, and determines whether the flatness of the operating environment meets the international definition of the flatness of the operating environment of the automated guided vehicle according to the warehouse map. If the definition of the environmental flatness is met, the pre-pickup motion plan is converted into a formal pickup motion plan, and the unmanned forklift picks up the pallet goods according to the formal pickup motion plan. Otherwise, step S6 is executed;

[0032] When the pick-up safety factor A=1 or A=2, and When , execute step S6;

[0033] S6: Calculate the coordinates of the center of gravity of the pallet cargo in the picking plane coordinate system based on the moment balance principle , by coordinates Generate the optimal target picking posture for the current pallet scene ;

[0034] Generate optimal target picking pose The specific steps are as follows:

[0035] S6-1: Based on the values ​​collected by the four weighing sensors, due to the moment balance, the pallet obtains the coordinates of the center of gravity in the picking plane coordinate system As shown in the following formula:

[0036] ;

[0037] ;

[0038] S6-2: Generate the optimal target picking posture based on the projection coordinate value of the pallet cargo center of gravity on the picking plane as the reference point for the unmanned forklift to pick up the target pallet :

[0039] ;

[0040] Among them, y2 is the ordinate value of the center of the bracket in the middle of the pallet detection section in the picking plane coordinate system, L S is the fork arm length of the unmanned forklift, d is the fork entry depth, d=L P -b, where L p is the longitudinal length of the pallet, and b is the effective detection distance limit of the obstacle avoidance sensor;

[0041] S7: The unmanned forklift system calculates the optimal target picking posture according to step S6 Generate the planned path for the final picking process and complete the picking.

[0042] Preferably, in step S1, the specific steps of estimating the position and posture of the pallet are as follows:

[0043] S1-1: The pallet pose estimation model defines a pallet detection category template through a top-down spatial distribution relationship. The pallet detection category template includes four types of targets: cargo, pallet body, pallet hole position and support plane. The pallet detection category template characterizes the top-down distribution structure and size relationship of the four types of targets, and uses RGB color configuration to define each category of the pallet detection category template to obtain an RGB feature image of the cargo pallet;

[0044] S1-2: Use the RGB feature image as the input of the image classifier and convert the RGB feature image into an HSI color gamut image;

[0045] S1-3: The feature spaces of cargo, pallet body, pallet holes and support plane are represented as A, B, C, D respectively, and the sample feature vector [x ij, c ij ] T It is expressed as follows:

[0046] [x ij, c ij ] T =[R,G,B,H,S,c ij ] T ;

[0047] c ij = ;

[0048] H= ;

[0049] S= ;

[0050] Among them, i is the pixel row, j is the pixel column, and c ij is the category matrix element, R is the red grayscale value in the RGB three-color channel, G is the green grayscale value in the RGB three-color channel, B is the blue grayscale value in the RGB three-color channel, H is the hue in the HSI color gamut model, S is the saturation in the HSI color gamut model, x ij is the RGB channel feature information, T is the transposed representation of the matrix;

[0051] Use the SVM model to process and classify the four types of targets in the RGB feature image, and match the classification output with the defined pallet detection category template;

[0052] S1-4: A grid compression matching processing method is used to establish a grid compression unit according to the pallet detection category template, so that the compression unit is a single category, the grid size is w, and the compressed pallet detection category template is expressed as follows:

[0053] ;

[0054] ;

[0055] ;

[0056] in, represents the element in the i-th row and j-th column of the template matrix, represents the element in the i-th row and j-th column of the category matrix, For the nth category in Quantity within the unit;

[0057] S1-5: Locate the position of the pallet by using the window sliding method, and calculate the similarity between the pallet detection category template and the window:

[0058] ;

[0059] ;

[0060] ;

[0061] in, is the matching degree of the i-th and j-th units, is the threshold ratio of matching degree;

[0062] Using a correlation matching algorithm to perform similarity matching on a result of subtracting the mean from the pallet detection category template and a result of subtracting the mean from the original image;

[0063] S1-6: When the similarity is greater than the preset similarity threshold, the pallet posture corresponding to the window is captured, and the NMS non-maximum suppression method is used to process the output problem of multiple pallet postures when outputting the captured pallet postures;

[0064] S1-7: Output the center point coordinates p1 (x1, y1, z1), p2 (x2, y2, z2), and p3 (x3, y3, z3) of the left, middle, and right brackets of the captured pallet posture according to the similarity measurement, and calculate the estimated value of the angle offset at time t To determine the pallet position:

[0065] ;

[0066] Among them, n represents the number of points, here are three points p1, p2, and p3. and Indicates the corresponding bracket center point coordinate value;

[0067] Processing is done by filtering the average of multiple frames of data:

[0068] According to the estimated value of the angle offset at time t Calculate the plane coordinates of the pallet posture center at time t :

[0069] ;

[0070] .

[0071] Preferably, in step S2, the pre-picking step at the initial point is as follows:

[0072] S2-1: According to the pallet position output in step S1, the unmanned forklift receives the route to be traveled generated in the RCS dispatching system;

[0073] S2-2: The unmanned forklift drives to the initial point according to the route to be traveled;

[0074] S2-3: After each weighing sensor is calibrated and zeroed, the unmanned forklift controls the fork arm to lift for pre-pickup and lift the pallet to a preset height.

[0075] An unmanned forklift pallet picking system based on center of gravity correction posture includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any of the above methods is implemented.

[0076] By adopting the aforementioned design scheme, the beneficial effects of the present invention are as follows: the present application proposes an unmanned forklift pallet picking method based on center of gravity correction posture, which fully considers the stability problem of the center of gravity structure of pallet picking to meet the needs of automated unmanned picking operations, and not only gets rid of the situation of relying entirely on experience in the manual operation scenario of pallet picking, but also solves the limitation of the existing unmanned forklift automatic picking system that only uses the pallet body as a reference during execution, and provides a method to deal with the problem of lack of consideration of the stability of the center of gravity structure of the overall transportation of the pallet, and solves the stability problem of unmanned heavy-load picking by industrial unmanned forklifts. The method not only accurately locates the pallet posture, but also ensures the transportation safety and stability of the picking movement process. In the scenario of limited computing power resources on the local end, the algorithm execution process can be completed with a basic computing cost, ensuring that the unmanned forklift system can operate efficiently and autonomously in an industrial production environment, effectively improving the level of automation in industrial warehousing and logistics, and is of significance to the overall industry application development. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1A schematic diagram is defined for a pallet positioning and picking system based on object-level center of gravity positioning of the present invention;

[0078] Figure 2 The category template and grid compression structure diagram of the pallet recognition detection of the present invention;

[0079] Figure 3 It is a segmented state data coordinate diagram of the sensor weighing curve of the present invention;

[0080] Figure 4 This is a flow chart of pallet detection and picking by an unmanned forklift of the present invention. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only partial embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0082] A pallet picking method for unmanned forklift based on center of gravity correction posture, such as Figure 4 As shown, the steps include the following steps performed in sequence:

[0083] S1: The unmanned forklift obtains the RGB feature image of the target pallet cargo, establishes a pallet pose estimation model, and defines a pallet detection category template. The pallet pose estimation model estimates the pallet pose based on the RGB feature image and the pallet detection category template, and outputs the estimated pallet pose. The unmanned forklift obtains the estimated pallet pose and initializes the pre-pickup motion planning. According to the pre-pickup motion planning, the unmanned forklift is controlled to pre-pick up the pallet cargo at the initial point.

[0084] In step S1, the specific steps of the pallet posture estimation are as follows:

[0085] S1-1: The pallet pose estimation model defines the pallet detection category template through the top-down spatial distribution relationship. The pallet detection category template includes four types of targets: cargo, pallet body, pallet hole position and support plane. The pallet detection category template represents the top-down distribution structure and size relationship of the four types of targets, and uses RGB color configuration to define each category of the pallet detection category template to obtain the RGB feature image of the cargo pallet.

[0086] S1-2: Use RGB feature images as input to the image classifier. The target categories are defined as 4, corresponding to the upper cargo, the middle pallet body and pallet holes, and the lower support plane. The support plane in this article refers to the ground on which the pallet is placed. Convert the RGB feature image to an HSI color gamut image so that image processing only needs to be done in one dimension to reduce the computational cost and improve the image processing speed.

[0087] S1-3: The feature spaces of cargo, pallet body, pallet holes and support plane are represented as A, B, C, D respectively, and the sample feature vector [x ij, c ij ] T It is expressed as follows:

[0088] [x ij, c ij ] T =[R,G,B,H,S,c ij ] T ;

[0089] c ij = ;

[0090] H= ;

[0091] S= ;

[0092] Among them, i is the pixel row, j is the pixel column, and c ij is the category matrix element, R is the red grayscale value in the RGB three-color channel, G is the green grayscale value in the RGB three-color channel, B is the blue grayscale value in the RGB three-color channel, H is the hue in the HSI color gamut model, S is the saturation in the HSI color gamut model, x ij is the RGB channel feature information, T is the transposed representation of the matrix;

[0093] Use the SVM model to process and classify the four types of targets in the RGB feature image, and match the classification output with the defined pallet detection category template;

[0094] S1-4: Using a grid compression matching processing algorithm, a grid compression unit is established according to the pallet detection category template, so that the compression unit is a single category, the grid size is w, and the compressed pallet detection category template is expressed as follows:

[0095] ;

[0096] ;

[0097] ;

[0098] in, represents the element in the i-th row and j-th column of the template matrix, represents the element in the i-th row and j-th column of the category matrix, For the nth category in Quantity within the unit;

[0099] The category matrix c is transformed into ij The pallet detection category template is compressed to obtain small-size data that retains key category information, thereby speeding up the matching speed of the pallet detection category template.

[0100] S1-5: Locate the pallet posture through the window sliding method and calculate the similarity between the pallet detection category template and the window: ;

[0101] ;

[0102] ;

[0103] in, is the matching degree of the i-th and j-th units, is the threshold ratio of matching degree;

[0104] A correlation matching algorithm is used to perform similarity matching on the result of subtracting the mean from the pallet detection category template and the result of subtracting the mean from the original image; the correlation matching algorithm is helpful in solving the influence caused by the different brightness between the template and the image.

[0105] S1-6: When the similarity is greater than the preset similarity threshold, the pallet posture corresponding to the window is captured, and the NMS non-maximum suppression method is used to process the output problem of multiple pallet postures when outputting the captured pallet postures;

[0106] S1-7: Output the center point coordinates p1 (x1, y1, z1), p2 (x2, y2, z2), and p3 (x3, y3, z3) of the left, middle, and right brackets of the captured pallet posture according to the similarity measurement, and calculate the estimated value of the angle offset at time t To determine the pallet position:

[0107] ;

[0108] Among them, n represents the number of points, here are three points p1, p2, and p3. and Indicates the corresponding bracket center point coordinate value;

[0109] Since the calculation structure error of a single frame is relatively large, when estimating the plane coordinates and angle offset of the pallet posture center, a filtering method of averaging multiple frames of data is used to process it, so that the accuracy is higher. The filtering method of averaging multiple frames of data is used for processing:

[0110] According to the estimated value of the angle offset at time t Calculate the plane coordinates of the pallet posture center at time t :

[0111] ;

[0112] ;

[0113] In this embodiment, Figure 1 As shown, the unmanned forklift is equipped with 4 weighing sensors (C1, C2, C3, C4), 1 RGBD camera and 2 obstacle avoidance sensors. The above sensors and cameras are all conventional electronic devices.

[0114] The data collected by the four weighing sensors are processed by the transmitter for signal power amplification and the data is output. In this embodiment, each unmanned forklift AGV is equipped with a conventional transmitter, which converts the electrical signal of the weighing sensor into a standard industrial signal output after isolation, and uses a differential amplifier circuit and a high-precision signal processing chip for processing, and transmits the signal to the serial port server through RS485 communication to realize data communication.

[0115] The camera optical axis of the RGBD camera is parallel to the ground. The RGBD camera is calibrated to obtain the camera intrinsic and extrinsic distortion parameters.

[0116] An RGBD camera is used to collect RGB feature images of pallet cargo in industrial scenes, and the collected RGB feature images are distorted according to calibration parameters.

[0117] Two obstacle avoidance sensors are installed one by one at the ends of the fork teeth of the two fork arms of the unmanned forklift, forming a sensing area to handle safety detection for fork picking.

[0118] S2: The unmanned forklift is equipped with 4 weighing sensors, namely C1, C2, C3 and C4. C1 and C3 are installed one by one on the two fork arms of the unmanned forklift, respectively, near the front end. C2 and C4 are installed one by one on the two fork arms of the unmanned forklift, respectively, near the rear end. In the reference coordinate system of the picking plane, the coordinates of each weighing sensor are (x C1 ,y C1 ,0),(x C1 ,y C2 ,0),(x C3 ,y C1 ,0),(xC3 ,y C2 , 0),

[0119] When the unmanned forklift pre-picks up the pallet cargo at the initial point, the real-time collection values ​​of the four weighing sensors are: M1, M2, M3 and M4 ;

[0120] In step S2, the pre-pickup steps at the initial point are as follows:

[0121] S2-1: According to the pallet position output in step S1, the unmanned forklift receives the route to be traveled generated in the RCS dispatching system;

[0122] S2-2: The unmanned forklift drives to the initial point according to the route to be traveled;

[0123] S2-3: After the weighing sensors are calibrated and zeroed, the unmanned forklift controls the fork arm to lift for pre-pickup and lift the pallet to a preset height, which is lower than the lifting height during normal operation. The pallet has a ground clearance of 10 mm.

[0124] The preset height in this embodiment is 10 mm.

[0125] After receiving the position of the pallet, the path planner of the unmanned forklift generates a route to be traveled. The dispatching system releases the traffic control signal to control the movement of the unmanned forklift through the motion controller of the unmanned forklift. During the movement, planning tasks are allocated according to task priorities and obstacle avoidance safety requests. After the unmanned forklift arrives at the initial target position for pallet picking, the attachment actuator controls the fork arm to lift and pre-pick up the initial point.

[0126] S3: Perform a final value consistency analysis to determine whether the cargo pallet pre-picked at the initial point meets the safety operation requirements. The specific judgment steps of the final value consistency analysis are as follows:

[0127] When the unmanned forklift is stationary at a low lifting position and the weighing sensor is in a data holding state, a final value check is performed on M1, M2, M3, and M4 based on the characteristics of the time series data.

[0128] S3-1: Use the following formula to calculate the allowable deviation error ratio of the current pre-picked cargo pallet :

[0129] ;

[0130] in, M Indicates the mass value of the pallet cargo. M=M1+M2+M3+M4, M1 and M3 Collect values ​​for the weighing sensor at the front end of the fork arm of the unmanned forklift ，M2 and M4Collect values ​​for the weighing sensor at the rear end of the forklift’s fork arm, and M1 and M2 Collect values ​​for the load cells on the same fork arm, M3 and M4 Collect values ​​for the load cells on the same fork arm;

[0131] The constraints for picking the safety factor A are as follows:

[0132] ;

[0133] in, Indicates the value collected by the weighing sensor and , Indicates the value collected by the weighing sensor and ;

[0134] Define the pick-up safety factor A=0, which means that the collected value of the weighing sensor meets the final value consistency;

[0135] The picking safety factor A=1 is defined, indicating that the pallet cargo has a longitudinal center of gravity offset;

[0136] The picking safety factor A=2 is defined, indicating that the pallet cargo has a lateral center of gravity offset;

[0137] S4: Perform dynamic fluctuation stability analysis, obtain n groups of collected data of the 0-point calibration state, dynamic fluctuation state, final value holding state and pallet release stage of the collected values ​​of the four weighing sensors respectively, perform consistency test of ICC intra-group correlation coefficient based on the obtained n groups of collected data, and judge the dynamic fluctuation stability of the collected values ​​of the weighing sensors in the group based on the data performance of ICC;

[0138] The specific steps of dynamic fluctuation stability analysis are as follows:

[0139] S4-1: In a pre-picking process, n groups of collected data are divided into a 0-point calibration stage, a dynamic fluctuation stage, a final value data stage and a tray release stage; the tray release stage is obtained here only to express the integrity of the data, and the final value data stage does not need to be applied later.

[0140] S4-2: The sampling data of the 0-point calibration stage, dynamic fluctuation stage and final value data stage were analyzed by ICC intraclass correlation coefficient method;

[0141] S4-3: Select the two-way mixed effects model and use the absolute consistency metric to calculate the ICC intraclass correlation coefficient. The calculation formula is as follows:

[0142] ICC= ;

[0143] Among them, MSr is the mean square of row variables, that is, the mean square of sensor variables, MSe is the mean square of error, MSc is the mean square of column variables, that is, the mean square of sampled weighing data variables, and n is the number of collected data groups;

[0144] Define the pallet picking scenario , indicating that the data consistency is excellent;

[0145] Define the pallet picking scenario , indicating that the data consistency is good;

[0146] Define the pallet picking scenario , indicating that the data consistency is moderate or poor;

[0147] S5: When the pick-up safety factor A=0 and In the case of a pre-pickup motion plan, the pre-pickup motion plan is converted into a formal pickup motion plan, and the unmanned forklift picks up the pallet cargo according to the formal pickup motion plan;

[0148] When the pick-up safety factor A=1 or A=2, and When the warehouse map is obtained, the control system of the unmanned forklift obtains the warehouse map, and determines whether the flatness of the operating environment meets the definition of the international flatness of the operating environment of the automated guided vehicle according to the warehouse map. If the definition of the environmental flatness is met, the pre-pickup motion plan is converted into a formal pick-up motion plan, and the unmanned forklift picks up the pallet goods according to the formal pick-up motion plan, otherwise, step S6 is executed; in this embodiment, the definition of the international flatness of the operating environment of the automated guided vehicle refers to the indoor flat ground area meeting the definition of ground flatness in the "GB / T 20721-2022 General Technical Conditions for Automated Guided Vehicles", based on the national standard ground meeting the benchmark range , flatness , height difference Inside;

[0149] When the pick-up safety factor A=1 or A=2, and When , execute step S6;

[0150] S6: Calculate the coordinates of the center of gravity of the pallet cargo in the picking plane coordinate system based on the moment balance principle , by coordinates Generate the optimal target picking posture for the current pallet scene ;

[0151] Generate optimal target picking pose The specific steps are as follows:

[0152] S6-1: Based on the values ​​collected by the four weighing sensors, due to the moment balance, the pallet obtains the coordinates of the center of gravity in the picking plane coordinate system As shown in the following formula:

[0153] ;

[0154] ;

[0155] S6-2: Generate the optimal target picking posture based on the projection coordinate value of the pallet cargo center of gravity on the picking plane as the reference point for the unmanned forklift to pick up the target pallet :

[0156] ;

[0157] Among them, y2 is the ordinate value of the center of the bracket in the middle of the pallet detection section in the picking plane coordinate system, L S is the fork arm length of the unmanned forklift, d is the fork entry depth, d=L P -b, where L p is the longitudinal length of the pallet, and b is the effective detection distance limit of the obstacle avoidance sensor;

[0158] S7: The unmanned forklift system calculates the optimal target picking posture according to step S6 Generate the planned path for the final picking process and complete the picking.

[0159] This embodiment also provides a system for implementing the above method.

[0160] An unmanned forklift pallet picking system based on center of gravity correction posture includes a memory and a processor. The memory stores a computer program, and the processor implements any of the above methods when executing the computer program.

[0161] The following specific embodiments are used to further illustrate the unmanned forklift pallet picking method based on center of gravity correction posture of the present invention.

[0162] like Figure 1 As shown, the reference coordinate system for picking up is established, and the unmanned forklift Fork arm length ,width , 4 weighing sensors C1, C2, C3, C4 are symmetrically distributed on the left and right fork arms, and the lateral distribution distance of the sensors is , longitudinal distribution distance , RGB-D camera is installed at the center point of the attachment .

[0163] The pallet size is defined as , tray support width The center of the pallet in the attachment picking plane coordinate system is ,dotted line The middle area formed is the pore channel on the left side of the tray. The spatial constraints of the adjustable fork range of the attachment are defined together. This spatial constraint is effective for the function of the obstacle avoidance sensor. The left, middle and right center points of the pallet bracket to be observed are The projection coordinates of the center of gravity of the pallet on the picking plane are .

[0164] Establish a pallet pose estimation model, such as Figure 2 As shown in the figure, a pallet detection category template for pallet detection and identification is defined. The content of the pallet detection category template includes goods, pallet body, pallet holes and supporting plane. Its spatial relationship is a fixed distribution structure from top to bottom. Establishing the category template of the pallet and the surrounding environment through spatial constraints can not only improve the accuracy of pallet recognition, but also effectively expand the detection distance.

[0165] like Figure 2 As shown, the pallet detection category template represents the cargo-the area above the illustration, the pallet body-the middle area of ​​the illustration, the pallet holes-the two box areas in the middle area of ​​the illustration and the support plane-the area below the illustration, their spatial distribution and pallet size information. The four categories of spatial relationships defined in the pallet detection category template are a fixed distribution structure from top to bottom, in which each parameter represents the size of the pallet and the pallet detection category template in the image.

[0166] Using RGB features as the input of the image classifier, in order to reduce the computational cost, the RGB format is converted to the HSI color domain to separate the color and brightness information, so that the image processing only needs to be done in a single dimension, which improves the image processing speed.

[0167] The four-category feature space of cargo, pallet body, pallet holes, and support plane is represented as A, B, C, and D.

[0168] The SVM model is used to process and classify the four types of targets in the RGB feature image and match them with the defined pallet detection category template. In order to solve the problem of the partial occlusion or excessive tilt of the target pallet on the matching efficiency of the pallet detection category template, a grid compression matching processing method is proposed to improve the anti-interference ability and algorithm stability; because the size information of the template pallet detection category template and the image directly affects the operation efficiency of the matching algorithm, such as Figure 2 As shown, according to the size of the pallet detection category template, a grid compression unit is established and the compression unit is made into a single category. The grid size is w. The compressed pallet detection category template is represented as the template matrix and the category matrix ,This method compresses the category matrix and template to obtain small-size data that retains key category information,,speeding up the category template matching speed.

[0169] Calculate the similarity between the template and the window by using the window sliding method , locate the pallet pose. When the similarity is greater than the minimum threshold, the pallet detection target is captured, the pallet candidate position is output, and the NMS non-maximum suppression method is used to handle multiple pallet position output problems. The threshold is adjusted according to the complexity of the working environment of the unmanned forklift camera.

[0170] The main parameters of the pallet posture include the coordinates of the center points of the left, middle and right brackets of the target pallet (x1, y1, z1), (x2, y2, z2), (x3, y3, z3) and the angle offset ;like Figure 1 As shown, the coordinates of the left, middle, and right brackets of the pallet on the picking plane are represented as p1(x1, y1), p2(x2, y2), and p3(x3, y3). The estimated angle of the pallet by the least squares fitting in the RGB-D camera coordinate system at time t is calculated. , the calculation accuracy is improved through multi-frame mean filtering.

[0171] The core components involved in a complete picking process of the unmanned forklift system include motion controller, path planner and fork arm actuator. The path planner generates the target point and pre-planned route after capturing the pallet posture. The motion controller controls the steering wheel movement and steering movement of the unmanned forklift after receiving the traffic control release signal from the RCS dispatching system. The fork arm actuator controls the lifting and lowering of the fork arm of the unmanned forklift when the unmanned forklift reaches the pallet picking target point.

[0172] The pallet posture is output to the unmanned forklift, and the unmanned forklift system pre-picks the pallet. The path planner generates the route to be traveled based on the captured pallet posture data. Then the motion controller controls the movement of the unmanned forklift according to the traffic control signal released by the RCS. During the movement, the planning tasks are allocated according to the task priority and obstacle avoidance safety request. After reaching the initial target position for pallet picking, the fork arm actuator controls the fork arm to lift and pre-pick up the initial point.

[0173] During the initial point pre-pickup process, the four weighing sensors C1, C2, C3, and C4 collect data in real time. The weighing sensors used are strain type, which are key components of electronic scales that convert gravity into electrical signals. The weighing data output of each sensor is processed by the transmitter power amplifier. In the reference coordinate system of the picking plane, the distribution distance Lc=x between C1 and C3, C2 and C4 C3 -x C1 , the sensor distribution distance between C1 and C2, C3 and C4 is Wc=y C1 -y C2 .

[0174] After the sensors are calibrated and zeroed, unmanned forklifts can pre-pick up the items. The total mass of the pallet is M=M1+M2+M3+M4.

[0175] Compare and analyze the consistency and dynamic fluctuation consistency of the 4-point weighing data:

[0176] Dynamic fluctuation consistency analysis,In a pre-pickup process, the weighing sensor curve data is divided into four state stages, such as Figure 3 As shown, during t0~t1, the period when the weighing data curve is approximately fitted with f(t)=0, k=0 is the 0-point calibration state segment; during t1~t2, the period when f(t)<0, k<0 is the dynamic fluctuation state segment; during t2~t3, the period when f(t)=Mn<0, k=0 is the final value data holding state segment; during t3~t4, the period when f(t)<0, k>0 is the tray release state segment.

[0177] The ICC intragroup correlation coefficient method was used to check data consistency. The sensor sampling rate was set. During a pre-pickup process, four weighing sensors sampled n groups of state data. The tested samples mainly contained three parts of data, namely, 0-point calibration state data, dynamic fluctuation state data, and final value holding state data.

[0178] A two-way mixed effects model was selected because each group of sampling data was not related in an additive manner, and the values ​​of the same group of test samples were theoretically similar, and the absolute consistency measure could be used to calculate the ICC intragroup correlation coefficient.

[0179] The four weighing sensors were grouped and analyzed by ICC intragroup correlation coefficient to test the consistency of weighing data within the group.

[0180] C1 and C2, C1 and C3, C4 and C2, C4 and C3 were independently grouped for ICC intragroup correlation coefficient analysis to test the consistency of the weighing data within the group of M1 and M2, M1 and M3, M4 and M2, M4 and M3.

[0181] According to the consistency analysis of the ICC group data, the ICC 1-2 , ICC 1-3 , ICC 4-2 , ICC 4-3 The data value determines whether the picking condition is met.

[0182] When the pick-up safety factor A=0 and In the case of a pre-pickup motion plan, the pre-pickup motion plan is converted into a formal pickup motion plan, and the unmanned forklift picks up the pallet cargo according to the formal pickup motion plan;

[0183] When the pick-up safety factor A=1 or A=2, and When the unmanned forklift control system obtains the warehouse map, it determines whether the operating environment flatness meets the international definition of the operating environment flatness of the automated guided vehicle. The definition of indoor flat ground area meets the definition of ground flatness in GB / T 20721-2022 General Technical Requirements for Automated Guided Vehicles. The national standard ground meets the benchmark range. , flatness , height difference If the environment flatness definition is met, the pre-pickup motion plan is converted to the formal pickup motion plan, and the unmanned forklift picks up the pallet goods according to the formal pickup motion plan. Otherwise, the optimal pickup target point is generated according to the moment balance principle.

[0184] When the pick-up safety factor A=1 or A=2, and When , the optimal pickup target point is generated according to the moment balance principle;

[0185] According to the moment balance principle, calculate the X and Y axis coordinates of the center of gravity of the pallet load projected on the reference coordinate system of the picking plane , from which the coordinate value of the center of gravity of the pallet load on the picking plane projection is obtained. The fork entry depth d of the unmanned forklift AGV fork arm is related to the attachment length Ls and the longitudinal dimension length Lp of the pallet. Usually, there is a significant difference, Ls>Lp, d=L P -b, where b is the effective detection distance limit of the obstacle avoidance sensor.

[0186] The final algorithm generates the recommended optimal picking target point of the picking plane :

[0187] At this point, the unmanned forklift AGV pallet picking method based on the center of gravity corrected posture completes the recommendation and generation process of the optimal target picking point posture. The path planner captures the final formally picked target point and generates a planned path. The motion controller and attachment actuator execute related motion control and fork arm action processes.

[0188] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An unmanned forklift pallet picking method based on center of gravity correction posture, characterized in that: The steps include the following steps: S1: The unmanned forklift obtains the RGB feature image of the target pallet cargo, establishes a pallet pose estimation model, and defines a pallet detection category template. The pallet pose estimation model estimates the pallet pose based on the RGB feature image and the pallet detection category template, and outputs an estimated pallet pose. The unmanned forklift obtains the estimated pallet pose and initializes the pre-pickup motion planning. The unmanned forklift is controlled to pre-pick up the pallet cargo at the initial point according to the pre-pickup motion planning. S2: The unmanned forklift is equipped with 4 weighing sensors, namely C1, C2, C3 and C4. C1 and C3 are installed one by one on the two fork arms of the unmanned forklift, respectively, near the front end. C2 and C4 are installed one by one on the two fork arms of the unmanned forklift, respectively, near the rear end. In the reference coordinate system of the picking plane, the coordinates of each weighing sensor are (x C1 ,y C1 ,0),(x C1 ,y C2 ,0),(x C3 ,y C1 ,0),(x C3 ,y C2 , 0), When the unmanned forklift pre-picks up the pallet cargo at the initial point, the real-time collection values ​​of the four weighing sensors are: M1, M2, M3 and M4 ; S3: Perform final value consistency analysis: Determine whether the cargo pallet pre-picked at the initial point meets the safety operation requirements. The specific judgment steps of the final value consistency analysis are as follows: S3-1: Use the following formula to calculate the allowable deviation error ratio of the current pre-picked cargo pallet : ； in, M Indicates the mass value of the pallet cargo. M=M1+M2+M3+M4, M1 and M3 Collect values ​​for the weighing sensor at the front end of the fork arm of the unmanned forklift ，M2 and M4 Collect values ​​for the weighing sensor at the rear end of the forklift’s fork arm, and M1 and M2 Collect values ​​for the load cells on the same fork arm, M3 and M4 Collect values ​​for the load cells on the same fork arm; The constraints for picking the safety factor A are as follows: ； in, Indicates the value collected by the weighing sensor and , Indicates the value collected by the weighing sensor and ; Define the pick-up safety factor A=0, which means that the collected value of the weighing sensor meets the final value consistency; The picking safety factor A=1 is defined, indicating that the pallet cargo has a longitudinal center of gravity offset; The picking safety factor A=2 is defined, indicating that the pallet cargo has a lateral center of gravity offset; S4: Perform dynamic fluctuation stability analysis: obtain n groups of collected data of the 0-point calibration state, dynamic fluctuation state, final value holding state and pallet release stage of the collected values ​​of the four weighing sensors respectively, perform consistency test of the ICC intra-group correlation coefficient based on the obtained n groups of collected data, and judge the dynamic fluctuation stability of the collected values ​​of the weighing sensors within the group based on the data performance of the ICC; The specific steps of dynamic fluctuation stability analysis are as follows: S4-1: In a pre-pickup process, n groups of collected data are divided into a 0-point calibration phase, a dynamic fluctuation phase, and a final value data phase; S4-2: The sampling data of the 0-point calibration stage, dynamic fluctuation stage and final value data stage were analyzed by ICC intraclass correlation coefficient method; S4-3: Select the two-way mixed effects model and use the absolute consistency metric to calculate the ICC intraclass correlation coefficient. The calculation formula is as follows: ICC= ; Among them, MSr is the mean square of row variables, that is, the mean square of sensor variables, MSe is the mean square of error, MSc is the mean square of column variables, that is, the mean square of sampled weighing data variables, and n is the number of collected data groups; Define the pallet picking scenario , indicating that the data consistency is excellent; Define the pallet picking scenario , indicating that the data consistency is good; Define the pallet picking scenario , indicating that the data consistency is moderate or poor; S5: When the pick-up safety factor A=0 and In the case of a pre-pickup motion plan, the pre-pickup motion plan is converted into a formal pickup motion plan, and the unmanned forklift picks up the pallet cargo according to the formal pickup motion plan; When the pick-up safety factor A=1 or A=2, and When the unmanned forklift control system obtains the warehouse map, and determines whether the flatness of the operating environment meets the international definition of the flatness of the operating environment of the automated guided vehicle according to the warehouse map. If the definition of the environmental flatness is met, the pre-pickup motion plan is converted into a formal pickup motion plan, and the unmanned forklift picks up the pallet goods according to the formal pickup motion plan. Otherwise, step S6 is executed; When the pick-up safety factor A=1 or A=2, and When , execute step S6; S6: Calculate the coordinates of the center of gravity of the pallet cargo in the picking plane coordinate system based on the moment balance principle , by coordinates Generate the optimal target picking posture for the current pallet scene ; Generate optimal target picking pose The specific steps are as follows: S6-1: Based on the values ​​collected by the four weighing sensors, due to the moment balance, the pallet obtains the coordinates of the center of gravity in the picking plane coordinate system As shown in the following formula: ； ； S6-2: Generate the optimal target picking posture based on the projection coordinate value of the pallet cargo center of gravity on the picking plane as the reference point for the unmanned forklift to pick up the target pallet : ； Among them, y2 is the ordinate value of the center of the bracket in the middle of the pallet detection section in the picking plane coordinate system, L S is the fork arm length of the unmanned forklift, d is the fork entry depth, d=L P -b, where L p is the longitudinal length of the pallet, and b is the effective detection distance limit of the obstacle avoidance sensor; S7: The unmanned forklift system calculates the optimal target picking posture according to step S6 Generate the planned path for the final picking process and complete the picking.

2. The unmanned forklift pallet picking method based on center of gravity correction posture according to claim 1 is characterized in that: In step S1, the specific steps of the pallet posture estimation are as follows: S1-1: The pallet pose estimation model defines a pallet detection category template through a top-down spatial distribution relationship. The pallet detection category template includes four types of targets: cargo, pallet body, pallet hole position and support plane. The pallet detection category template characterizes the top-down distribution structure and size relationship of the four types of targets, and uses RGB color configuration to define each category of the pallet detection category template to obtain an RGB feature image of the cargo pallet; S1-2: Using the RGB feature image as an input of an image classifier, converting the RGB feature image into an HSI color gamut; S1-3: The feature spaces of cargo, pallet body, pallet holes and support plane are represented as A, B, C, D respectively, and the sample feature vector [x ij, c ij ] T It is expressed as follows: [x ij, c ij ] T =[R,G,B,H,S,c ij ] T ; c ij = ； H= ； S= ； Among them, i is the pixel row, j is the pixel column, and c ij is the category matrix element, R is the red grayscale value in the RGB three-color channel, G is the green grayscale value in the RGB three-color channel, B is the blue grayscale value in the RGB three-color channel, H is the hue in the HSI color gamut model, S is the saturation in the HSI color gamut model, x ij is the RGB channel feature information, T is the transposed representation of the matrix; Use the SVM model to process and classify the four types of targets in the RGB feature image, and match the classification output with the defined pallet detection category template; S1-4: Using a grid compression matching processing algorithm, a grid compression unit is established according to the pallet detection category template, so that the compression unit is a single category, the grid size is w, and the compressed pallet detection category template is expressed as follows: ； ； ； in, represents the element in the i-th row and j-th column of the template matrix, represents the element in the i-th row and j-th column of the category matrix, For the nth category in Quantity within the unit; S1-5: Locate the position of the pallet by using the window sliding method, and calculate the similarity between the pallet detection category template and the window: ； ； ； in, is the matching degree of the i-th and j-th units, is the threshold ratio of matching degree; Using a correlation matching algorithm to perform similarity matching on a result of subtracting the mean from the pallet detection category template and a result of subtracting the mean from the original image; S1-6: When the similarity is greater than the preset similarity threshold, the pallet posture corresponding to the window is captured, and the NMS non-maximum suppression method is used to process the output problem of multiple pallet postures when outputting the captured pallet postures; S1-7: Output the center point coordinates p1 (x1, y1, z1), p2 (x2, y2, z2), and p3 (x3, y3, z3) of the left, middle, and right brackets of the captured pallet posture according to the similarity measurement, and calculate the estimated value of the angle offset at time t To determine the pallet position: Among them, n represents the number of points, here are three points p1, p2, and p3. and Indicates the corresponding bracket center point coordinate value; Processing is done by filtering the data using the average of multiple frames: According to the estimated value of the angle offset at time t Calculate the plane coordinates of the pallet posture center at time t ； 。 3. The unmanned forklift pallet picking method based on center of gravity correction posture according to claim 1 is characterized in that: In step S2, the pre-pickup steps at the initial point are as follows: S2-1: According to the pallet position output in step S1, the unmanned forklift receives the route to be traveled generated in the RCS dispatching system; S2-2: The unmanned forklift drives to the initial point according to the route to be traveled; S2-3: After each weighing sensor is calibrated and zeroed, the unmanned forklift controls the fork arm to lift for pre-pickup and lifts the pallet to a preset height.

4. An unmanned forklift pallet picking system based on center of gravity correction posture, comprising a memory and a processor, wherein a computer program is stored in the memory, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

Images