Machine learning-based method and device for automatic stacking of pallets and mobile robot

Abstract

The application discloses a kind of based on machine learning's material frame automatic stacking method, device and mobile robot, it is applied to the fork type mobile robot being provided with camera, the method includes: by establishing key point detection model, the Z-axis coordinate position of lower layer material frame stand in the image data photographed by camera is identified, and whether the Z-axis coordinate position of lower layer material frame stand is out of limit is judged, if not out of limit, then upper layer material frame is moved to its bowl groove and the corresponding stand of lower layer material frame close overlap, and the X-axis and Z-axis coordinate position of bowl groove of upper layer material frame far end at this time and the X-axis difference and Z-axis difference between the stand of lower layer material frame far end and the bowl groove of upper layer material frame far end are detected whether out of limit, if out of limit, then the position of fork type mobile robot fork is adjusted, if not out of limit, then control fork and stack upper layer material frame on lower layer material frame.The application realizes the automatic stacking of material frame by fork type mobile robot, and improves stacking efficiency.

Description

Technical Field

[0001] This invention relates to the field of warehouse management technology, and more specifically, to a machine learning-based method, apparatus, and mobile robot for automatic stacking of material boxes. Background Technology

[0002] In existing technologies, multi-layered crates are typically stacked manually using forklifts to improve warehouse utilization. However, manual stacking of crates is inefficient and labor-intensive, making it difficult to meet the needs of automation. Furthermore, stacking high-layer crates requires workers to be proficient in forklift operation and operate the forklifts precisely to ensure that the crates are not squeezed or collided during the stacking process. Summary of the Invention

[0003] The purpose of this invention is to provide a machine learning-based automatic stacking method, apparatus and related components for material frames, which aims to solve the problems of low stacking efficiency and difficulty in meeting automation requirements in the prior art.

[0004] In a first aspect, embodiments of the present invention provide a machine learning-based automatic stacking method for material frames, applied to a forklift mobile robot, wherein the forklift mobile robot is equipped with a camera, and the method includes:

[0005] The system acquires sample image data captured by the camera, performs data augmentation, data preprocessing, and data annotation on the sample image data to obtain a dataset, and divides the dataset into training set data and test set data. A deep neural network algorithm model is trained using the training set data to obtain a key point detection model, and the key point detection model is deployed to the forklift mobile robot. The data annotation includes labeling the X-axis and Z-axis coordinate positions of multiple bowls in the upper material frame and the X-axis and Z-axis coordinate positions of multiple columns in the lower material frame.

[0006] When performing stacking operations, the fork-type mobile robot controls the forks to move the upper material frame above the lower material frame. By inputting the real-time image data captured by the camera into the key point detection model, the X-axis and Z-axis coordinate positions of multiple bowls in the upper material frame and the X-axis and Z-axis coordinate positions of multiple columns in the lower material frame are predicted.

[0007] Based on the Z-axis coordinate position of at least one column of the lower material frame, determine whether the Z-axis difference between the column at the far end of at least one side of the lower material frame and the column at the near end of the same side exceeds the limit, and obtain a first judgment result; if the first judgment result is that it exceeds the limit, an abnormal alarm is triggered.

[0008] If the first judgment result is within limits, the forklift mobile robot moves the upper material frame until its bowl is nearly overlapping with the corresponding column of the lower material frame. Based on the X-axis and Z-axis coordinates of the column at the far end of the lower material frame and the bowl at the far end of the upper material frame, it determines whether the X-axis difference and Z-axis difference between the column at the far end of the lower material frame and the bowl at the far end of the upper material frame exceed limits, thus obtaining a second judgment result. If the second judgment result exceeds limits, the position of the forks is adjusted.

[0009] If the second judgment result is within the limit, then control the forks to stack the upper material frame onto the lower material frame.

[0010] Secondly, embodiments of the present invention provide an automatic stacking device for material frames based on machine learning, comprising:

[0011] An acquisition unit is used to acquire sample image data captured by the camera, perform data augmentation, data preprocessing, and data annotation on the sample image data to obtain a dataset, divide the dataset into training set data and test set data, train a deep neural network algorithm model using the training set data to obtain a key point detection model, and deploy the key point detection model to the forklift mobile robot; wherein, the data annotation includes annotating the X-axis and Z-axis coordinate positions of multiple bowls in the upper material frame and the X-axis and Z-axis coordinate positions of multiple columns in the lower material frame;

[0012] The prediction unit is used to predict the X-axis and Z-axis coordinates of multiple bowls in the upper layer and the X-axis and Z-axis coordinates of multiple columns in the lower layer when the forklift mobile robot controls the forks to move the upper layer material frame above the lower layer material frame during stacking operations.

[0013] An anomaly detection unit is used to determine whether the Z-axis difference between the far end of at least one side of the lower material frame and the near end of the same side exceeds the limit based on the Z-axis coordinate position of the column on at least one side of the lower material frame, and obtain a first detection result; if the first detection result is that it exceeds the limit, an anomaly alarm is triggered.

[0014] An adjustment unit is configured to, if the first judgment result is within limits, move the forklift mobile robot until its bowl is nearly overlapping with the corresponding column of the lower material frame, and determine whether the X-axis difference and Z-axis difference between the column at the far end of the lower material frame and the bowl at the far end of the upper material frame exceed limits based on the X-axis and Z-axis coordinates of the column at the far end of the lower material frame and the bowl at the far end of the upper material frame, thereby obtaining a second judgment result; if the second judgment result is exceeding limits, adjust the position of the forks.

[0015] The stacking unit is used to control the forks to stack the upper material frame onto the lower material frame if the second judgment result is within the limit.

[0016] Thirdly, embodiments of the present invention provide a forklift mobile robot, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the machine learning-based automatic stacking method for material frames as described above.

[0017] The advantages of this invention compared to the prior art are: Based on machine learning technology, this invention uses a forklift mobile robot to control the stacking of several material frames, thereby achieving automated stacking, reducing manual intervention, and further improving the stacking efficiency of the material frames. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating an automatic stacking method for material frames based on machine learning, provided in an embodiment of the present invention;

[0020] Figure 2 A flowchart illustrating a sub-unit of a machine learning-based automatic stacking method for material frames provided in an embodiment of the present invention;

[0021] Figure 3 This is a flowchart illustrating another sub-unit of a machine learning-based automatic stacking method for material frames provided in an embodiment of the present invention.

[0022] Figure 4 This is a flowchart illustrating another sub-unit of a machine learning-based automatic stacking method for material frames provided in an embodiment of the present invention.

[0023] Figure 5 This is a flowchart illustrating another sub-unit of a machine learning-based automatic stacking method for material frames provided in an embodiment of the present invention.

[0024] Figure 6 This is a flowchart illustrating another sub-unit of a machine learning-based automatic stacking method for material frames provided in an embodiment of the present invention.

[0025] Figure 7A schematic diagram of a machine learning-based automatic stacking device for material frames provided in an embodiment of the present invention;

[0026] Figure 8 A schematic diagram of the structure of a sub-unit of an automatic stacking device for material frames based on machine learning, provided in an embodiment of the present invention;

[0027] Figure 9 A schematic diagram of the structure of another sub-unit of an automatic stacking device for material frames based on machine learning, provided in an embodiment of the present invention;

[0028] Figure 10 A schematic diagram of the structure of another sub-unit of an automatic stacking device for material frames based on machine learning, provided in an embodiment of the present invention;

[0029] Figure 11 A schematic diagram of the structure of another sub-unit of an automatic stacking device for material frames based on machine learning, provided in an embodiment of the present invention;

[0030] Figure 12 A schematic diagram of the structure of another sub-unit of an automatic stacking device for material frames based on machine learning, provided in an embodiment of the present invention;

[0031] Figure 13 A schematic diagram of the structure of a material frame in an automatic material frame stacking device based on machine learning, provided for an embodiment of the present invention;

[0032] Figure 14 This is a schematic diagram of the structure of an automatic stacking device for material frames based on machine learning, provided in an embodiment of the present invention, for performing stacking operations.

[0033] Markings in the image:

[0034] 1. Columns; 2. Bowl trough; 3. Steel frame; 4. Forklift mobile robot; 5. Upper material frame; 6. Lower material frame; A, B, C, and D are the four columns of the material frame.

[0035] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0036] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0037] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "circumferential," and "radial," etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings and are only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0038] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0039] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0040] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0041] Combination Figure 1 , Figure 1 This is a flowchart illustrating an automatic stacking method for material frames based on machine learning, provided in an embodiment of the present invention. The method is applied to a forklift mobile robot and includes the following steps S101-S108.

[0042] S101. Acquire sample image data captured by the camera, and perform data augmentation, data preprocessing, and data annotation on the sample image data to obtain a dataset. Divide the dataset into training set data and test set data. Train a deep neural network algorithm model using the training set data to obtain a key point detection model, and deploy the key point detection model to the forklift mobile robot. Among them, the data annotation includes annotating the X-axis and Z-axis coordinate positions of multiple bowls in the upper material frame and the X-axis and Z-axis coordinate positions of multiple columns in the lower material frame.

[0043] The automatic stacking method for material frames based on machine learning provided in this embodiment of the invention is applied to a forklift mobile robot. The forks of the forklift mobile robot can move forward and backward and laterally, and the camera can be set on the forks of the forklift mobile robot. The camera transmits the captured real-time image data to the forklift mobile robot or to the device that implements the method of this embodiment. The device can be deployed on the forklift mobile robot or on a background or cloud server. In this way, data processing and calculation can be performed on the background or cloud server, and the results can be returned to the forklift mobile robot, so that the forklift mobile robot can perform the final control and output. In this embodiment, multiple image data points captured by the camera are first acquired and used as sample image data. These sample image data are then processed, including data augmentation, data preprocessing, and data annotation. Data annotation includes labeling key points in the sample image data, specifically the X and Z coordinates of multiple bowls in the upper material frame and the X and Z coordinates of multiple columns in the lower material frame, to obtain a dataset. Data augmentation includes applying translation, vertical shifting, flipping, mean filtering, Gaussian filtering, and random occlusion to the sample image data to increase the number of sample image data points. Data preprocessing includes normalizing the data in the dataset. Next, the dataset is divided into training and testing sets. The training set data is input into a deep neural network algorithm model for extensive training to obtain a keypoint detection model. This model is then tested using the testing set data. Once the test is passed, the keypoint detection model can be deployed on a forklift robot, enabling the forklift robot to perform keypoint detection on the image data captured by the camera.

[0044] In this embodiment of the invention, it further includes:

[0045] The performance of the key point detection model is evaluated.

[0046] Specifically, after a large amount of training is performed on the deep neural network algorithm model to obtain the key point detection model, it is still necessary to evaluate the performance of the key point detection model.

[0047] In embodiments of the present invention, such as Figure 2As shown, the performance evaluation of the keypoint detection model includes steps S201-S204:

[0048] S201. Calculate the mean square error in the training of the deep neural network algorithm model;

[0049] S202 and determine whether the mean square error result is similar to the prediction result of the test set data. If they are similar, proceed to step S203; otherwise, proceed to step S204.

[0050] S203. The performance of the key point detection model is determined to meet the application standards.

[0051] S204. The performance of the key point detection model was determined to be below the application standard.

[0052] In this embodiment of the invention, the keypoint detection model is evaluated by comparing the mean squared error (MSE) result during the training of the deep neural network algorithm model with the prediction results in the test set data. This evaluation determines whether the keypoint detection model meets the application standard performance. If the MSE result is similar to the prediction results in the test set data, the keypoint detection model meets the application standard. If the MSE result differs significantly from the prediction results in the test set data, the keypoint detection model does not meet the application standard. When the keypoint detection model meets the application standard, it is deployed on the forklift robot. When the keypoint detection model does not meet the application standard, more image data needs to be collected, the structure and parameters of the deep neural network algorithm model need to be adjusted, and it needs to be retrained until the keypoint detection model meets the application standard.

[0053] S102. When performing stacking operations, the forklift mobile robot controls the forks to move the upper material frame above the lower material frame. By inputting the real-time image data captured by the camera into the key point detection model, the X-axis and Z-axis coordinate positions of multiple bowls in the upper material frame and the X-axis and Z-axis coordinate positions of multiple columns in the lower material frame are predicted.

[0054] In this embodiment of the invention, the forklift mobile robot controls the forks to lift the upper layer of the stacked material frames and move them to the corresponding lower layer of the material frames in preparation for unloading and stacking. At this time, the position of the forks of the forklift mobile robot is higher than the height of the lower layer of the material frames, and the upper layer of the material frames is located above the lower layer of the material frames. The camera located on the forks can capture multiple bowls of the upper layer of the material frames and multiple columns of the lower layer of the material frames. The captured real-time image data is input into the key point detection model to obtain the X-axis and Z-axis coordinate positions of the multiple bowls of the upper layer of the material frames and the X-axis and Z-axis coordinate positions of the multiple columns of the lower layer of the material frames in the real-time image data.

[0055] S103. Determine whether the Z-axis difference between the column at the far end of at least one side of the lower material frame and the column at the near end of the same side exceeds the limit based on the Z-axis coordinate position of the column on at least one side of the lower material frame, and obtain the first judgment result; if the first judgment result is that it exceeds the limit, proceed to step S104; if the first judgment result is that it does not exceed the limit, proceed to step S105.

[0056] S104, Abnormal alarm.

[0057] In embodiments of the present invention, such as Figure 13 As shown, Figure 13 This is a schematic diagram of a standard material frame structure. The four uprights of the lower material frame are labeled as upright A, upright B, upright C, and upright D. The structure of the material frame is such that the uprights on the same side are connected by a steel frame (e.g., upright A and upright B are connected by a steel frame). Based on the Z-axis coordinates of the uprights of the lower material frame obtained in step S102, the Z-axis difference between the uprights of the lower material frame is calculated (i.e., the Z-axis difference between upright A and upright D in the figure, and / or the Z-axis difference between upright B and upright C). If the distance exceeds the limit, it means that the distance between the two uprights is too large or too small, making it impossible to accurately stack a certain bowl of the upper material frame on the corresponding upright of the lower material frame. In other words, it is impossible to correctly and stably stack the upper material frame on the lower material frame. If stacking continues in this case, stacking offset will occur, which may easily cause an operational accident.

[0058] Specifically, based on the real-time image data captured by the camera, the Z-axis coordinates of the four columns of the lower material frame are identified and obtained. Based on the Z-axis coordinates of at least two columns on one side, the Z-axis difference between the far and near columns on at least one side of the lower material frame is calculated to determine whether the column on that side has exceeded the deformation limit, thus obtaining a first judgment result. Figure 13 For example, calculate the Z-axis difference Z between columns A and D located on the same side. AD That is, the distance between column A and column D along the Z-axis, and / or the calculated Z-axis difference Z between column B and column C. BC That is, the distance between column A and column D along the Z-axis, and then according to Z... AD and / or Z BC Determine whether the deformation exceeds the limit and obtain the first judgment result.

[0059] When the first judgment result is an out-of-limit, the stacking operation of the forklift robot is suspended and an abnormal alarm is triggered; the Z-axis difference (Z0) of the lower material frame column is adjusted manually. AD and / or Z BC ), until the lower frame no longer deforms, the Z-axis difference (Z AD and / or Z BC If the limit is not exceeded, the forklift mobile robot can continue the interrupted stacking operation.

[0060] In an embodiment of the present invention, as Figure 3 shown, step S103 includes steps S301 - S303:

[0061] S301. According to the Z - axis coordinate positions of the columns on at least one side of the lower material frame, calculate the Z - axis difference between the columns at the distal end and the proximal end on at least one side of the lower material frame, so as to obtain the Z - axis difference of the columns on at least one side;

[0062] S302. Compare the Z - axis difference of the columns on at least one side with the standard Z - axis difference of the corresponding columns of the standard lower material frame to obtain a comparison difference;

[0063] S303. If the comparison difference exceeds the first threshold, then take the over - limit as the first judgment result; if the comparison difference does not exceed the first threshold, then take the non - over - limit as the first judgment result.

[0064] In a specific application scenario, please continue to refer to Figure 13 and combine with Figure 14 shown, Figure 13 the material frame in Figure 14 will be used as the lower material frame. In

[0065] Figure 14 , the forklift mobile robot forks up a material frame from the AB surface. Among them, the A column and the B column are connected by a steel frame, and the C column and the D column are also connected by a steel frame. The forklift mobile robot forks up an upper - layer material frame from the AB surface of the material frame. The upper - layer material frame has the same specifications as the aforementioned lower material frame. That is to say, the A column and the D column are respectively the proximal column and the distal column on the same side of the lower material frame, and the B column and the C column are respectively the proximal column and the distal column on the other side of the lower material frame (when the forklift mobile robot forks up the lower material frame from the CD surface, the A column and the D column are respectively the distal column and the proximal column on the same side of the lower material frame, and the B column and the C column are respectively the distal column and the proximal column on the other side of the lower material frame). Similarly, the distal bowl groove and the proximal bowl groove of the upper - layer material frame are also the same.

[0065] In an embodiment of the present invention, calculate the Z - axis difference between the distal column and the proximal column on at least one side of the current lower material frame and the standard Z - axis difference between the distal column and the proximal column on at least one side of the standard. Compare the Z - axis difference with the standard Z - axis difference to obtain a comparison difference. In the embodiment of the present invention, the standard Z - axis difference can be set as Z0, and the first threshold is set as Z1 = 3 cm. Specifically, calculate the Z - axis difference Z AD of the AD side of the lower material frame, compare Z AD with Z0 to obtain a comparison difference Z (Z AD - Z0 = Z), compare the comparison difference with the first threshold. If |Z|>Z1, the first judgment result is that Z AD is over - limit; if |Z|<Z1, the first judgment result is that Z AD is not over - limit.

[0066] S105, the forklift mobile robot moves the upper material frame until its bowl and the corresponding column of the lower material frame are nearly overlapping;

[0067] S106. Based on the X-axis and Z-axis coordinates of the column at the far end of the lower material frame and the bowl at the far end of the upper material frame, determine whether the X-axis difference and Z-axis difference between the column at the far end of the lower material frame and the bowl at the far end of the upper material frame exceed the limit, and obtain the second judgment result; if the second judgment result is that it exceeds the limit, proceed to step S107; if the second judgment result is that it does not exceed the limit, proceed to step S108.

[0068] In this embodiment of the invention, if the Z-axis difference between the columns of the lower material frame does not exceed the limit, i.e., the lower material frame is not deformed, the forklift robot continues the stacking operation, moving the upper material frame directly above the lower material frame, so that the position of the bowl groove of the upper material frame is nearly overlapping with the position of the column of the lower material frame. In other words, the sides of the upper and lower material frames are currently nearly on the same plane. Figure 14 As shown. At this time, the camera's shooting is limited, only able to capture the column at the far end of the lower material frame and the trough at the far end of the upper material frame. That is, it can only identify the X-axis and Z-axis coordinates of the trough at the far end of the upper material frame and the X-axis and Z-axis coordinates of the column at the far end of the lower material frame in the real-time image data. By judging whether the X-axis and Z-axis coordinates of the trough at the far end of the upper material frame and the X-axis and Z-axis differences between them and the column at the far end of the lower material frame exceed the limits, a second judgment result is obtained.

[0069] In a specific application scenario, the upper material frame is moved directly above the lower material frame, making them nearly overlap. At this point, the X-axis and Z-axis coordinates of the lower material frame's columns A, B, C, and D nearly overlap with the corresponding slots (slots A, B, C, and D) of the upper material frame. The X-axis and Z-axis differences between the X-axis and Z-axis coordinates of any distal slot and its corresponding column are calculated, and it is determined whether these differences exceed limits, ultimately yielding a second judgment result.

[0070] In embodiments of the present invention, such as Figure 4 As shown, step S106 includes steps S401-S406:

[0071] S401. Based on the X-axis coordinates of the column at the far end of the lower material frame and the X-axis coordinates of the bowl at the far end of the upper material frame, calculate the X-axis difference between the column at the far end of at least one side of the lower material frame and the bowl at the far end of the upper material frame on the same side, and obtain the X-axis difference between at least one side.

[0072] S402. Based on the Z-axis coordinate position of the column at the far end of the lower material frame and the Z-axis coordinate position of the bowl groove at the far end of the upper material frame, calculate the Z-axis difference between the column at the far end of at least one side of the lower material frame and the bowl groove at the far end of the upper material frame on the same side, and obtain the Z-axis difference at the far end of at least one side.

[0073] S403. Determine whether the X-axis difference exceeds the second threshold; if the X-axis difference exceeds the second threshold, proceed to step S405; if the X-axis difference does not exceed the second threshold, proceed to step S406.

[0074] S404. Determine whether the Z-axis difference exceeds the third threshold; if the Z-axis difference exceeds the third threshold, proceed to step S405; if the X-axis difference does not exceed the second threshold and the Z-axis difference does not exceed the third threshold, proceed to step S406.

[0075] S405. Treat exceeding the limit as the second judgment result;

[0076] S406. Take not exceeding the limit as the second judgment result.

[0077] Specifically, the camera transmits the captured real-time image data to the forklift robot, which can identify the X-axis and Z-axis coordinates of the two bowls at the far end of the upper material frame and the two pillars at the far end of the lower material frame in the image data. It calculates the X-axis difference between one bowl in the upper material frame and its corresponding pillar in the lower material frame; then it calculates the Z-axis difference between the bowl in the upper material frame and its corresponding pillar in the lower material frame; and it determines whether the X-axis difference exceeds a second threshold and whether the Z-axis difference exceeds a third threshold. If at least one of the X-axis and Z-axis differences exceeds its corresponding threshold (the threshold for the X-axis difference is the second threshold, and the threshold for the Z-axis difference is the third threshold), the second judgment result is considered not to exceed the limit; if neither the X-axis nor the Z-axis difference exceeds its corresponding threshold, the second judgment result is considered not to exceed the limit. For example, the second threshold is set to X2, and the third threshold is set to Z3; the X-axis difference X between the bowl position D and the pillar position D is calculated. D and the Z-axis difference Z between the Z-axis coordinate positions D If X D More than X2, and / or Z D If it exceeds Z3, then exceeding the limit will be used as the second judgment result. If X D Not exceeding X2, and Z D If the value does not exceed Z3, then "not exceeding the limit" is taken as the second judgment result. When the second judgment result is "not exceeding the limit," the position of the forks needs to be adjusted for safer unloading. Of course, in this embodiment of the invention, the design is based on the radius of the material frame bowl, and the values ​​of the second threshold and the third threshold are equal.

[0078] S107. Adjust the position of the forks.

[0079] S108. Control the forks to stack the upper material frame onto the lower material frame.

[0080] In this embodiment of the invention, when the second judgment result is within the limit, the forks can be directly controlled to stack the current upper material frame on the lower material frame.

[0081] In embodiments of the present invention, such as Figure 5 As shown, step S105 further includes steps S501-S503:

[0082] S501. If the second judgment result is out of limit and the X-axis difference exceeds the second threshold, then control the fork to move sideways.

[0083] S502. If the second judgment result is out of limit and the Z-axis difference exceeds the third threshold, then control the fork to move forward and backward.

[0084] S503. If the second judgment result is out of limit, and the difference on the X-axis exceeds the second threshold and the difference on the Z-axis exceeds the third threshold, then control the side movement and forward and backward movement of the forks.

[0085] Specifically, when the second judgment result is an out-of-limit, the position of the upper material frame can be adjusted by controlling the lateral or forward / backward movement of the forks. When the X-axis difference X... D If the value exceeds the second threshold X2, the forks will be controlled to move to the left or right; when the Z-axis difference Z... D If the value exceeds the third threshold Z3, the forks will be controlled to move forward or backward; when the Z-axis difference Z... D The difference between the X-axis and the X-axis. D When both exceed their corresponding thresholds, the forks need to move forward and backward after lateral movement. That is, to make the X-axis difference X... D Not exceeding the second threshold X2, Z-axis difference X D It does not exceed the third threshold Z3.

[0086] The machine learning-based automatic stacking method for material frames in this embodiment of the invention further includes:

[0087] The system detects the stacking status of the upper and lower material frames. If the stacking is not completed correctly, an alarm is triggered. If the stacking is completed correctly, the system controls the forklift robot to move backward and exit the stacking operation.

[0088] Specifically, after the forks are controlled to stack the upper material frame onto the lower material frame, the forklift robot moves backward. It determines whether the stacking operation is complete by detecting the current stacking status of the upper and lower material frames. If it is complete, the next stacking operation is performed. If it is not complete, an alarm is triggered and manual intervention is required.

[0089] In embodiments of the present invention, such as Figure 6As shown, the system detects the stacking status of the upper and lower material frames. If stacking fails to be completed correctly, an alarm is triggered. If stacking is completed correctly, the forklift robot is controlled to move backward and exit the stacking operation. This includes:

[0090] S601. Detect the plane angle between the upper and lower material frames;

[0091] S602. Determine whether all sides of the upper and lower material frames are on the same plane. If the sides of the upper and lower material frames are not on the same plane, proceed to step S603. If the sides of the upper and lower material frames are on the same plane, proceed to step S604.

[0092] S603, The stacking process failed to complete correctly;

[0093] S604, Determine that the stacking is completed correctly.

[0094] Specifically, the planar angle between the upper and lower material frames is detected. If either side of the two frames is not on the same plane, the stacking offset between the upper and lower material frames is determined, the stacking operation is not completed correctly, and the forklift robot will issue an abnormal alarm. In other words, if either side of the two frames is on the same plane, the current stacking operation is determined to be completed correctly, and the forklift robot will proceed to the next stacking operation.

[0095] The automatic stacking method for material frames based on machine learning disclosed in this invention uses machine learning technology to control the forks of a forklift mobile robot to stack several material frames, thereby achieving automated stacking, reducing manual intervention, and further improving the stacking efficiency of material frames.

[0096] like Figure 7 As shown, corresponding to the above-mentioned machine learning-based automatic stacking method for material frames, this embodiment of the invention also provides a machine learning-based automatic stacking device for material frames, the device comprising:

[0097] The acquisition unit 701 is used to acquire sample image data captured by the camera, and to perform data augmentation, data preprocessing and data annotation on the sample image data to obtain a dataset. The dataset is divided into training set data and test set data. A deep neural network algorithm model is trained using the training set data to obtain a key point detection model, and the key point detection model is deployed to the forklift mobile robot. The data annotation includes annotating the X-axis and Z-axis coordinate positions of multiple bowls in the upper material frame and the X-axis and Z-axis coordinate positions of multiple columns in the lower material frame.

[0098] The prediction unit 702 is used to predict the X-axis and Z-axis coordinates of multiple bowls in the upper layer and the X-axis and Z-axis coordinates of multiple columns in the lower layer when the forklift mobile robot controls the forks to move the upper layer frame to the top of the lower layer frame during stacking operations. By inputting real-time image data captured by the camera into the key point detection model, the prediction unit 702 predicts the X-axis and Z-axis coordinates of multiple bowls in the upper layer frame and the X-axis and Z-axis coordinates of multiple columns in the lower layer frame.

[0099] Anomaly judgment unit 703 is used to determine whether the Z-axis difference between the column at the far end of at least one side of the lower material frame and the column at the near end of the same side exceeds the limit based on the Z-axis coordinate position of the column on at least one side of the lower material frame, and obtain a first judgment result.

[0100] An abnormal alarm unit 704 is used to trigger an abnormal alarm if the first judgment result is that the limit is exceeded.

[0101] The overlapping unit 705 is used to move the upper material frame to a position where its bowl and the corresponding column of the lower material frame are nearly overlapped if the first judgment result is not exceeded.

[0102] The over-limit judgment unit 706 is used to determine whether the difference between the X-axis and the Z-axis between the column at the far end of the lower material frame and the bowl at the far end of the upper material frame exceeds the limit based on the X-axis and Z-axis coordinate positions of the column at the far end of the lower material frame and the bowl at the far end of the upper material frame, and to obtain a second judgment result.

[0103] Adjustment unit 707: If the second judgment result is an over-limit, adjust the position of the forks;

[0104] The stacking unit 708 is used to control the forks to stack the upper material frame onto the lower material frame if the second judgment result is within the limit.

[0105] The present invention discloses an automatic stacking device for material frames based on machine learning. Based on machine learning technology, it uses a forklift mobile robot to control the forks to stack several material frames, thereby achieving automated stacking, reducing manual intervention, and further improving the stacking efficiency of material frames.

[0106] In embodiments of the present invention, such as Figure 8 As shown, the anomaly detection unit 703 includes:

[0107] The first calculation unit 801 is used to calculate the Z-axis difference between the column at the far end of at least one side of the lower material frame and the column at the near end of the same side based on the Z-axis coordinate position of the column on at least one side of the lower material frame, so as to obtain the Z-axis difference of the column on at least one side.

[0108] The first comparison unit 802 is used to compare the Z-axis difference of at least one side column with the Z-axis difference of the corresponding side column of the standard lower layer material frame to obtain a comparison difference.

[0109] The first judgment unit 803 is used to determine whether the difference exceeds a first threshold as the first judgment result, or whether the difference does not exceed the first threshold as the first judgment result.

[0110] In embodiments of the present invention, such as Figure 9 As shown, the over-limit judgment unit 706 includes:

[0111] The second calculation unit 901 is used to calculate the X-axis difference between the column at the far end of the lower material frame and the bowl at the far end of the upper material frame based on the X-axis coordinate position of the column at the far end of the lower material frame and the X-axis coordinate position of the bowl at the far end of the upper material frame, and to obtain the X-axis difference between at least one far end.

[0112] The second comparison unit 902 is used to calculate the Z-axis difference between the column at the far end of the lower material frame and the bowl at the far end of the upper material frame based on the Z-axis coordinate position of the column at the far end of the lower material frame and the Z-axis coordinate position of the bowl at the far end of the upper material frame, and to obtain the Z-axis difference between at least one far end.

[0113] The third judgment unit 903 is used to determine whether the X-axis difference exceeds the second threshold.

[0114] The fourth judgment unit 904 is used to determine whether the Z-axis difference exceeds the third threshold.

[0115] The first result unit 905 takes the exceeding of the limit as the second judgment result;

[0116] The second result unit 906 takes "not exceeding the limit" as the second judgment result.

[0117] In embodiments of the present invention, such as Figure 10 As shown, the over-limit judgment unit 706 further includes:

[0118] The lateral shift unit 1001 is used to control the lateral shift of the forks if the second judgment result is out of limit and the X-axis difference exceeds the second threshold.

[0119] The forward and backward movement unit 1002 is used to control the fork to move forward and backward if the second judgment result is out of limit and the Z-axis difference exceeds the third threshold.

[0120] The hybrid movement unit 1003 is used to control the lateral and forward / backward movement of the forks if the second judgment result is out of limit, and the difference in the X-axis exceeds the second threshold and the difference in the Z-axis exceeds the third threshold.

[0121] In this embodiment of the invention, the machine learning-based automatic stacking device for material frames further includes:

[0122] The detection unit is used to detect the stacking status of the upper and lower material frames. If the stacking is not completed correctly, an alarm will be triggered; if the stacking is completed correctly, the forklift robot will be controlled to move backward and exit the stacking operation.

[0123] In embodiments of the present invention, such as Figure 11 As shown, the detection unit includes:

[0124] The planar detection unit 1101 is used to detect the planar angle between the upper and lower material frames;

[0125] The plane judgment unit 1102 is used to determine whether all sides of the upper material frame and the lower material frame are on the same plane;

[0126] The stacking identification unit 1103 is used to determine that the stacking has not been completed correctly if the sides of the upper and lower material frames are not on the same plane.

[0127] The stacking completion unit 1104 is used to determine that the stacking is correctly completed if the sides of the upper and lower material frames are both on the same plane.

[0128] In this embodiment of the invention, the machine learning-based automatic stacking device for material frames further includes:

[0129] The performance evaluation unit is used to evaluate the performance of the key point detection model.

[0130] In embodiments of the present invention, such as Figure 12 As shown, the performance evaluation unit includes:

[0131] The third calculation unit 1201 is used to calculate the mean squared error result in the training of the deep neural network algorithm model.

[0132] Identify similar unit 1202 and determine whether the mean square error result is similar to the prediction result of the test set data;

[0133] The fifth judgment unit 1203 is used to determine that the performance of the key point detection model meets the application standard if the mean square error result is similar to the prediction result of the test set.

[0134] The sixth judgment unit 1204 is used to determine that the performance of the key point detection model has not met the application standard if the mean square error result is not similar to the prediction result of the test set data.

[0135] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0136] This invention also provides a forklift mobile robot, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the machine learning-based automatic stacking method for material frames as described above. Of course, the computer device may also include various network interfaces, power supplies, and other components. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this invention without departing from the principle of the invention, and these improvements and modifications also fall within the protection scope of the claims of this invention.

[0137] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A machine learning-based automatic stacking method for material frames, applied to a forklift robot, characterized in that, The forklift mobile robot is equipped with a camera, and the method includes: The system acquires sample image data captured by the camera, performs data augmentation, data preprocessing, and data annotation on the sample image data to obtain a dataset, and divides the dataset into training set data and test set data. A deep neural network algorithm model is trained using the training set data to obtain a key point detection model, and the key point detection model is deployed to the forklift mobile robot. The data annotation includes labeling the X-axis and Z-axis coordinate positions of multiple bowls in the upper material frame and the X-axis and Z-axis coordinate positions of multiple columns in the lower material frame. When performing stacking operations, the fork-type mobile robot controls the forks to move the upper material frame above the lower material frame. By inputting the real-time image data captured by the camera into the key point detection model, the X-axis and Z-axis coordinate positions of multiple bowls in the upper material frame and the X-axis and Z-axis coordinate positions of multiple columns in the lower material frame are predicted. Based on the Z-axis coordinate position of at least one column of the lower material frame, determine whether the Z-axis difference between the column at the far end of at least one side of the lower material frame and the column at the near end of the same side exceeds the limit, and obtain a first judgment result; if the first judgment result is that it exceeds the limit, an abnormal alarm is triggered. If the first judgment result is within limits, the forklift mobile robot moves the upper material frame until its bowl is nearly overlapping with the corresponding column of the lower material frame. Based on the X-axis and Z-axis coordinates of the column at the far end of the lower material frame and the bowl at the far end of the upper material frame, it determines whether the X-axis difference and Z-axis difference between the column at the far end of the lower material frame and the bowl at the far end of the upper material frame exceed limits, thus obtaining a second judgment result. If the second judgment result exceeds limits, the position of the forks is adjusted. If the second judgment result is within the limit, then control the forks to stack the upper material frame onto the lower material frame.

2. The automatic stacking method for material frames based on machine learning according to claim 1, characterized in that, The step of determining whether the Z-axis difference between the far end of at least one side of the lower material frame and the near end of the same side exceeds the limit based on the Z-axis coordinate position of at least one side of the column of the lower material frame, and obtaining a first determination result, includes: Based on the Z-axis coordinate position of at least one column of the lower material frame, calculate the Z-axis difference between the column at the far end of at least one side of the lower material frame and the column at the near end of the same side, and obtain the Z-axis difference of at least one column. The Z-axis difference of at least one side column is compared with the standard Z-axis difference of the corresponding side column of the standard lower layer material frame to obtain the comparison difference; If the comparison difference exceeds the first threshold, the exceeding the threshold is taken as the first judgment result; if the comparison difference does not exceed the first threshold, the not exceeding the threshold is taken as the first judgment result.

3. The automatic stacking method for material frames based on machine learning according to claim 1, characterized in that, The step of determining whether the X-axis and Z-axis differences between the far-end column of the lower material frame and the far-end trough of the upper material frame exceed the limits based on the X-axis and Z-axis coordinate positions of the column at the far end of the lower material frame and the trough at the far end of the upper material frame, and obtaining a second determination result, includes: Based on the X-axis coordinates of the column at the far end of the lower material frame and the X-axis coordinates of the trough at the far end of the upper material frame, the X-axis difference between the column at the far end of at least one side of the lower material frame and the trough at the far end of the upper material frame on the same side is calculated, thus obtaining the X-axis difference between at least one side. Based on the Z-axis coordinates of the column at the far end of the lower material frame and the Z-axis coordinates of the trough at the far end of the upper material frame, the Z-axis difference between the column at the far end of at least one side of the lower material frame and the trough at the far end of the upper material frame on the same side is calculated, thus obtaining the Z-axis difference between at least one side. Determine whether the X-axis difference exceeds a second threshold and whether the Z-axis difference exceeds a third threshold; If the X-axis difference exceeds the second threshold and / or the Z-axis difference exceeds the third threshold, then exceeding the limit is taken as the second judgment result; if the X-axis difference does not exceed the second threshold and the Z-axis difference does not exceed the third threshold, then not exceeding the limit is taken as the second judgment result.

4. The automatic stacking method for material frames based on machine learning according to claim 3, characterized in that, If the second determination result is an over-limit, then adjusting the position of the forks includes: If the second judgment result is out of limit, and the X-axis difference exceeds the second threshold, then control the fork to move sideways; If the second judgment result is out of limit, and the Z-axis difference exceeds the third threshold, then control the forks to move back and forth; If the second judgment result is out of limit, and the X-axis difference exceeds the second threshold and the Z-axis difference exceeds the third threshold, then control the fork to move sideways and forward and backward.

5. The automatic stacking method for material frames based on machine learning according to claim 1, characterized in that, Also includes: The stacking status of the upper and lower material frames is detected. If the stacking is not completed correctly, an abnormal alarm is triggered. If the stacking is completed correctly, control the fork-type mobile robot to move backward and exit the stacking operation.

6. The automatic stacking method for material frames based on machine learning according to claim 5, characterized in that, The system detects the stacking status of the upper and lower material frames. If the stacking is not completed correctly, an abnormal alarm is triggered. If the stacking is completed correctly, control the forklift robot to move backward and exit the stacking operation, including: Detect the planar angle between the upper material frame and the lower material frame; If none of the sides of the upper material frame and the lower material frame are on the same plane, it is determined that the stacking has not been completed correctly. If the sides of the upper and lower material frames are both on the same plane, then the stacking is considered to be completed correctly.

7. The automatic stacking method for material frames based on machine learning according to claim 1, characterized in that, Also includes: The performance of the key point detection model is evaluated.

8. The automatic stacking method for material frames based on machine learning according to claim 7, characterized in that, The performance evaluation of the key point detection model includes: Calculate the mean squared error result during the training of the deep neural network algorithm model, and determine whether the mean squared error result is similar to the prediction result of the test set data; If the mean square error result is similar to the prediction result of the test set, then the performance of the key point detection model is determined to meet the application standard. If the mean square error result is not similar to the prediction result of the test set data, it is determined that the performance of the key point detection model has not met the application standard.

9. An automatic stacking device for material frames based on machine learning, characterized in that, include: An acquisition unit is used to acquire sample image data captured by a camera, and to perform data augmentation, data preprocessing, and data annotation on the sample image data to obtain a dataset. The dataset is then divided into training set data and test set data. A deep neural network algorithm model is trained using the training set data to obtain a key point detection model, and the key point detection model is deployed to the forklift mobile robot. The data annotation includes annotating the X-axis and Z-axis coordinate positions of multiple bowls in the upper material frame and the X-axis and Z-axis coordinate positions of multiple columns in the lower material frame. The prediction unit is used to predict the X-axis and Z-axis coordinates of multiple bowls in the upper layer and the X-axis and Z-axis coordinates of multiple columns in the lower layer when the forklift mobile robot controls the forks to move the upper layer material frame above the lower layer material frame during stacking operations. An anomaly detection unit is used to determine whether the Z-axis difference between the far end of at least one side of the lower material frame and the near end of the same side exceeds the limit based on the Z-axis coordinate position of the column on at least one side of the lower material frame, and obtain a first detection result; if the first detection result is that it exceeds the limit, an anomaly alarm is triggered. An adjustment unit is configured to, if the first judgment result is within limits, move the forklift mobile robot until its bowl is nearly overlapping with the corresponding column of the lower material frame, and determine whether the X-axis difference and Z-axis difference between the column at the far end of the lower material frame and the bowl at the far end of the upper material frame exceed limits based on the X-axis and Z-axis coordinates of the column at the far end of the lower material frame and the bowl at the far end of the upper material frame, thereby obtaining a second judgment result; if the second judgment result is exceeding limits, adjust the position of the forks. The stacking unit is used to control the forks to stack the upper material frame onto the lower material frame if the second judgment result is within the limit.

10. A forklift mobile robot, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the machine learning-based automatic stacking method for material frames as described in any one of claims 1 to 8.

Images