Material Taking Method, Device and Computer Readable Storage Medium Based on a Three-Axis Manipulator

Abstract

The present application provides a material taking method, device and computer-readable storage medium based on a three-axis manipulator for automatically taking materials in a material placement area. The method includes: S1: Detect whether the materials in the material placement area meet the preset material taking conditions; if so, execute S2; if not, execute S4; S2: Based on the height distribution information of the materials in the material placement area, obtain the movement strategy of the three-axis manipulator; S3: Based on the movement strategy of the three-axis manipulator, use the drive assembly of the three-axis manipulator to drive the target picking assembly of the three-axis manipulator to move above one or more of the materials and pick up one or more of the materials, and execute S1; S4: After a preset time period, execute S1 again. It can intelligently determine the material taking order of each material, with high material taking efficiency and beautiful material stacking.

Description

Technical Field

[0001] This application relates to the technical fields of mechanical automation and intelligent manufacturing, and particularly to a material picking method, device, and computer-readable storage medium based on a three-axis manipulator. Background Art

[0002] A robotic arm (English: robotic arm) is an automatic control device that mimics the functions of a human arm and can perform various operations, allowing movement in a plane or three-dimensional space or using linear displacement for movement. The main requirement of a robotic arm is to complete the movements of the "wrist and hand". Classified by shape and size, robotic arms include types such as linear arms, SCARA arms, and articulated multi-axis robotic arms. A three-axis robotic arm used to perform a material picking task can adopt a linear arm. Specifically, a three-axis connection mechanism can be used, which has a similar shape to a gantry mechanism and is combined with a picking mechanism such as a suction cup or a gripper to form a linkage robotic arm.

[0003] Patent CN213415470U discloses a vision-guided automatic loading mechanism, which includes a feeding conveyor assembly, a guiding and detecting assembly located above the feeding conveyor assembly, a picking assembly located on one side of the feeding conveyor assembly and used for picking materials on the feeding conveyor assembly, and a loading conveyor assembly used for placing the products grabbed by the picking assembly; the guiding and detecting assembly includes a guiding linear module, a guiding transport frame installed on the guiding linear module, and at least one group of detection cameras installed on the guiding transport frame, and the detection end of the detection camera faces the feeding conveyor assembly. This mechanism performs visual detection on the products during the conveying process. After detecting the position of the products, the picking assembly grabs and places the products onto the loading conveyor assembly for conveying. However, when there are multiple materials placed in the material placement area and they cannot be picked and placed in one go, it is impossible to intelligently determine the picking order of each material. If any material is randomly picked each time, the complexity of the calculation process and the control process is high, the control difficulty is large, resulting in low picking efficiency and an unaesthetic material stack.

[0004] Based on this, this application provides a material picking method, device, and computer-readable storage medium based on a three-axis manipulator to improve the prior art. Summary of the Invention

[0005] The purpose of this application is to provide a material picking method, device, and computer-readable storage medium based on a three-axis manipulator, which can intelligently determine the picking order of each material, with high picking efficiency and an aesthetic material stack.

[0006] The purpose of this application is achieved by adopting the following technical solutions:

[0007] In a first aspect, this application provides a material picking method based on a three-axis manipulator for automatically picking materials in a material placement area. The method includes:

[0008] S1: Detect whether the materials in the material placement area meet the preset material picking conditions; if yes, execute S2; if not, execute S4;

[0009] S2: Based on the height distribution information of the materials in the material placement area, obtain the movement strategy of the three-axis manipulator, where the movement strategy is used to indicate the X-axis displacement, Y-axis displacement, and Z-axis displacement of the target picking component of the three-axis manipulator in a preset rectangular coordinate system;

[0010] S3: Based on the movement strategy of the three-axis manipulator, use the drive component of the three-axis manipulator to drive the target picking component of the three-axis manipulator to move above one or more of the materials and pick up one or more of the materials, then execute S1;

[0011] S4: After a preset time period, execute S1 again.

[0012] The beneficial effects of this technical solution are as follows: It can intelligently determine the picking order of each material, with high picking efficiency and beautiful material stacking.

[0013] First, detect whether the material placement area meets the preset material picking conditions (for example, there is a relatively large amount of materials stacked or the quality is relatively large in the material placement area), and execute different subsequent steps according to the judgment results; if yes, it indicates that the three-axis manipulator needs to be used for material picking at the current moment, so enter the movement strategy planning step; if not, it indicates that the three-axis manipulator does not need to be used for material picking at the current moment, so after a preset time period, execute the detection step again.

[0014] In the movement strategy planning step, the planning basis is the height distribution information of the materials in the material placement area (this height distribution information is used to indicate the material height at each point on the bearing surface of the material placement area). The advantage of this is that based on the height distribution, one or more materials to be picked each time are selected, so that the remaining materials in the material placement area meet different requirements in actual applications. For example, it can make the remaining materials in the material placement area always in a relatively uniform height distribution (a uniform height distribution means that the difference between the maximum value and the minimum value of the material height on the bearing surface of the material placement area does not exceed a preset height threshold), or make the remaining materials in the material placement area be picked up in sub-regions (that is, pick up another sub-region after one sub-region is picked up).

[0015] The height distribution of the materials is uniform, making it easy to meet the kinematic constraint conditions of the three-axis manipulator and avoid motion interference that may prevent the picking task from being completed. Interference refers to the contact between parts (the distance is less than the set clearance value, not necessarily zero). Motion interference means that interference occurs during the movement of parts, such as collisions and obstructions between the picking component and other materials (materials not to be picked this time).

[0016] The materials are picked in regions, which is suitable for the situation where materials need to be picked and placed into external equipment. It can greatly reduce the travel of the three-axis manipulator and improve the picking efficiency of the materials. For example, the material placement area can be divided into multiple sub-regions, namely sub-region A, sub-region B, sub-region C...; move the AGV to a position near sub-region A, and use the three-axis manipulator to pick the materials in sub-region A from the material placement area onto the AGV; after the materials in sub-region A are picked up, move the AGV to a position near sub-region B, and use the three-axis manipulator to pick the materials in sub-region B from the material placement area onto the AGV; after the materials in sub-region B are picked up, move the AGV to a position near sub-region C, and use the three-axis manipulator to pick the materials in sub-region C from the material placement area onto the AGV; and so on. The AGV can be successively moved to positions near each sub-region to complete the automatic transportation process of the materials in the corresponding sub-region. Compared with parking the AGV at a fixed position, it reduces the travel of the three-axis manipulator after picking up the materials and before placing them; compared with moving the AGV to a nearby position successively according to the position of the material picked each time, the number of movements of the AGV is less, reducing the linkage difficulty between the three-axis manipulator and the AGV.

[0017] After obtaining the movement strategy, the driving component of the three-axis manipulator can be used to drive the target picking component to move above the material and pick up the material, and one or more materials can be picked up each time. Among them, the three-axis manipulator is provided with multiple picking components, and the target picking component is one of the multiple picking components, that is, the picking component connected by the driving component when the three-axis manipulator executes the picking task.

[0018] The above picking process realizes a cycle based on the detection result of whether the picking conditions are met. The achieved effect is that for the picking requirements of different materials or the picking requirements of the same material at different stages, combined with the performance requirements and cost requirements in actual applications, different picking processes can be realized by setting different picking conditions, setting different preset durations, setting different single-pick quantities, etc., so as to intelligently determine the picking order of each material, with high intelligence, high picking efficiency, wide application range, meeting the picking requirements of various types of materials, enabling fine control of the entire picking process, and (compared with randomly picking materials each time) the visual effect formed by the material stacking is more beautiful.

[0019] In some alternative embodiments, the process of detecting whether the materials in the material placement area meet the material taking conditions includes:

[0020] Obtaining image information of the material placement area by using an image acquisition device;

[0021] Inputting the image information of the material placement area into an image recognition model to obtain the quantity of the materials in the material placement area;

[0022] When the quantity of the materials in the material placement area is not less than a preset quantity threshold, it is determined that the materials in the material placement area meet the material taking conditions;

[0023] When the quantity of the materials in the material placement area is less than the preset quantity threshold, it is determined that the materials in the material placement area do not meet the material taking conditions.

[0024] The beneficial effects of this technical solution are as follows: The image recognition model can achieve or even exceed the human image recognition ability in image recognition. By using the image recognition model to perform image recognition on the collected image information to obtain the quantity of materials, the labor cost can be greatly reduced, and the accuracy of the image recognition result is high, which is convenient for popularization and replication. In addition, a preset quantity threshold is set in advance, and the quantity of materials is used as the judgment basis for whether to take materials, which can avoid excessive accumulation of materials in the material placement area.

[0025] In some alternative embodiments, the step of inputting the image information of the material placement area into an image recognition model to obtain the quantity of the materials in the material placement area includes:

[0026] Inputting the image information of the material placement area into the image recognition model to obtain the quantity of the materials in the material placement area and the height distribution information.

[0027] The beneficial effects of this technical solution are as follows: On the one hand, using the image recognition model to output the quantity of materials and the height distribution information simultaneously has high calculation efficiency; on the other hand, it realizes the reuse of image information, that is, collecting the image information once, the collected image information is simultaneously used to obtain the quantity and the height distribution information. Compared with obtaining the image information separately and using it separately to obtain the quantity and the height distribution information, the number of image acquisitions is reduced, and the overall material taking efficiency is improved.

[0028] In some alternative embodiments, the training process of the image recognition model includes:

[0029] Obtaining a training set, where the training set includes a plurality of training data, and each training data includes sample image information of a sample material placement area and annotation data of the quantity and height distribution information of the materials in the sample material placement area;

[0030] For each piece of training data in the training set, perform the following processing:

[0031] Input the sample image information in the training data into a preset deep learning model to obtain prediction data on the quantity and height distribution information of the materials in the sample material placement area;

[0032] Based on the prediction data and annotation data on the quantity and height distribution information of the materials in the sample material placement area, update the model parameters of the deep learning model;

[0033] Detect whether a preset training end condition is satisfied; if so, use the trained deep learning model as the image recognition model; if not, continue to train the deep learning model using the next piece of training data.

[0034] The beneficial effects of this technical solution are as follows: Through design, by establishing an appropriate number of neuron computing nodes and a multi-layer operation hierarchy, and selecting appropriate input and output layers, a preset deep learning model can be obtained. Through the learning and optimization of this deep learning model, a functional relationship from input to output is established. Although the functional relationship between input and output cannot be found 100%, it can approximate the real correlation relationship as much as possible. Thus, the trained image recognition model can obtain corresponding output data (i.e., quantity and height distribution information) based on any input data (i.e., image information), with a wide range of applications, high accuracy, and high reliability of the calculation results.

[0035] In some alternative embodiments, the process of detecting whether the materials in the material placement area meet the material taking conditions includes:

[0036] Use a weighing sensor to obtain the mass of the materials in the material placement area;

[0037] When the mass of the materials in the material placement area is not less than the preset mass threshold, determine that the materials in the material placement area meet the material taking conditions;

[0038] When the mass of the materials in the material placement area is less than the preset mass threshold, determine that the materials in the material placement area do not meet the material taking conditions.

[0039] The beneficial effects of this technical solution are as follows: On the one hand, using a weighing sensor to obtain the mass of the materials has high mass acquisition efficiency and strong real-time performance; on the other hand, by presetting the preset mass threshold and using the mass of the materials as the basis for judging whether to take the materials, it can avoid the overloading of the materials in the material placement area, damaging the bearing surface of the material placement area, and causing the materials to pour into the surrounding area.

[0040] In some alternative embodiments, before the step S1, the method further includes:

[0041] Based on the types of materials in the material placement area, obtain the picking strategy of the three-axis manipulator, where the picking strategy is used to indicate one or more of the following picking parameters: the picking type, quantity, shape, size, and material type of the picking mechanism;

[0042] Based on the picking strategy of the three-axis manipulator, determine one of the multiple picking components as the target picking component;

[0043] Connect the driving component of the three-axis manipulator to the target picking component;

[0044] Based on the target picking component, determine the driving strategy of the three-axis manipulator;

[0045] When the driving component of the three-axis manipulator uses a motor, the driving strategy is used to indicate one or more of the following driving parameters: rotational speed, torque, output power, and power factor;

[0046] When the driving component of the three-axis manipulator uses a cylinder, the driving strategy is used to indicate one or more of the following driving parameters: output force, piston stroke, and piston movement speed.

[0047] The beneficial effects of this technical solution are as follows: Multiple picking components are preset in advance, and the picking parameters corresponding to each picking component are different, that is, they correspond to different picking types, quantities, shapes, sizes, or material types of the picking mechanism. For different types of materials, different picking strategies are adopted to determine the target picking component connected during the material picking process, and the driving strategy corresponding to the target picking component is adopted to achieve stable and efficient operation of the material picking process, which is conducive to achieving refined control, taking into account the picking efficiency and the safety during the picking process (including the safety of the material and the safety of the picking mechanism itself), avoiding damage to the material, and at the same time extending the service life of the picking mechanism. For example, for materials that are easily deformed under force, a lifting-type picking mechanism can be selected and the lifting method can be used to pick up the material to avoid deformation of the material; for materials that are prone to slipping, a picking mechanism with an anti-slip function can be selected to pick up the material to avoid slipping of the material; for materials arranged in a regular shape, multiple picking mechanisms can be selected to pick up multiple materials at a time to improve the picking efficiency; for materials with different shapes and sizes, picking mechanisms with corresponding shapes and sizes can be selected to improve the stability during the picking process and the moving process; for materials with different material types, picking mechanisms with different material types can be selected to avoid damage to the material; for materials that are prone to tipping over, a picking mechanism with an anti-tipping function can be selected, and the target picking component can be driven with appropriate driving parameters to achieve a lower moving speed and avoid tipping over of the material.

[0048] In some alternative embodiments, the movement strategy based on the three-axis manipulator uses the drive assembly of the three-axis manipulator to drive the target picking assembly of the three-axis manipulator to move above one or more of the materials and pick up one or more of the materials, including:

[0049] Based on the target picking assembly, determine the target quantity of the materials picked up each time;

[0050] Based on the movement strategy of the three-axis manipulator, use the drive assembly of the three-axis manipulator to drive the target picking assembly of the three-axis manipulator to move above the target quantity of materials and pick up the target quantity of materials.

[0051] The beneficial effect of this technical solution is that after determining the target picking assembly, the target quantity of the materials picked up each time is also determined. Therefore, after moving the three-axis manipulator to the specified position corresponding to the movement strategy, the target picking assembly can be used to pick up the target quantity of materials.

[0052] In some alternative embodiments, the method further includes:

[0053] Control the three-axis manipulator to place the picked-up materials on the bearing surface of the AGV, so that the AGV transports the placed materials to the target position.

[0054] The beneficial effect of this technical solution is that the AGV can be used to move the picked-up materials to the preset target position, realizing the automatic transportation process of the materials. AGV is the abbreviation of Automated Guided Vehicle, usually also called an AGV cart, which refers to a transport vehicle equipped with automatic navigation devices such as electromagnetic or optical ones, capable of traveling along a predetermined driving path, having safety protection and various transfer functions. Its driving path can be flexibly changed according to different needs, with the advantages of high automation level, fast movement, small floor area, high positioning accuracy, etc., greatly reducing the labor cost, and being simple to install and debug, adapting to the development trend of flexible manufacturing, and improving the safety and standardization of industrial production.

[0055] In a second aspect, the present application provides a material picking device based on a three-axis manipulator for automatically picking materials in a material placement area. The device includes a processor configured to implement the following steps:

[0056] S1: Detect whether the materials in the material placement area meet the preset material picking conditions; if they meet, execute S2; if they do not meet, execute S4;

[0057] S2: Based on the height distribution information of the materials in the material placement area, obtain the movement strategy of the three-axis manipulator, where the movement strategy is used to indicate the X-axis displacement, Y-axis displacement, and Z-axis displacement of the target picking component of the three-axis manipulator in a preset rectangular coordinate system;

[0058] S3: Based on the movement strategy of the three-axis manipulator, use the driving component of the three-axis manipulator to drive the target picking component of the three-axis manipulator to move above one or more of the materials and pick up one or more of the materials, and execute S1;

[0059] S4: After a preset time period, execute S1 again.

[0060] In some alternative embodiments, the processor is configured to detect whether the materials in the material placement area meet the material picking conditions in the following manner:

[0061] Obtain the image information of the material placement area by using an image acquisition device;

[0062] Input the image information of the material placement area into an image recognition model to obtain the quantity of the materials in the material placement area;

[0063] When the quantity of the materials in the material placement area is not less than a preset quantity threshold, determine that the materials in the material placement area meet the material picking conditions;

[0064] When the quantity of the materials in the material placement area is less than the preset quantity threshold, determine that the materials in the material placement area do not meet the material picking conditions.

[0065] In some alternative embodiments, the processor is configured to obtain the quantity of the materials in the material placement area in the following manner:

[0066] Input the image information of the material placement area into the image recognition model to obtain the quantity and height distribution information of the materials in the material placement area.

[0067] In some alternative embodiments, the training process of the image recognition model includes:

[0068] Obtain a training set, where the training set includes a plurality of training data, and each training data includes the sample image information of a sample material placement area and the annotation data of the quantity and height distribution information of the materials in the sample material placement area;

[0069] For each training data in the training set, perform the following processing:

[0070] Input the sample image information in the training data into a preset deep learning model to obtain prediction data on the quantity and height distribution information of the materials in the sample material placement area;

[0071] Update the model parameters of the deep learning model based on the prediction data and annotation data on the quantity and height distribution information of the materials in the sample material placement area;

[0072] Detect whether a preset training end condition is satisfied; if so, use the trained deep learning model as the image recognition model; if not, continue to train the deep learning model using the next piece of training data.

[0073] In some alternative embodiments, the processor is configured to detect whether the materials in the material placement area meet the material taking conditions in the following manner:

[0074] Obtain the mass of the materials in the material placement area using a weighing sensor;

[0075] When the mass of the materials in the material placement area is not less than the preset mass threshold, determine that the materials in the material placement area meet the material taking conditions;

[0076] When the mass of the materials in the material placement area is less than the preset mass threshold, determine that the materials in the material placement area do not meet the material taking conditions.

[0077] In some alternative embodiments, the processor is further configured to implement the following steps before implementing step S1:

[0078] Based on the types of materials in the material placement area, obtain the picking strategy of the three-axis manipulator, where the picking strategy is used to indicate one or more of the following picking parameters: the picking type, quantity, shape, size, and material type of the picking mechanism;

[0079] Based on the picking strategy of the three-axis manipulator, determine one of the multiple picking components as the target picking component;

[0080] Connect the drive component of the three-axis manipulator to the target picking component;

[0081] Based on the target picking component, determine the drive strategy of the three-axis manipulator;

[0082] When the drive component of the three-axis manipulator uses a motor, the drive strategy is used to indicate one or more of the following drive parameters: rotational speed, torque, output power, and power factor;

[0083] When the driving component of the three-axis manipulator adopts a cylinder, the driving strategy is used to indicate one or more of the following driving parameters: output force, piston stroke, and piston movement speed.

[0084] In some alternative embodiments, the processor is configured to pick up one or more of the materials in the following manner:

[0085] Based on the target picking component, determine the target quantity of the material picked up each time;

[0086] Based on the movement strategy of the three-axis manipulator, use the driving component of the three-axis manipulator to drive the target picking component of the three-axis manipulator to move above the target quantity of materials and pick up the target quantity of materials.

[0087] In some alternative embodiments, the processor is further configured to implement the following steps:

[0088] Control the three-axis manipulator to place the picked-up material on the carrying surface of the AGV, so that the AGV transports the placed material to the target position.

[0089] In a third aspect, the present application provides a computer-readable storage medium storing a computer program, which when executed by a processor, implements the steps of any one of the above methods or implements the functions of any one of the above devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] The present application will be further described below with reference to the drawings and embodiments.

[0091] Figure 1 Shows a schematic structural diagram of a three-axis manipulator provided by an embodiment of the present application.

[0092] Figure 2 Shows a schematic flow chart of a material picking method based on a three-axis manipulator provided by an embodiment of the present application.

[0093] Figure 3 Shows a schematic flow chart of determining a driving strategy of a three-axis manipulator provided by an embodiment of the present application.

[0094] Figure 4 Shows a schematic block diagram of a material picking device based on a three-axis manipulator provided by an embodiment of the present application.

[0095] Figure 5 Shows a schematic structural diagram of a program product provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0096] The technical solution in the present application will be described below in conjunction with the drawings and specific implementation methods of the specification of the present application. It should be noted that, under the premise of no conflict, the various implementation methods or technical features described below can be arbitrarily combined to form a new implementation method.

[0097] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, a and b and c, where a, b and c can be single or multiple. It is worth noting that "at least one" can also be interpreted as "one or more items".

[0098] It should also be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any implementation or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other implementations or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0099] (Method Example)

[0100] See also Figure 1 and Figure 2 , Figure 1 A schematic diagram of the structure of a three-axis manipulator provided in an embodiment of the present application is shown. Figure 2 A schematic flow chart of a material picking method based on a three-axis manipulator provided in an embodiment of the present application is shown.

[0101] The three-axis manipulator in the embodiment of the present application can move in the X-axis direction, the Y-axis direction, and the Z-axis direction. Specifically, the three-axis manipulator is provided with a drive component and a pickup component, and the drive component drives the pickup component to move in the X-axis direction, the Y-axis direction, and the Z-axis direction.

[0102] The embodiment of the present application provides a material picking method based on a three-axis manipulator, which is used to automatically pick up materials from a material placement area. The method includes:

[0103] S1: Detect whether the materials in the material placement area meet the preset material picking conditions; if they do, execute S2; if not, execute S4;

[0104] S2: Based on the height distribution information of the materials in the material placement area, obtain the movement strategy of the three-axis manipulator, where the movement strategy is used to indicate the X-axis displacement, Y-axis displacement, and Z-axis displacement of the target picking component of the three-axis manipulator in a preset rectangular coordinate system;

[0105] S3: Based on the movement strategy of the three-axis manipulator, use the driving component of the three-axis manipulator to drive the target picking component of the three-axis manipulator to move above one or more of the materials and pick up one or more of the materials, then execute S1;

[0106] S4: After a preset time period, execute S1 again.

[0107] In this way, the picking order of each material is determined intelligently, and the driving component is controlled to drive the target picking component to move above the corresponding material to pick up the material, with high picking efficiency and beautiful stacking of materials.

[0108] First, detect whether the material placement area meets the preset material picking conditions (for example, there are relatively many or heavy materials stacked in the material placement area), and execute different subsequent steps according to the judgment result; if it meets, it indicates that the three-axis manipulator needs to be used for material picking at the current moment, so enter the planning step of the movement strategy; if not, it indicates that the three-axis manipulator does not need to be used for material picking at the current moment, so after a preset time period, execute the detection step again.

[0109] In the planning step of the movement strategy, the planning basis is the height distribution information of the materials in the material placement area (this height distribution information is used to indicate the material height at each point on the bearing surface of the material placement area). The advantage of this is that one or more materials to be picked each time are selected based on the height distribution, so that the remaining materials in the material placement area meet different requirements in practical applications. For example, it can make the remaining materials in the material placement area always be in a relatively uniform height distribution (uniform height distribution means that the difference between the maximum and minimum values of the material height on the bearing surface of the material placement area is not greater than a preset height threshold), or make the remaining materials in the material placement area be picked up in sub-regions (that is, pick up another sub-region after one sub-region is picked up).

[0110] The material height is evenly distributed, making it easy to meet the kinematic constraint conditions of the three-axis manipulator and avoid motion interference that may prevent the pick-up task from being completed. Interference refers to the contact between parts (the distance is less than the set clearance value, not necessarily zero). Motion interference refers to the occurrence of interference during the movement of parts, such as collisions and obstructions between the pick-up component and other materials (materials other than the material to be picked up this time).

[0111] The materials are picked up in zones, which is suitable for the situation where materials need to be picked up and placed on external equipment. It can greatly reduce the travel of the three-axis manipulator and improve the material pick-up efficiency. For example, the material placement area can be divided into multiple sub-zones, namely sub-zone A, sub-zone B, sub-zone C...; move the AGV to a position near sub-zone A, and use the three-axis manipulator to pick up the materials in sub-zone A from the material placement area onto the AGV; after the materials in sub-zone A are picked up, move the AGV to a position near sub-zone B, and use the three-axis manipulator to pick up the materials in sub-zone B from the material placement area onto the AGV; after the materials in sub-zone B are picked up, move the AGV to a position near sub-zone C, and use the three-axis manipulator to pick up the materials in sub-zone C from the material placement area onto the AGV; and so on. The AGV can be moved to a position near each sub-zone in turn to complete the automatic transportation process of the materials in the corresponding sub-zone. Compared with parking the AGV in a fixed position, it reduces the travel of the three-axis manipulator after picking up the material and before placing the material; compared with moving the AGV to a position near the material each time according to the position of the picked material, the number of times the AGV moves is less, reducing the linkage difficulty between the three-axis manipulator and the AGV.

[0112] After obtaining the movement strategy, the driving component of the three-axis manipulator can be used to drive the target pick-up component to move above the material and pick up the material, and one material or multiple materials can be picked up each time.

[0113] The above pick-up process is realized in a loop based on the detection results of whether the pick-up conditions are met. The achieved effect is that for the pick-up requirements of different materials or the pick-up requirements of the same material at different stages, combined with the performance requirements and cost requirements in actual applications, different pick-up processes can be realized by setting different pick-up conditions, setting different preset durations, setting different single pick-up quantities, etc., so as to intelligently determine the pick-up order of each material, with high intelligence, high pick-up efficiency, wide application range, meeting the pick-up requirements of various types of materials, enabling fine control of the entire pick-up process, and (compared with randomly picking up materials each time) the visual effect formed by the material stacking is more beautiful.

[0114] In the embodiments of the present application, the material placement area can be set indoors or outdoors, for example. As an example, the material placement area can be set inside the factory building.

[0115] The material placement area is provided with a bearing surface for bearing materials. The shape of the bearing surface can be circular, oval, polygonal, bow-shaped, arc-shaped, annular, racetrack-shaped, etc. Among them, polygons include triangles, quadrilaterals, pentagons, hexagons, octagons, decagons, etc.

[0116] The embodiments of the present application do not limit the quantity of materials in the material placement area, which can be, for example, 0, 1, 3, 5, 10, 50, 100, 1000, 10000, 100000, etc.

[0117] The embodiments of the present application do not limit the shape of the materials in the material placement area, which can be, for example, a cylinder, a cone, a solid of revolution, a sectional body, or an irregular shape. Among them, cylinders include circular cylinders and prisms, cones include circular cones and pyramids, solids of revolution include circular cylinders, frustums of cones, circular cones, spheres, ellipsoids, spherical caps, bow rings, toroids, dike rings, sector rings, jujube pit shapes, etc., and sectional bodies include frustums of prisms, frustums of cones, obliquely truncated circular cylinders, obliquely truncated prisms, obliquely truncated circular cones, spherical caps, spherical segments, etc.

[0118] The embodiments of the present application do not limit the size of the materials in the material placement area, and the size can be, for example, in the order of millimeters, centimeters, decimeters, meters, etc.

[0119] The embodiments of the present application do not limit the material type of the materials in the material placement area, which can be, for example, metal materials, organic polymer materials, inorganic non-metallic materials, composite materials, etc. Among them, metal materials include metals and alloys, organic polymer materials include synthetic plastics, fibers, rubbers, natural wool and cotton, etc., inorganic non-metallic materials include glass, ceramics, etc., and composite materials are composed of two or more materials, such as cement, wood, etc.

[0120] In the embodiments of the present application, the materials can be packaged or unpackaged. As an example, the materials in the material placement area are lipsticks or sneakers with packaging boxes. As another example, the materials in the material placement area are packages with packaging bags. As yet another example, the materials in the material placement area are unpackaged chips. As yet another example, the materials in the material placement area are bagged rice.

[0121] In the embodiments of the present application, the height distribution information of the materials in the material placement area is used to indicate the height of the materials at each point on the bearing surface of the material placement area. Each point on the bearing surface here refers to each sampling point on the bearing surface. The sampling points on the bearing surface of the material placement area can be arranged in the shape of M rows and N columns, for example. The row spacing and column spacing between the sampling points can adopt a preset distance, which can be 1 micron, 1 millimeter, 1 centimeter, etc., for example. M and N are integers greater than 1. Alternatively, the sampling points on the bearing surface of the material placement area can be arranged in the shape of multiple concentric circles, for example.

[0122] In the embodiments of the present application, the rectangular coordinate system, that is, the Cartesian coordinate system, is an orthogonal coordinate system in mathematics. The X-axis displacement, Y-axis displacement, and Z-axis displacement in the preset rectangular coordinate system respectively refer to the displacements in the X-axis direction, Y-axis direction, and Z-axis direction in the preset rectangular coordinate system.

[0123] In the embodiments of the present application, the three-axis manipulator is provided with multiple picking components. The target picking component is one of the multiple picking components, that is, the picking component connected to the driving component when the three-axis manipulator executes the picking task (i.e., the material picking task).

[0124] The embodiments of the present application do not limit the preset duration, which can be 3 seconds, 5 seconds, 10 seconds, 30 seconds, 1 minute, 3 minutes, 5 minutes, 10 minutes, etc., for example. The preset duration should not be set too long or too short. If the preset duration is set too long, it is impossible to detect in real time whether the material picking condition is met and pick the material in a timely manner; if the preset duration is set too short, more computing resources will be consumed, which is not conducive to energy conservation and emission reduction.

[0125] In some alternative embodiments, the process of detecting whether the materials in the material placement area meet the material picking condition includes:

[0126] Obtain the image information of the material placement area by using an image acquisition device;

[0127] Input the image information of the material placement area into an image recognition model to obtain the quantity of the materials in the material placement area;

[0128] When the quantity of the materials in the material placement area is not less than a preset quantity threshold, it is determined that the materials in the material placement area meet the material picking condition;

[0129] When the quantity of the materials in the material placement area is less than the preset quantity threshold, it is determined that the materials in the material placement area do not meet the material picking condition.

[0130] Thus, the image recognition model can achieve or even exceed the human image recognition ability in image recognition. By using the image recognition model to perform image recognition on the collected image information to obtain the quantity of materials, the labor cost can be greatly reduced, and the accuracy of the image recognition result is high, which is convenient for popularization and replication. In addition, by presetting a preset quantity threshold and using the quantity of materials as the judgment basis for whether to take materials, it is possible to avoid excessive accumulation of materials in the material placement area.

[0131] The embodiments of the present application do not limit the preset quantity threshold, which may be, for example, 1, 2, 3, 5, 10, 30, 50, 100, 1000, etc.

[0132] The embodiments of the present application do not limit the image acquisition device, which may include, for example, one or more cameras. The cameras include optical cameras and / or infrared cameras.

[0133] In some alternative embodiments, the step of inputting the image information of the material placement area into the image recognition model to obtain the quantity of materials in the material placement area includes:

[0134] Inputting the image information of the material placement area into the image recognition model to obtain the quantity and height distribution information of the materials in the material placement area.

[0135] Thus, on the one hand, by using the image recognition model to simultaneously output the quantity and height distribution information of the materials, the calculation efficiency is high; on the other hand, the reuse of image information is realized, that is, by collecting the image information once, the collected image information is simultaneously used to obtain the quantity and height distribution information. Compared with separately obtaining the image information and separately using it to obtain the quantity and height distribution information, the number of image acquisitions is reduced, and the overall material taking efficiency is improved.

[0136] In some alternative embodiments, the training process of the image recognition model includes:

[0137] Obtaining a training set, the training set includes a plurality of training data, and each training data includes sample image information of a sample material placement area and annotation data of the quantity and height distribution information of the materials in the sample material placement area;

[0138] For each training data in the training set, perform the following processing:

[0139] Inputting the sample image information in the training data into a preset deep learning model to obtain prediction data of the quantity and height distribution information of the materials in the sample material placement area;

[0140] Updating the model parameters of the deep learning model based on the prediction data and labeled data of the quantity and height distribution information of the materials in the sample material placement area;

[0141] Detecting whether a preset training end condition is satisfied; if so, using the trained deep learning model as the image recognition model; if not, continuing to train the deep learning model with the next piece of training data.

[0142] Thus, through design, by establishing an appropriate number of neuron computing nodes and a multi-layer operation hierarchy, and selecting appropriate input and output layers, a preset deep learning model can be obtained. Through the learning and optimization of this deep learning model, a functional relationship from input to output can be established. Although the functional relationship between input and output cannot be found 100%, it can approximate the real correlation relationship as much as possible. The image recognition model trained thereby can obtain corresponding output data (i.e., quantity and height distribution information) based on any input data (i.e., image information), has a wide range of applications, and high accuracy and reliability of calculation results.

[0143] In some alternative embodiments, an image recognition model can be trained in the embodiments of the present application. In some other alternative embodiments, a pre-trained image recognition model can be used in the present application.

[0144] In some alternative embodiments, for example, historical data can be mined to obtain sample image information of the sample material placement area in the training set. That is to say, the sample image information of these sample material placement areas can be the image information obtained by image acquisition of the real material placement area. In addition, the sample image information of the sample material placement area can also be automatically generated by the generation network of the GAN model.

[0145] Among them, the GAN model, namely the Generative Adversarial Network, consists of a generation network and a discriminant network. The generation network randomly samples from the latent space as input, and its output result needs to mimic the real samples in the training set as much as possible. The input of the discriminant network is either the real sample or the output of the generation network, and its purpose is to distinguish the output of the generation network from the real samples as much as possible. And the generation network has to deceive the discriminant network as much as possible. The two networks confront each other and continuously adjust the parameters. The ultimate goal is to make the discriminant network unable to judge whether the output result of the generation network is real. Using the GAN model can generate sample image information of multiple sample material placement areas for the training process of the image recognition model, which can effectively reduce the amount of data collected from the original data and greatly reduce the costs of data collection and annotation.

[0146] The present application does not limit the method for obtaining labeled data. For example, the manual labeling method can be adopted, or the automatic labeling or semi-automatic labeling method can be adopted. When the sample image information of the sample material placement area is the image information obtained by collecting images of the actual material placement area, the real data can be obtained from the historical data by keyword extraction as the labeled data.

[0147] The present application does not limit the training process of the image recognition model. For example, the above-mentioned supervised learning training method can be adopted, or the semi-supervised learning training method can be adopted, or the unsupervised learning training method can be adopted.

[0148] The present application does not limit the preset training end condition. For example, it can be that the number of training times reaches the preset number of times (the preset number of times is, for example, 1 time, 3 times, 10 times, 100 times, 1000 times, 10000 times, etc.), or it can be that all the training data in the training set have completed one or more trainings, or it can be that the total loss value obtained from this training is not greater than the preset loss value.

[0149] In some alternative embodiments, the process of detecting whether the material in the material placement area meets the material taking condition includes:

[0150] Obtaining the mass of the material in the material placement area by using a weighing sensor;

[0151] When the mass of the material in the material placement area is not less than the preset mass threshold, it is determined that the material in the material placement area meets the material taking condition;

[0152] When the mass of the material in the material placement area is less than the preset mass threshold, it is determined that the material in the material placement area does not meet the material taking condition.

[0153] Thus, on the one hand, the mass of the material is obtained by using a weighing sensor, and the mass acquisition efficiency is high and the real-time performance is strong; on the other hand, the preset mass threshold is set in advance, and the mass of the material is used as the judgment basis for whether to take the material, which can avoid the material in the material placement area from piling up too heavily and damaging the bearing surface, resulting in the material pouring into the surrounding area.

[0154] The present application embodiment does not limit the preset mass threshold. For example, it can be 1 milligram, 2 milligrams, 3 milligrams, 5 milligrams, 10 milligrams, 30 milligrams, 50 milligrams, 100 milligrams, 1 kilogram, 2 kilograms, 3 kilograms, 5 kilograms, 10 kilograms, 30 kilograms, 50 kilograms, 100 kilograms, 1000 kilograms, etc.

[0155] In some alternative embodiments, the process of detecting whether the material in the material placement area meets the material taking condition includes:

[0156] Obtain the mass of the material in the material placement area using a load cell;

[0157] Calculate the quantity of the material in the material placement area using the mass of the material in the material placement area and the mass of a single material;

[0158] When the quantity of the material in the material placement area is not less than a preset quantity threshold, determine that the material in the material placement area meets the material taking condition;

[0159] When the quantity of the material in the material placement area is less than the preset quantity threshold, determine that the material in the material placement area does not meet the material taking condition.

[0160] That is to say, the corresponding quantity can be calculated using the mass of the material in the material placement area and the mass of a single material. Among them, the mass of a single material can be manually input, or obtained by measuring the mass of several randomly selected materials and calculating the average value.

[0161] See Figure 3 , Figure 3 shows a schematic flow chart of a method for determining a driving strategy of a three-axis manipulator provided by an embodiment of the present application.

[0162] In some alternative embodiments, before the S1, the method further includes:

[0163] F1: Based on the type of the material in the material placement area, obtain a picking strategy of the three-axis manipulator, where the picking strategy is used to indicate one or more of the following picking parameters: the picking type, quantity, shape, size, and material type of the picking mechanism;

[0164] F2: Based on the picking strategy of the three-axis manipulator, determine one of the multiple picking components as the target picking component;

[0165] F3: Connect the driving component of the three-axis manipulator to the target picking component;

[0166] F4: Based on the target picking component, determine the driving strategy of the three-axis manipulator;

[0167] When the driving component of the three-axis manipulator uses a motor, the driving strategy is used to indicate one or more of the following driving parameters: rotational speed, torque, output power, and power factor;

[0168] When the driving component of the three-axis manipulator uses a cylinder, the driving strategy is used to indicate one or more of the following driving parameters: output force, piston stroke, and piston movement speed.

[0169] Therefore, multiple picking components are preset, and the picking parameters corresponding to each picking component are different, that is, corresponding to different picking types, quantities, shapes, sizes or material types of the picking mechanisms.

[0170] In the embodiments of the present application, the picking types of the picking mechanisms may include, for example, one or more of suction, grasping and hooking.

[0171] In the embodiments of the present application, the number of the picking mechanisms may be, for example, 1, 2, 3, 4, etc. Each suction-type picking mechanism may be provided with one or more air holes, each grasping-type picking mechanism may be provided with 2, 4, 8 or more jaws, and each hooking-type picking mechanism may be provided with one or more hooking ends.

[0172] In the embodiments of the present application, the shape of the picking mechanism is not limited. The suction-type picking mechanism may be, for example, a flat suction cup, an oval suction cup, a corrugated suction cup, a special-shaped suction cup, etc. The grasping-type picking mechanism may be provided with, for example, two-finger jaws, three-finger jaws or special-shaped jaws. The hooking ends of the hooking-type picking mechanism may move closer to each other inward or extend outward.

[0173] In the embodiments of the present application, the size of the picking mechanism is not limited, and its size may be, for example, at the millimeter level, centimeter level, decimeter level, meter level, etc.

[0174] In the embodiments of the present application, the material type of the picking mechanism is not limited, and it may be, for example, a metal material, an organic polymer material, an inorganic non-metal material, a composite material, etc.

[0175] For different types of materials, different picking strategies are adopted to determine the target picking component connected during the material picking process, and the driving strategy corresponding to the target picking component is adopted to achieve the stable and efficient operation of the material picking process, which is beneficial to realizing refined control, taking into account the picking efficiency and the safety during the picking process (including the safety of the material and the safety of the picking mechanism itself), avoiding damage to the material, and at the same time extending the service life of the picking mechanism.

[0176] For example, for materials that are easily deformed under force, a lifting-type picking mechanism can be selected and the materials can be picked up in a lifting manner to avoid deformation of the materials; for materials that are prone to slipping, a picking mechanism with an anti-slip function can be selected to pick up the materials to avoid slipping; for materials arranged in a regular shape, multiple picking mechanisms can be selected to pick up multiple materials at a time to improve the picking efficiency; for materials of different shapes and sizes, picking mechanisms corresponding to the shapes and sizes can be selected to improve the stability during the picking process and the moving process; for materials of different material types, picking mechanisms of different material types can be selected to avoid damaging the materials; for materials that are prone to tipping over, a picking mechanism with an anti-tipping function can be selected, and the target picking component can be driven with appropriate driving parameters to achieve a relatively low moving speed and avoid tipping over of the materials.

[0177] As an example, the type of the materials in the material placement area is lipstick and they have packaging, and the packaging is a cuboid paper box. The picking strategy of the three-axis manipulator corresponding to this material type can be, for example, "the picking type of the picking mechanism is suction type, the number is 4, the shape is a flat suction cup, the size is 3 cm in diameter, and the material type is nitrile rubber (NBR)". One of the picking components whose picking parameters meet the requirements is determined from multiple picking components as the target picking component, the driving component of the three-axis manipulator is connected to the target picking component, and based on the target picking component, the driving strategy of the three-axis manipulator is determined as "the output force is 0.5 Newton, the piston stroke is 10 mm, and the piston moving speed is 50 - 500 mm per second".

[0178] In some alternative embodiments, based on the moving strategy of the three-axis manipulator, driving the target picking component of the three-axis manipulator to move above one or more of the materials and pick up one or more of the materials by using the driving component of the three-axis manipulator (i.e., step S3) includes:

[0179] Based on the target picking component, determining the target number of materials picked up each time;

[0180] Based on the moving strategy of the three-axis manipulator, driving the target picking component of the three-axis manipulator to move above the target number of materials and pick up the target number of materials.

[0181] Thus, after determining the target picking component, the target number of materials picked up each time is also determined. Therefore, after moving the three-axis manipulator to the specified position corresponding to the moving strategy, the target number of materials can be picked up by using the target picking component.

[0182] As an example, when the target picking component is provided with 4 vacuum flat suction cups, the target number of materials picked up each time is 4.

[0183] In some alternative embodiments, the method further includes:

[0184] Controlling the three-axis manipulator to place the picked material on the carrying surface of the AGV, so that the AGV transports the placed material to the target position.

[0185] Thus, the picked material can be moved to a preset target position by using the AGV, realizing the automatic transportation process of the material. AGV is the abbreviation of Automated Guided Vehicle, usually also called an AGV cart, which refers to a transport vehicle equipped with automatic navigation devices such as electromagnetic or optical ones, capable of traveling along a predetermined driving path, having safety protection and various transfer functions. Its driving path can be flexibly changed according to different requirements, with the advantages of high automation, fast movement, small floor area, high positioning accuracy, etc., greatly reducing the labor cost, and being simple to install and debug, adapting to the development trend of flexible manufacturing, and improving the safety and standardization of industrial production.

[0186] In the embodiments of the present application, the target position can be set manually or intelligently. When the intelligent setting method is adopted, the intelligent warehousing management system based on the AGV uses the AGV as a carrying platform, takes the intelligent warehousing design and management optimization algorithm as the core, and intelligently sets the target position of each AGV through multi-AGV cooperation and scheduling technology. Combining with the warehousing management software and the interface of automated logistics equipment, it can realize functions such as automatic transportation and automatic picking of the AGV, achieving a high degree of automation in the whole process of warehousing, loading and unloading, handling, stacking, storage, picking, packaging, outbound, and shipping, thereby improving the logistics turnover efficiency, ensuring the timeliness and accuracy of logistics supply, and realizing the flexible storage function.

[0187] In a specific application scenario, the embodiments of the present application also provide a material picking method based on a three-axis manipulator for automatically picking materials in a material placement area. The method includes:

[0188] F1: Based on the types of materials in the material placement area, obtain the picking strategy of the three-axis manipulator, and the picking strategy is used to indicate one or more of the following picking parameters: the picking type, quantity, shape, size, and material type of the picking mechanism;

[0189] F2: Based on the picking strategy of the three-axis manipulator, determine one of the multiple picking components as the target picking component;

[0190] F3: Connect the driving component of the three-axis manipulator to the target picking component;

[0191] F4: Based on the target picking component, determine the driving strategy of the three-axis manipulator; when the driving component of the three-axis manipulator uses a motor, the driving strategy is used to indicate one or more of the following driving parameters: rotational speed, torque, output power, and power factor; when the driving component of the three-axis manipulator uses a cylinder, the driving strategy is used to indicate one or more of the following driving parameters: output force, piston stroke, and piston movement speed;

[0192] S1: Detect whether the materials in the material placement area meet the preset picking conditions; if so, execute S2; if not, execute S4;

[0193] S2: Based on the height distribution information of the materials in the material placement area, obtain the movement strategy of the three-axis manipulator, and the movement strategy is used to indicate the X-axis displacement, Y-axis displacement, and Z-axis displacement of the target picking component of the three-axis manipulator in a preset rectangular coordinate system;

[0194] S3: Based on the movement strategy of the three-axis manipulator, use the driving component of the three-axis manipulator to drive the target picking component of the three-axis manipulator to move above one or more of the materials and pick up one or more of the materials, and then execute S1;

[0195] S4: After a preset time period, execute S1 again;

[0196] After S3, the method further includes: controlling the three-axis manipulator to place the picked materials on the carrying surface of the AGV, so that the AGV transports the placed materials to the target position.

[0197] Among them, the process of detecting whether the materials in the material placement area meet the picking conditions includes:

[0198] Use an image acquisition device to obtain the image information of the material placement area;

[0199] Input the image information of the material placement area into the image recognition model to obtain the quantity and height distribution information of the materials in the material placement area;

[0200] When the quantity of the materials in the material placement area is not less than the preset quantity threshold, determine that the materials in the material placement area meet the picking conditions;

[0201] When the quantity of the materials in the material placement area is less than the preset quantity threshold, determine that the materials in the material placement area do not meet the picking conditions.

[0202] Among them, the training process of the image recognition model includes:

[0203] Obtain a training set, where the training set includes a plurality of training data, and each piece of training data includes sample image information of a sample material placement area and annotation data on the quantity and height distribution information of the materials in the sample material placement area;

[0204] For each piece of training data in the training set, perform the following processing:

[0205] Input the sample image information in the training data into a preset deep learning model to obtain prediction data on the quantity and height distribution information of the materials in the sample material placement area;

[0206] Based on the prediction data and annotation data on the quantity and height distribution information of the materials in the sample material placement area, update the model parameters of the deep learning model;

[0207] Detect whether a preset training end condition is met; if so, use the trained deep learning model as the image recognition model; if not, continue to train the deep learning model using the next piece of training data.

[0208] Among them, the moving strategy based on the three-axis manipulator uses the driving component of the three-axis manipulator to drive the target picking component of the three-axis manipulator to move above one or more of the materials and pick up one or more of the materials, including: based on the target picking component, determine the target quantity of the materials picked up each time; based on the moving strategy of the three-axis manipulator, use the driving component of the three-axis manipulator to drive the target picking component of the three-axis manipulator to move above the target quantity of materials and pick up the target quantity of materials.

[0209] (Device embodiment)

[0210] An embodiment of the present application further provides a material picking device based on a three-axis manipulator. Its specific embodiment is consistent with the embodiment and the achieved technical effects described in the above method embodiment, and some content will not be repeated.

[0211] The material picking device based on the three-axis manipulator is used for automatically picking materials in a material placement area. The device includes a processor, and the processor is configured to implement the following steps:

[0212] S1: Detect whether the materials in the material placement area meet the preset picking conditions; if they meet, execute S2; if they do not meet, execute S4;

[0213] S2: Based on the height distribution information of the materials in the material placement area, obtain a movement strategy for the three-axis manipulator, where the movement strategy is used to indicate the X-axis displacement, Y-axis displacement, and Z-axis displacement of the target picking component of the three-axis manipulator in a preset rectangular coordinate system;

[0214] S3: Based on the movement strategy of the three-axis manipulator, use the driving component of the three-axis manipulator to drive the target picking component of the three-axis manipulator to move above one or more of the materials and pick up one or more of the materials, and execute S1;

[0215] S4: After a preset time period, execute S1 again.

[0216] In some alternative embodiments, the processor is configured to detect whether the materials in the material placement area meet the material picking conditions in the following manner:

[0217] Use an image acquisition device to obtain the image information of the material placement area;

[0218] Input the image information of the material placement area into an image recognition model to obtain the quantity of the materials in the material placement area;

[0219] When the quantity of the materials in the material placement area is not less than a preset quantity threshold, determine that the materials in the material placement area meet the material picking conditions;

[0220] When the quantity of the materials in the material placement area is less than the preset quantity threshold, determine that the materials in the material placement area do not meet the material picking conditions.

[0221] In some alternative embodiments, the processor is configured to obtain the quantity of the materials in the material placement area in the following manner:

[0222] Input the image information of the material placement area into the image recognition model to obtain the quantity and height distribution information of the materials in the material placement area.

[0223] In some alternative embodiments, the training process of the image recognition model includes:

[0224] Obtain a training set, where the training set includes a plurality of training data, and each training data includes the sample image information of a sample material placement area and the annotation data of the quantity and height distribution information of the materials in the sample material placement area;

[0225] For each training data in the training set, perform the following processing:

[0226] Input the sample image information in the training data into a preset deep learning model to obtain prediction data on the quantity and height distribution information of the materials in the sample material placement area;

[0227] Update the model parameters of the deep learning model based on the prediction data and annotation data on the quantity and height distribution information of the materials in the sample material placement area;

[0228] Detect whether a preset training end condition is satisfied; if so, use the trained deep learning model as the image recognition model; if not, continue to train the deep learning model using the next piece of training data.

[0229] In some alternative embodiments, the processor is configured to detect whether the materials in the material placement area meet the material taking condition in the following manner:

[0230] Obtain the mass of the materials in the material placement area using a weighing sensor;

[0231] When the mass of the materials in the material placement area is not less than the preset mass threshold, determine that the materials in the material placement area meet the material taking condition;

[0232] When the mass of the materials in the material placement area is less than the preset mass threshold, determine that the materials in the material placement area do not meet the material taking condition.

[0233] In some alternative embodiments, the processor is further configured to implement the following steps before implementing step S1:

[0234] Based on the types of materials in the material placement area, obtain the picking strategy of the three-axis manipulator, and the picking strategy is used to indicate one or more of the following picking parameters: the picking type, quantity, shape, size, and material type of the picking mechanism;

[0235] Based on the picking strategy of the three-axis manipulator, determine one of the multiple picking components as the target picking component;

[0236] Connect the driving component of the three-axis manipulator to the target picking component;

[0237] Based on the target picking component, determine the driving strategy of the three-axis manipulator;

[0238] When the driving component of the three-axis manipulator uses a motor, the driving strategy is used to indicate one or more of the following driving parameters: rotational speed, torque, output power, and power factor;

[0239] When the driving component of the three-axis manipulator adopts a cylinder, the driving strategy is used to indicate one or more of the following driving parameters: output force, piston stroke, and piston movement speed.

[0240] In some alternative embodiments, the processor is configured to pick up one or more of the materials in the following manner:

[0241] Based on the target picking component, determine the target quantity of the material picked up each time;

[0242] Based on the movement strategy of the three-axis manipulator, use the driving component of the three-axis manipulator to drive the target picking component of the three-axis manipulator to move above the target quantity of materials and pick up the target quantity of materials.

[0243] In some alternative embodiments, the processor is further configured to implement the following steps:

[0244] Control the three-axis manipulator to place the picked-up material on the bearing surface of the AGV, so that the AGV transports the placed material to the target position.

[0245] See Figure 4 , Figure 4 shows a structural block diagram of a material picking device based on a three-axis manipulator provided by an embodiment of the present application.

[0246] The material picking device based on a three-axis manipulator may include, for example, at least one memory 210, at least one processor 220, and a bus 230 connecting different platform systems.

[0247] The memory 210 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 211 and / or a cache memory 212, and may further include a read-only memory (ROM) 213.

[0248] Among them, the memory 210 also stores a computer program, and the computer program can be executed by the processor 220, so that the processor 220 realizes the functions of any of the above methods. Its specific embodiments are the same as those described in the above method embodiments and the achieved technical effects, and some contents will not be repeated.

[0249] The memory 210 may further include a utility 214 having at least one program module 215. Such program modules 215 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. The implementation of a network environment may be included in each or some combination of these examples.

[0250] Correspondingly, the processor 220 can execute the above computer program and can also execute the utility 214.

[0251] The processor 220 may be implemented using one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0252] The bus 230 may be one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any bus structure with multiple bus structures.

[0253] The material taking device based on a three-axis manipulator may also communicate with one or more external devices 240 such as a keyboard, a pointing device, a Bluetooth device, etc., and may also communicate with one or more devices capable of interacting with the material taking device based on a three-axis manipulator, and / or communicate with any device (such as a router, a modem, etc.) that enables the material taking device based on a three-axis manipulator to communicate with one or more other computing devices. Such communication may be performed through the input / output interface 250. Moreover, the material taking device based on a three-axis manipulator may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 260. The network adapter 260 may communicate with other modules of the material taking device based on a three-axis manipulator through the bus 230. It should be understood that although not shown in the figure, other hardware and / or software modules may be used in conjunction with the material taking device based on a three-axis manipulator, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0254] (Medium Embodiment)

[0255] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of any of the above methods or the functions of any of the above devices. Its specific embodiments are consistent with the embodiments described in the above method embodiments and the achieved technical effects, and some content will not be elaborated again.

[0256] The computer-readable storage medium may be, for example, a program product.

[0257] SeeFigure 5 , Figure 5 shows a schematic structural diagram of a program product provided by an embodiment of the present application.

[0258] The program product is used to implement the steps of any of the above methods. The program product may be a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In the embodiments of the present application, the readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in conjunction with an instruction execution system, apparatus, or device. The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, but not be limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0259] The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium that can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the above. The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as C language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, executed as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (for example, by using an Internet service provider to connect through the Internet).

[0260] This application is described from the perspectives of purpose of use, effectiveness, progressiveness, and novelty, and has met the functional enhancement and usage requirements emphasized by the patent law. The above description and accompanying drawings of this application are only preferred embodiments of this application, and do not limit this application thereby. Therefore, all those that are similar or identical to the structure, device, features, etc. of this application, that is, all equivalent substitutions or modifications made according to the scope of the patent application of this application, shall fall within the scope of protection of the patent application of this application.

Claims

1. A material taking method based on a three-axis manipulator, characterized in that, For automatically picking up materials in the material placement area, the method includes: S1: Detect whether the materials in the material placement area meet the preset picking conditions; if so, execute S2; if not, execute S4; S2: Based on the height distribution information of the materials in the material placement area, obtain the movement strategy of the three-axis manipulator, where the movement strategy is used to indicate the X-axis displacement, Y-axis displacement, and Z-axis displacement of the target picking component of the three-axis manipulator in a preset rectangular coordinate system; S3: Based on the movement strategy of the three-axis manipulator, use the drive component of the three-axis manipulator to drive the target picking component of the three-axis manipulator to move above one or more of the materials and pick up one or more of the materials, and then execute S1; S4: After a preset time period, execute S1 again; The process of detecting whether the materials in the material placement area meet the picking conditions includes: Use an image acquisition device to obtain the image information of the material placement area; Input the image information of the material placement area into an image recognition model to obtain the quantity of the materials in the material placement area; When the quantity of the materials in the material placement area is not less than a preset quantity threshold, determine that the materials in the material placement area meet the picking conditions; When the quantity of the materials in the material placement area is less than the preset quantity threshold, determine that the materials in the material placement area do not meet the picking conditions; The step of using the drive component of the three-axis manipulator to drive the target picking component of the three-axis manipulator to move above one or more of the materials and pick up one or more of the materials based on the movement strategy of the three-axis manipulator includes: Based on the target picking component, determine the target quantity of the materials picked up each time.

2. The material taking method based on a three-axis manipulator according to claim 1, characterized in that, The step of inputting the image information of the material placement area into an image recognition model to obtain the quantity of the materials in the material placement area includes: Input the image information of the material placement area into the image recognition model to obtain the quantity and height distribution information of the materials in the material placement area.

3. The material taking method based on a three-axis manipulator according to claim 2, characterized in that, The training process of the image recognition model includes: Obtain a training set, where the training set includes a plurality of training data, and each training data includes the sample image information of a sample material placement area and the annotation data of the quantity and height distribution information of the materials in the sample material placement area; For each training data in the training set, perform the following processing: Input the sample image information in the training data into a preset deep learning model to obtain the predicted data of the quantity and height distribution information of the materials in the sample material placement area; Based on the predicted data and annotation data of the quantity and height distribution information of the materials in the sample material placement area, update the model parameters of the deep learning model; Detect whether the preset training end condition is met; if so, use the trained deep learning model as the image recognition model; if not, continue to train the deep learning model with the next training data.

4. The material taking method based on a three-axis manipulator according to claim 1, characterized in that, The process of detecting whether the materials in the material placement area meet the picking conditions includes: Obtain the mass of the material in the material placement area using a load cell; When the mass of the material in the material placement area is not less than a preset mass threshold, determine that the material in the material placement area meets the material taking condition; When the mass of the material in the material placement area is less than the preset mass threshold, determine that the material in the material placement area does not meet the material taking condition.

5. The material taking method based on a three-axis manipulator according to claim 1, characterized in that, Before the S1, the method further includes: Based on the type of the material in the material placement area, obtain the picking strategy of the three-axis manipulator, and the picking strategy is used to indicate one or more of the following picking parameters: the picking type, quantity, shape, size, and material type of the picking mechanism; Based on the picking strategy of the three-axis manipulator, determine one of the plurality of picking components as the target picking component; Connect the driving component of the three-axis manipulator to the target picking component; Based on the target picking component, determine the driving strategy of the three-axis manipulator; When the driving component of the three-axis manipulator uses a motor, the driving strategy is used to indicate one or more of the following driving parameters: rotational speed, torque, output power, and power factor; When the driving component of the three-axis manipulator uses a cylinder, the driving strategy is used to indicate one or more of the following driving parameters: output force, piston stroke, and piston movement speed.

6. The material taking method based on a three-axis manipulator according to claim 5, characterized in that, The step of driving the target picking component of the three-axis manipulator to move above one or more of the materials and pick up one or more of the materials by using the driving component of the three-axis manipulator based on the moving strategy of the three-axis manipulator further includes: Based on the moving strategy of the three-axis manipulator, use the driving component of the three-axis manipulator to drive the target picking component of the three-axis manipulator to move above the target quantity of materials and pick up the target quantity of materials.

7. The material taking method based on a three-axis manipulator according to claim 1, wherein The method further includes: Control the three-axis manipulator to place the picked material on the carrying surface of the AGV, so that the AGV transports the placed material to the target position.

8. A material taking device based on a three-axis manipulator, wherein For automatically picking up materials in the material placement area, the device includes a processor, and the processor is configured to implement the following steps: S1: Detect whether the materials in the material placement area meet the preset material taking conditions; if so, execute S2; if not, execute S4; S2: Based on the height distribution information of the materials in the material placement area, obtain the moving strategy of the three-axis manipulator, and the moving strategy is used to indicate the X-axis displacement, Y-axis displacement, and Z-axis displacement of the target picking component of the three-axis manipulator in a preset rectangular coordinate system; S3: Based on the moving strategy of the three-axis manipulator, use the driving component of the three-axis manipulator to drive the target picking component of the three-axis manipulator to move above one or more of the materials and pick up one or more of the materials, and execute S1; S4: After a preset time period, execute S1 again; The process of detecting whether the materials in the material placement area meet the material taking conditions includes: Obtain the image information of the material placement area using an image acquisition device; Input the image information of the material placement area into an image recognition model to obtain the quantity of materials in the material placement area; When the quantity of materials in the material placement area is not less than a preset quantity threshold, determine that the materials in the material placement area meet the material taking condition; When the quantity of materials in the material placement area is less than the preset quantity threshold, determine that the materials in the material placement area do not meet the material taking condition; The moving strategy based on the three-axis manipulator, using the driving component of the three-axis manipulator to drive the target picking component of the three-axis manipulator to move above one or more of the materials and pick up one or more of the materials, includes: Based on the target picking component, determine the target quantity of materials picked up each time.

9. A computer-readable storage medium, wherein The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images