Vision-based multi-sensor fusion sorting system and sorting method

Abstract

The invention belongs to the field of sorting transmission lines, and particularly relates to a multi-sensor fusion sorting system and method based on vision, and the sorting system comprises a master control device, a sorting device and a transmission device; the master control device is used for mutually transmitting information with the transmission device and the sorting device, and controlling the transmission device and / or sending an instruction to the sorting device after receiving the information; and a sensor is carried on the sorting device. Compared with the prior art, various sensors are used in a fused mode, the tail end executing mechanism flexibly adapting to three-dimensional objects in different shapes is selected to achieve stable grabbing of the sorting assembly line, and the matched AGV carrying the multiple sensors can achieve autonomous path planning; and through cooperative communication with the portal frame on the assembly line for visual identification, the materials on the assembly line can be monitored and classified in real time, so that efficient sorting is further achieved, and automatic grabbing and automatic classification of the materials in different shapes on the transmission line are achieved.

Description

Technical Field

[0001] The present invention belongs to the field of sorting transmission lines, and in particular relates to a picking system and a picking method based on vision-based multi-sensor fusion. Background Art

[0002] In the actual application of industrial manufacturing and scientific research, the grasping and sorting functions of robots are becoming more and more critical. As a device that simulates the basic movements of human hands, robotic grippers have been widely used in many fields such as aviation, aerospace, and manufacturing. These gripper devices are usually installed at the end of the robot arm, so they are also called end effectors. Production line picking robots are mainly used in factory production lines, logistics warehouse sorting workshops, computer rooms and other environments where sorting and sorting work is required. They can greatly save the demand for manpower for simple and repetitive sorting work, and also save the cost of sorting and sorting for enterprises.

[0003] Among the existing technical solutions, there are some indoor inspection robots that can realize path planning, image collection and recognition, real-time monitoring of assembly line operations, etc. There are gripping and picking robots dedicated to assembly lines, and there are also sorting and picking robots for logistics warehouses. Most of them use laser radar or high-definition cameras for target recognition, but when faced with different product shapes in assembly lines or logistics sorting scenarios, it is difficult to have a suitable mechanical gripper to achieve the purpose of gripping. Although the metamorphic mechanism and underactuated design improve the adaptability of the robot, the flexibility of the robot still needs to be improved when faced with rapidly changing production needs and diverse operating environments. For some customized picking robots, their manufacturing costs are too high and professional technicians are required to operate and maintain high-end picking robots. In this context, we are required to propose new end effectors and picking robots that can adapt to products of different shapes, and propose matching picking systems and picking methods that can efficiently realize sorting tasks.

[0004] Patent CN109550697A discloses an AGV intelligent sorting system, including a general control device, a barcode scanning device, an AGV sorting device and a positioning device; the barcode scanning device, the AGV sorting device and the positioning device are all wirelessly connected to the general control device; the barcode scanning device obtains the identification information of the object and transmits it to the general control device, the general control device selects the optimal path based on the identification information and feeds it back to the AGV sorting device, and the AGV sorting device drives to the target destination along the optimal path; the positioning device identifies and feeds back the position coordinates of the AGV sorting device to the general control device. However, this device requires the goods to be transported to the barcode scanning device for scanning and identification, but in this process, the barcode needs to face up, and then transmit the information to the AGV trolley for sorting, and this device does not control or improve the terminal actuator.

[0005] Patent CN117816579A discloses a logistics sorting device based on machine vision, and the technical solution of the sorting device includes: a conveyor belt for conveying logistics packages, two industrial robotic arms set up on the left and right sides of the conveyor belt, and an array camera set on the upper surface of the conveyor belt. In addition, in this technical solution, an industrial robotic arm and an array-arranged industrial camera are combined to obtain image data of objects from multiple angles using industrial cameras. On the basis of the traditional image-controlled robotic arm to perform sorting actions, a three-dimensional model of the object is constructed using multi-angle image data, and the three-dimensional model is used to assist the robotic arm in setting the grabbing position during the sorting process. However, in this patent design, the industrial robotic arm is usually installed in a fixed position and can only operate on a specific area. It is only suitable for tasks with high repetitiveness and fixed positions, but has low flexibility. At the same time, due to its fixedness, although the time for moving and repositioning the equipment is reduced, it can improve work efficiency for a single or a few task points. However, if a wider area is to be covered, multiple robotic arms are required, which greatly increases the cost of the production line. Summary of the invention

[0006] The purpose of the present invention is to provide a picking system and a picking method based on multi-sensor fusion based on vision in order to overcome the shortcomings of the above-mentioned prior art, so as to solve the problem that the transmission line in the prior art cannot grasp and automatically classify materials of different shapes. By fusing multiple sensors and selecting an end-effector that can flexibly adapt to three-dimensional objects of different shapes to achieve stable grasping of the sorting line, the matching AGV car equipped with multiple sensors can realize autonomous path planning, and communicate with the gantry on the assembly line for visual recognition, so as to monitor and classify the materials on the assembly line in real time, thereby further realizing efficient sorting, and realizing automatic grasping and automatic classification of materials of different shapes on the transmission line.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] One of the technical solutions of the present invention is to disclose a picking system based on vision-based multi-sensor fusion, comprising a general control device, a picking device and a transmission device; the picking device and the transmission device are both electrically connected to the general control device;

[0009] The general control device is used to transmit information to and from the transmission device and the picking device, and after receiving the information, controls the transmission device and / or issues instructions to the picking device;

[0010] The transmission device includes a transmission line, a gantry and a gantry integrated module; the gantry is arranged at the starting end of the transmission line; the gantry integrated module is assembled at the upper end of the gantry, includes a visual recognition module and a communication module, and is electrically connected to the general control device;

[0011] The picking device includes a mechanical arm, a control cabinet, an AGV trolley and an end-actuator, wherein the end-actuator is arranged at the end of the mechanical arm, the base of the mechanical arm passes through the top of the control cabinet and is connected to the control cabinet, and the control cabinet is arranged on the AGV trolley; the control cabinet is electrically connected to the general control device, the mechanical arm, the AGV trolley and the end-actuator, and is used to control the movement of the mechanical arm, the AGV trolley and the end-actuator;

[0012] The picking device is equipped with sensors, including environmental perception sensors carried on the AGV vehicle.

[0013] Furthermore, the general control device has an information processing module, a communication module and a control module which are electrically connected to each other, and is used to transmit information to and from the transmission device and the picking device, and after receiving the information, controls the transmission device and / or issues instructions to the picking device.

[0014] Furthermore, the control cabinet contains a variety of hardware devices, including servo drives, motion controllers, embedded control systems, sensor interface modules, safety protection modules, and a display interface for human-computer interaction; in addition, the control cabinet is also equipped with a heat dissipation and cooling system to ensure that the system maintains a stable temperature during long-term operation to avoid overheating.

[0015] Furthermore, the robotic arm has a built-in rotating motor for driving the retractable rotating shaft.

[0016] Furthermore, the robotic arm is equipped with a force sensor for detecting the interaction force between the picking device and the environment; the end actuator is equipped with a visual sensor, a force sensor, a speed sensor and a displacement sensor for visual identification and measurement of the relative position, moving speed and applied force of the end actuator; each sensor is electrically connected to the overall control device.

[0017] Furthermore, the displacement sensor is used to measure the position change of the end actuator relative to the reference position; the velocity sensor is used to measure the velocity of the end actuator; and the force sensor is used to measure the force applied by the end actuator.

[0018] Furthermore, the end effector is equipped with a 3D TOF vision sensor, which can accurately measure the position, direction and size of an object, is suitable for identifying objects of different shapes, and can calculate and transmit information such as the center of gravity, size and position of the object to the controller. The control cabinet is controlled according to the information fed back by the 3D TOF sensor.

[0019] Furthermore, the robotic arm is equipped with a six-dimensional force sensor, which is installed between the end actuator of the robotic arm and the robotic arm, and is used to detect the multi-dimensional interaction force / torque between the robot and the environment during operation, and to feed back the interaction force / torque to the robot force control system.

[0020] Furthermore, the robotic arm is a six-degree-of-freedom robotic arm, and the end actuator is a metamorphic robotic claw.

[0021] The six-degree-of-freedom robotic arm adopted in the present invention is more flexible in design and has a wider working range. Since it has six joints that can move independently, it can easily cover a larger action space, while also ensuring that the picking robot itself does not collide with the production transportation line as much as possible; and the metamorphic robotic gripper can adjust the separation and approach of the gripper to adapt to the clamping of cubic objects and sheet objects. When clamping cubic objects, the freedom of the gripper can match the shapes of different objects to achieve changes in the freedom during the clamping process. In addition, the flexible filling material between the grippers can ensure the stability of the clamping.

[0022] Furthermore, the environmental perception sensors carried on the AGV include laser radar and visual cameras. The sensors are used to measure the distance of surrounding objects and help the vehicle build an accurate environmental model by identifying marks in the map or features of objects in the environment, ensuring that it can complete the task efficiently and safely. Through the collaborative work of laser radar and visual cameras, the AGV can adjust its route in real time in a dynamic environment and achieve intelligent autonomous navigation.

[0023] Furthermore, the AGV is also equipped with a Hall sensor for determining the movement direction and speed of the AGV smart car.

[0024] Furthermore, the AGV is equipped with a drive system including a drive wheel set, a drive motor, etc., including omnidirectional wheels responsible for movement and two steering wheels responsible for steering, allowing the AGV to move freely in a narrow space and achieve complex movements such as lateral and diagonal movements. The combination of omnidirectional wheels and steering wheels enables the AGV to achieve complex movement modes such as lateral and diagonal movements, greatly improving its maneuverability and flexibility in narrow or complex environments.

[0025] Furthermore, the transmission device also includes an emergency start-stop operating lever, which is connected to the transmission line and is used to control the start and stop of the transmission line.

[0026] The second technical solution of the present invention is to disclose a picking method of the above-mentioned vision-based multi-sensor fusion picking system, comprising the following steps:

[0027] Step 1: The initial position of the picking device is a stop and rest area, which is marked as the Home area, for the picking device to wait or to transfer tasks between different transmission lines;

[0028] Step 2: The conveyor line is started and begins to run, and the gantry integrated module on the gantry visually identifies the material passing through the starting end of the conveyor line;

[0029] Step 3: If the gantry integrated module recognizes that there is material passing through the starting end of the transmission line, the recognized material information is transmitted to the general control device; if the gantry integrated module does not recognize that there is material passing through the starting end of the transmission line, no signal is sent and the recognition is continued;

[0030] Step 4: After receiving the information, the main control device instructs the transmission line to stop or slow down, and transmits the instruction and position information to the picking device to assign the picking task;

[0031] Step 5: After receiving the picking task, the picking device starts from the Home area. The AGV uses the environmental perception sensor and the control cabinet to perceive the environment, plan the path, and drive to the designated location.

[0032] Step 6: After the picking device moves to the designated position near the transmission line, the control cabinet controls the robotic arm to drive the end actuator to clamp the material on the transmission line. At this time, the visual recognition module on the gantry further determines whether the target material is clamped;

[0033] Step 7: The picking device keeps the material in the clamped state, and the AGV re-plans the route to the downstream position of the material packaging and puts the material down, then returns to the designated position near the transmission line to wait, and the transmission device resumes its original speed;

[0034] Step 8: Repeat steps 3-7 until all picking tasks are completed, the transmission line stops running, and the picking device returns to the Home area.

[0035] Furthermore, the material information recognized by the gantry integrated module includes at least one of the material's size, shape, barcode and label.

[0036] Furthermore, the gantry integrated module uses a high-definition camera to capture images of passing materials, collects images through an optical imaging system, and converts optical images into digital images. The captured images are preprocessed, including filtering, binarization, morphological processing, and edge detection, to obtain high-quality material images, so that the material features and background are separated as much as possible to facilitate the subsequent feature extraction and recognition. Image processing technology is used to automatically identify and measure the size of the material. This process relies on a computer vision library (such as OpenCV) to accurately identify the edges and features of the material in the image, thereby calculating the size of the material.

[0037] Furthermore, for barcode or label recognition, the gantry integrated module uses a barcode scanner or camera to capture the image of the barcode, and then pre-processes it through image processing technology to locate the barcode, identify bars and spaces, decode characters, and verify the results.

[0038] Furthermore, the control cabinet of the picking device converts the measurement values ​​of different sensor coordinate systems into the same coordinate system, and performs global path planning in the control cabinet according to global path planning and local path planning, calculates the optimal route, and adds obstacle avoidance in the actual navigation process. The current optimal path is selected to match the global optimal path as much as possible, so that the AGV car can travel from the optimal path to the designated position.

[0039] Furthermore, the visual recognition module on the gantry identifies and compares the actual position of the material with the expected position to determine whether it is accurately clamped; the force sensor or pressure sensor on the end actuator is used to monitor the clamping force of the clamp in real time to ensure the stability of the clamped object.

[0040] Furthermore, the picking device achieves accurate perception of the environment through the synergy of lidar and visual cameras. The lidar provides distance and angle information, while the visual camera provides detailed features such as color, texture and shape. The combination of the two can improve the recognition and positioning accuracy of material locations.

[0041] Furthermore, if an abnormality occurs in the transmission device or the picking device, the main control device controls the transmission device and the picking device to stop running after receiving the information, and transmits the status information to the technicians.

[0042] Furthermore, the picking device automatically plans a path to return to the Home area, waiting for the next task to be dispatched, and charging or performing a system self-check during the waiting process.

[0043] Furthermore, when the picking device is unable to complete clamping during work, the control cabinet transmits the information to the main control device, and the main control device dispatches another picking device to take over the work. At the same time, the picking device that cannot operate automatically goes to the Home area and is charged or repaired by technicians.

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] 1. Innovative synergy of the whole system: Compared with the current traditional single fixed robot arm sorting, the present invention coordinates the sorting robot arm and the end actuator with the AGV trolley, so that the sorting task can be flexibly implemented. The present invention emphasizes the coordinated communication between the AGV trolley and the gantry on the assembly line, which improves the collaborative efficiency of the entire picking system. The AGV trolley communicates with the gantry and uses visual recognition technology to monitor and classify materials on the assembly line in real time. This communication and collaboration mode enhances the dynamics of the robot system, can adapt to the changing production rhythm, and achieve efficient material sorting.

[0046] 2. Intelligent perception and path planning through multi-sensor fusion: Compared with the traditional AGV system that relies on a single sensor (such as lidar or vision), the AGV cart of the sorting device in the present invention achieves more accurate and robust environmental perception and path planning through the multi-sensor fusion of lidar and visual camera. This fusion can not only improve the ranging and object recognition accuracy of the AGV, but also optimize the obstacle avoidance and map building capabilities through the complementary characteristics of vision and radar.

[0047] 3. High flexibility and high-precision control: The six-degree-of-freedom robotic arm used in the present invention has high flexibility and can realize a retractable rotating axis through a built-in rotating motor. Moreover, through the six-degree-of-freedom multi-joint design, the robot can accurately grasp and operate objects in complex working environments. Combined with the mobility of AGV, such flexibility greatly improves the operating range and operating efficiency. In addition, a metamorphic robotic gripper is introduced as the end effector in the execution of picking tasks. The metamorphic robotic gripper can be flexibly adjusted according to the shape, size, material and other characteristics of the target object, adapting to various three-dimensional objects and achieving stable and efficient grasping.

[0048] 4. Modular design and system integration: The entire system adopts a modular design, and the robot arm, AGV car and control cabinet are independently designed and integrated together. Each part has a clear functional division. The control cabinet is equipped with efficient hardware configurations such as servo drives, motion controllers, and embedded systems, which improves the scalability and maintainability of the system. In addition, the system achieves a more friendly operation and monitoring experience through the human-computer interaction interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a structural schematic diagram of the intelligent picking system of the present invention.

[0050] Figure 2 It is a side structural schematic diagram of the intelligent picking system of the present invention.

[0051] Figure 3 This is a structural diagram of the metamorphic robot claw of the end-effector in the present invention.

[0052] Figure 4 Workflow diagram of the present invention when performing intelligent picking.

[0053] In the figure: 1-emergency start-stop operating lever; 2-transmission line; 3-control cabinet; 4-mechanical arm; 5-AGV trolley; 6-end actuator; 7-gantry; 8-gantry integrated module; 9-knuckle rod I; 10-knuckle rod II; 11-knuckle rod III; 12-knuckle rod IV; 13-knuckle rod V; 14-knuckle rod VI; 15-knuckle rod VII; 16-limit fixing plate; 17-knuckle rod VIII; 18-spiral lifting mechanism; 19-motor driven slide rail; 20-bearing bottom plate; 21-lifting plate. DETAILED DESCRIPTION

[0054] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0055] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0056] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0057] In the following implementation manners or examples, unless otherwise specified, functional components or structures are conventional components or conventional structures used in the art to achieve corresponding functions.

[0058] Example 1

[0059] like Figure 1-Figure 2As shown, this embodiment provides a picking system based on vision-based multi-sensor fusion, including a general control device, a picking device and a transmission device; the picking device and the transmission device are both electrically connected to the general control device;

[0060] The general control device is used to transmit information to and from the transmission device and the picking device, and after receiving the information, controls the transmission device or issues instructions to the picking device;

[0061] The transmission device includes an emergency start-stop operating lever 1, a transmission line 2, a gantry 7 and a gantry integrated module 8; the gantry 7 is arranged at the starting end of the transmission line 2; the gantry integrated module 8 is assembled at the upper end of the gantry 7, includes a visual recognition module and a communication module, and is electrically connected to the general control device; the emergency start-stop operating lever 1 is connected to the transmission line 2, and is used to control the start and stop of the transmission line 2;

[0062] The picking device includes a mechanical arm 4, a control cabinet 3, an AGV trolley 5 and an end actuator 6. The end actuator 6 is arranged at the end of the mechanical arm 4. The base of the mechanical arm 4 passes through the top of the control cabinet 3 and is connected to the control cabinet 3. The control cabinet 3 is arranged on the AGV trolley 5. The control cabinet 3 is electrically connected to the general control device, the mechanical arm 4, the AGV trolley 5 and the end actuator 6, and is used to control the movement of the mechanical arm 4, the AGV trolley 5 and the end actuator 6.

[0063] The picking device is equipped with sensors, including a Hall sensor, a laser radar and a visual camera on the AGV 5, a six-dimensional force sensor on the robotic arm 4, and a visual sensor, a displacement sensor, a speed sensor and a force sensor on the end actuator 6.

[0064] The general control device has built-in information processing modules, communication modules and control modules which are electrically connected to each other, and is used to transmit information to and from the transmission device and the picking device, and to control the transmission device and / or issue instructions to the picking device after receiving the information.

[0065] The control cabinet 3 contains a variety of hardware devices, including servo drives, motion controllers, embedded control systems, sensor interface modules, safety protection modules, and a display interface for human-computer interaction; in addition, the control cabinet 3 is also equipped with a heat dissipation and cooling system to ensure that the system maintains a stable temperature during long-term operation to avoid overheating.

[0066] The mechanical arm 4 has a built-in rotary motor for driving a retractable rotary shaft.

[0067] The displacement sensor is used to measure the position change of the end actuator 6 relative to the reference position; the speed sensor is used to measure the speed of the end actuator 6; and the force sensor is used to measure the force applied by the end actuator 6.

[0068] The end effector 6 is equipped with a 3D TOF visual sensor, which can accurately measure the position, direction and size of an object, is suitable for identifying objects of different shapes, and can calculate and transmit information such as the center of gravity, size and position of the object to the controller. The control cabinet 3 is controlled according to the information fed back by the 3D TOF sensor.

[0069] The robot arm 4 is equipped with a six-dimensional force sensor, which is installed between the robot arm end actuator 6 and the robot arm 4 to detect the multi-dimensional interaction force / torque between the robot and the environment during operation, and to feed back the interaction force / torque to the robot force control system.

[0070] The robot arm 4 is a six-degree-of-freedom robot arm, and the end actuator 6 is a metamorphic robot gripper.

[0071] See also Figure 3 As shown, all mechanical structures of the metamorphic robot gripper are arranged on a bearing base plate 20, and the bearing base plate 20 is fixedly connected to the six-degree-of-freedom robot arm. A single finger of the metamorphic robot gripper is composed of different finger joint mechanisms. There are a total of 4 such finger structures in the entire metamorphic robot gripper, with two fingers arranged on each side, and a corresponding limit fixing plate 16 is arranged to cooperate with the motor-driven slide rail 19 below to open and close the fingers. In order to ensure the reasonable change of the degree of freedom of the grasping mechanism, tension springs are arranged between the finger joint rod IV13 and the finger joint rod V12, and between the finger joint rod VI14 and the finger joint rod VII15. The two fingers on the same side of the metamorphic robot gripper are controlled to approach or separate by the motor-driven slide rail 19 below, and the opening and closing of the fingers are controlled to adapt to the clamping of block objects and sheet objects of different shapes. The principle of bending fingers to grasp objects is that the spiral lifting mechanism 18 located in the middle of the grasping mechanism drives the lifting plate 21 to lift. When the lifting plate 21 moves up and down, it drives the knuckle rods Ⅰ9 of the four fingers to bend and transmits the torque to different rods to achieve bending and grasping of the fingers.

[0072] Please see again Figure 3As shown, the specific process of the end actuator 6 accurately clamping the target material is: the control cabinet 3 controls the robot arm 4 and the end actuator 6 to hover above the material to be grasped, and then controls the robot arm 4 to descend. After the four fingers of the end actuator 6 surround the material, the motor drives the slide rail 19 to adjust the distance between the knuckles on both sides to a suitable position, and then the control cabinet 3 controls the motor to drive the spiral lifting mechanism 18 below to push the lifting plate 21 to lift upward. Under the action of the limit fixing plate 16, the knuckle rod I9 is ​​transmitted upward along the columnar rod and the torque is transmitted to the knuckle rod II9, the knuckle rod III11, the knuckle rod IV12, and the knuckle rod V15 in sequence, and then the torque is transmitted to the knuckle rod VI14, the knuckle rod VII15 and the knuckle rod V16. Rod VIII 17 is used to achieve grasping. At the beginning, the four knuckles do not touch the clamped object, so the freedom degree of the mechanism does not change. When one knuckle stops bending after contacting the object, the number of degrees of freedom becomes 1. Thereafter, the movement form is fixed, that is, the finger can only be driven to bend by the driving mechanism. When the second knuckle also contacts the object and stops bending, the driving force brought by the driving mechanism can only make the third knuckle bend and envelope the grasped object. Since the freedom degree of the main structure is greater than the number of driving machines, consider adding a spring in the middle of the knuckles on the same side as traction, so that in the initial position and when the first knuckle touches the clamped object, the second and third knuckles can maintain stability until the driving element continues to lift so that the clamping mechanism continues to bend, and finally completely envelopes or clamps the clamped object.

[0073] The six-degree-of-freedom robotic arm adopted in the present invention is more flexible in design and has a wider working range. Since it has six joints that can move independently, it can easily cover a larger action space, while also ensuring that the picking robot itself does not collide with the production transportation line as much as possible; and the metamorphic robotic gripper can adjust the separation and approach of the gripper to adapt to the clamping of cubic objects and sheet objects. When clamping cubic objects, the freedom of the gripper can match the shapes of different objects to achieve changes in the freedom during the clamping process. In addition, the flexible filling material between the grippers can ensure the stability of the clamping.

[0074] The laser radar and visual camera carried on the AGV trolley 5 are used to measure distance and identify map marks, objects or environmental features; the Hall sensor carried on the AGV trolley 5 is used to determine the movement direction and speed of the AGV trolley 5.

[0075] The AGV trolley 5 is equipped with a driving system including a driving wheel set, a driving motor, etc., including omnidirectional wheels responsible for movement and two steering wheels responsible for steering, allowing the AGV trolley 5 to move freely in a narrow space and realize complex movements such as lateral and diagonal movements. The combination of omnidirectional wheels and steering wheels enables the AGV trolley 5 to realize complex movement modes such as lateral and diagonal movements, greatly improving its maneuverability and flexibility in narrow or complex environments.

[0076] Example 2

[0077] like Figure 4 As shown, this embodiment provides a picking method of the picking system based on vision-based multi-sensor fusion as described above, comprising the following steps:

[0078] Step 1: The initial position of the picking device is a stop and rest area, which is marked as the Home area, for the picking device to wait or to transfer tasks between different transmission lines;

[0079] Step 2: The transmission line 2 is started and begins to run, and the gantry integrated module 8 on the gantry 7 visually identifies the material passing through the starting end of the transmission line 2;

[0080] Step 3: If the gantry integrated module 8 recognizes that there is material passing through the starting end of the transmission line 2, the recognized material information is transmitted to the general control device; if the gantry integrated module 8 does not recognize that there is material passing through the starting end of the transmission line 2, no signal is sent and the recognition is continued;

[0081] Step 4: After receiving the information, the main control device instructs the transmission line 2 to stop or slow down, and transmits the instruction and position information to the picking device to assign the picking task;

[0082] Step 5: After receiving the picking task, the picking device starts from the Home area. The AGV 5 senses the environment and plans the path through the environment sensing sensor and the control cabinet 3 and drives to the designated location.

[0083] Step 6: After the picking device moves to the designated position near the transmission line 2, the control cabinet 3 controls the robot arm 4 to drive the end actuator 6 to clamp the material on the transmission line 2. At this time, the visual recognition module on the gantry 7 further determines whether the target material is clamped;

[0084] Step 7: The picking device keeps the material in the clamped state, and the AGV 5 re-plans the route to the downstream position of the material packaging and puts the material down, then returns to the designated position near the transmission line 2 to wait, and the transmission device resumes its original speed;

[0085] Step 8: Repeat steps 3-7 until all picking tasks are completed, then the conveyor line 2 stops running and the picking device returns to the Home area.

[0086] The material information identified by the gantry integration module 8 includes at least one of the size, shape, barcode and label of the passing material.

[0087] The gantry integrated module 8 uses a high-definition camera to collect images of the passing materials, collects images through an optical imaging system, and converts the optical images into digital images. The collected images are pre-processed, including filtering, binarization, morphological processing, and edge detection, etc., to obtain high-quality material images, so that the features and background of the materials are separated as much as possible, which is convenient for the extraction and recognition of later features. The image processing technology is used to automatically identify and measure the size of the materials. This process relies on a computer vision library (such as OpenCV) to accurately identify the edges and features of the materials in the image, thereby calculating the size of the materials. For the recognition of barcodes or labels, the gantry integrated module 8 uses a barcode scanner or camera to capture the image of the barcode, and then pre-processes it through image processing technology, locates the barcode, identifies bars and spaces, decodes characters, and verifies the results.

[0088] The control cabinet 3 of the picking device converts the measurement values ​​of different sensor coordinate systems into the same coordinate system. According to the global path planning and local path planning, the global path planning is performed in the control cabinet 3 to calculate the optimal route. At the same time, obstacle avoidance is added in the actual navigation process, and the current optimal path is selected to conform to the global optimal path as much as possible, so that the AGV car 5 can travel from the optimal path to the specified position.

[0089] The visual recognition module on the gantry 7 identifies and compares the actual position of the material with the expected position to determine whether it is accurately clamped; the force sensor or pressure sensor on the end actuator 6 monitors the clamping force in real time to ensure the stability of the clamped object.

[0090] The picking device achieves accurate perception of the environment through the synergy of lidar and visual cameras. Lidar provides distance and angle information, while the visual camera provides detailed features such as color, texture and shape. The combination of the two can improve the recognition and positioning accuracy of material locations.

[0091] If an abnormality occurs in the transmission device or the picking device, the main control device controls the transmission device and the picking device to stop running after receiving the information, and transmits the status information to the technicians.

[0092] The picking device automatically plans a path to return to the Home area and waits for the next task to be dispatched. During the waiting process, it charges or performs a system self-check.

[0093] When the picking device cannot complete the clamping during work, the control cabinet 3 transmits the information to the main control device, and the main control device dispatches another picking device to take over the work. At the same time, the picking device that cannot operate automatically goes to the Home area and is charged or repaired by technicians.

[0094] Example 3

[0095] Please see again Figure 4As shown, the picking method of the picking system based on vision multi-sensor fusion in the present invention includes the following steps:

[0096] Step 1: The picking device is initially arranged in a stagnant rest area, which is marked as the "Home area". The main function of the Home area is for the picking device to wait or rest after completing a task. It can also be used for the picking device to perform task handover between multiple transmission lines. When the picking device is in the standby state, its control cabinet 3 maintains communication with the main control device to ensure that it can receive new picking tasks at any time. Setting up the Home area can effectively reduce the idling time of the picking device and improve the overall work efficiency. In addition, the Home area can also be used as a safe area for the picking device in case of system failure or maintenance;

[0097] Step 2: When materials such as parts, products or packages are transported to the transmission line 2, the visual recognition module in the gantry integrated module 8 located on the gantry 7 starts working. The module identifies the passing materials through technical means such as high-definition cameras and image processing systems, and records relevant information such as size, shape, barcode or label. This information is transmitted to the main control device through the communication module, and then the information and picking instructions are distributed to the transmission device and the picking device. The main control device controls the start and stop of the transmission device according to the received information to ensure that no unnecessary delays or failures will occur during the task. At the same time, the sorting task will also be automatically assigned to the nearest standby picking device;

[0098] Step 3: When the transmission line 2 is in normal working condition but no material passes through, the system will re-enter the standby state and periodically repeat step 2 to detect whether new materials have arrived. If the transmission line 2 or other equipment has an abnormality, such as mechanical failure or sensor failure, the communication module will immediately transmit the status information to the technician responsible for equipment safety in the system so that maintenance or repair can be carried out as soon as possible. This mechanism can effectively reduce production interruptions caused by equipment failures and ensure the stability and efficiency of the system;

[0099] Step 4: When the parts, products or packages are confirmed by the visual recognition system, the control system of the conveyor line 2 will make corresponding adjustments as needed, such as stopping the conveyor line 2 or adjusting the speed of the conveyor line 2. The master control device will transmit the precise position of the material on the conveyor line 2 to the picking device to ensure that the picking device can accurately locate the position of the material, thereby avoiding clamping errors and improving the accuracy of the operation and the efficiency of task completion;

[0100] Step 5: After receiving the dispatch signal, the picking device starts from the stagnant area. The AGV trolley 5 of the picking device has autonomous navigation and path planning functions. Relying on the built-in laser radar and visual sensor, the picking device can scan the working area in real time, plan an optimal path, avoid obstacles, and reach the designated picking position safely and efficiently in a complex or narrow working environment;

[0101] Step 6: When the picking device reaches the designated position, the robot arm 4 starts to work and accurately clamps the target material through the end effector 6. The recognition module on the gantry 7 will further confirm whether the picked object is consistent with the assigned task and ensure that the object is firmly clamped. If an identification error or technical failure occurs during this process, the system will automatically return to step S3 to notify the technician to check or repair. The multi-level inspection mechanism in this process greatly reduces the risk of errors in material operation;

[0102] Step 7: When the material is successfully picked up, the picking device will deliver the material to the preset packaging area according to the task requirements. These areas may be packaging boxes, secondary transmission lines or specific storage units. The picking device can accurately reach the target location of the material through autonomous navigation and place the material at the designated location as needed. Through the synergy of LiDAR and visual cameras, accurate perception of the environment is achieved. LiDAR provides distance and angle information, while visual cameras provide detailed features such as color, texture and shape. The combination of the two can improve the recognition and positioning accuracy of the material position. The coordinated use of sensors in this process can ensure that each task is executed accurately and efficiently to avoid misdelivery or omission.

[0103] Step 8: After completing the sorting task, the picking device will automatically return to the Home area and wait for the next task to be scheduled. The picking device can be charged or perform a system self-check in the Home area to ensure that it is in the best operating state for the next task. If there is no new task on the transmission line 2 in a short time, the picking device will remain in a low-power standby state to save energy.

[0104] Step 9: When all dispatching tasks are completed, the picking device will return to the Home area, and the overall control system and transmission line 2 will enter the maintenance phase according to the requirements of production scheduling. This phase includes comprehensive inspection, maintenance and care of transmission line 2 and the picking device to ensure that all equipment is in good operating condition before the next round of production. In addition, the control center of the entire system will store and analyze the operation data of the day to optimize future scheduling and task allocation strategies.

[0105] When the picking device encounters low power during work, the control cabinet 3 of the picking device uploads its own status and request information for battery replacement to the main control device through the communication module. The main control device will dispatch a picking device that is closer to take over the work. At the same time, the low-power picking device automatically goes to the Home area, and the central manager will perform battery replacement or charging operations. After completing the entire battery replacement or charging operation, the picking device continues to communicate with the main control device and wait for the deployment task.

[0106] The above description of the embodiments is to facilitate the understanding and use of the invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the present invention should be within the scope of protection of the present invention.

Claims

1. A picking system based on vision and multi-sensor fusion, characterized in that: It includes a general control device, a picking device and a transmission device; the picking device and the transmission device are both electrically connected to the general control device; The general control device is used to transmit information to and from the transmission device and the picking device, and after receiving the information, controls the transmission device and / or issues instructions to the picking device; The transmission device comprises a transmission line (2), a gantry (7) and a gantry integrated module (8); the gantry (7) is arranged at the starting end of the transmission line (2); the gantry integrated module (8) is assembled at the upper end of the gantry (7), comprises a visual recognition module and a communication module, and is electrically connected to the general control device; The picking device comprises a mechanical arm (4), a control cabinet (3), an AGV trolley (5) and an end actuator (6), wherein the end actuator (6) is arranged at the end of the mechanical arm (4), the base of the mechanical arm (4) passes through the top of the control cabinet (3) and is connected to the control cabinet (3), and the control cabinet (3) is arranged on the AGV trolley (5); the control cabinet (3) is electrically connected to the general control device, the mechanical arm (4), the AGV trolley (5) and the end actuator (6), and is used to control the movement of the mechanical arm (4), the AGV trolley (5) and the end actuator (6); The picking device is equipped with sensors, including an environmental perception sensor carried by the AGV vehicle (5).

2. The picking system based on vision and multi-sensor fusion according to claim 1 is characterized in that: The mechanical arm (4) is equipped with a force sensor for detecting the interaction force between the picking device and the environment; the end actuator (6) is equipped with a visual sensor, a force sensor, a speed sensor and a displacement sensor for visual recognition and measurement of the relative position, moving speed and applied force of the end actuator (6); Each sensor is electrically connected to the overall control device.

3. The picking system based on vision and multi-sensor fusion according to claim 1 is characterized in that: The robotic arm (4) is a six-degree-of-freedom robotic arm, and the end actuator (6) is a metamorphic robotic gripper.

4. The picking system based on vision and multi-sensor fusion according to claim 1 is characterized in that: The environmental perception sensors carried on the AGV vehicle (5) include a laser radar and a visual camera.

5. The picking system based on vision and multi-sensor fusion according to claim 1 is characterized in that: The transmission device further comprises an emergency start / stop operating rod (1), which is connected to the transmission line (2) and is used to control the start / stop of the transmission line (2).

6. A picking method of a picking system based on vision and multi-sensor fusion as claimed in claim 1, characterized in that: The steps include: Step 1: The initial position of the picking device is a stop and rest area, which is marked as the Home area, for the picking device to wait or to transfer tasks between different transmission lines; Step 2: The transmission line (2) is started and begins to operate, and the gantry integrated module (8) on the gantry (7) visually identifies the material passing through the starting end of the transmission line (2); Step 3: If the gantry integrated module (8) recognizes that there is material passing through the starting end of the transmission line (2), the recognized material information is transmitted to the general control device; if the gantry integrated module (8) does not recognize that there is material passing through the starting end of the transmission line (2), no signal is sent and the recognition is continued; Step 4: After receiving the information, the main control device instructs the transmission line (2) to stop or slow down, and at the same time transmits the instruction and position information to the picking device to assign the picking task; Step 5: After receiving the picking task, the picking device starts from the Home area. The AGV car (5) senses the environment and plans the path through the environment perception sensor and the control cabinet (3) and drives to the designated location; Step 6: After the picking device moves to the designated position near the transmission line (2), the control cabinet (3) controls the mechanical arm (4) to drive the end actuator (6) to clamp the material on the transmission line. At this time, the visual recognition module on the gantry (7) further determines whether the target material is clamped; Step 7: The picking device keeps the material in the clamped state, and the AGV (5) re-plans the route to the downstream position of the material and puts the material down, then returns to the designated position near the transmission line (2) to wait, and the transmission device resumes its original speed; Step 8: Repeat steps 3-7 until all picking tasks are completed, the conveyor line (2) stops running, and the picking device returns to the Home area.

7. The picking method of a picking system based on vision and multi-sensor fusion according to claim 6, characterized in that: The material information identified by the gantry integrated module (8) includes at least one of the size, shape, barcode and label of the material.

8. The picking method of a picking system based on vision and multi-sensor fusion according to claim 6, characterized in that: If an abnormality occurs in the transmission device or the picking device, the main control device controls the transmission device and the picking device to stop running after receiving the information, and transmits the status information to the technicians.

9. The picking method of a picking system based on vision and multi-sensor fusion according to claim 6, characterized in that: The picking device automatically plans a path to return to the Home area and waits for the next task to be dispatched. During the waiting process, it charges or performs a system self-check.

10. The picking method of a picking system based on vision and multi-sensor fusion according to claim 6, characterized in that: When the picking device is unable to complete the clamping during operation, the control cabinet (3) transmits the information to the main control device, and the main control device dispatches another picking device to take over the work. At the same time, the picking device that cannot operate automatically goes to the Home area and is charged or repaired by technicians.

Images