Machine vision-based automated grasping system, method, apparatus, and packaged confectionary

Abstract

The present application belongs to the technical field of automatic production of pastries, and particularly relates to an automatic grabbing system, method and device based on machine vision and packaged pastries. The system comprises a sorting conveyor belt, a packaging conveyor belt, a first visual detection module, a mechanical hand module and a control module. The packaging conveyor belt is arranged on both sides of the sorting conveyor belt. The first visual detection module is arranged above the sorting conveyor belt and used for acquiring image data. The mechanical hand module is arranged above the sorting conveyor belt. The control module acquires the image data to generate action instructions and sends the action instructions to the mechanical hand module. The mechanical hand module executes the action instructions to grab and distribute the to-be-sorted articles into the packaging containers. The present application can better coordinate the grabbing process of the mechanical hand module, thereby improving the efficiency of sorting and packaging. The grabbing range of the mechanical hand and the conveying of the packaging containers can be adjusted according to the real-time feeding situation of the to-be-sorted articles, so as to effectively avoid the situation of empty packaging containers.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of automatic production of pastries, and particularly relates to an automatic grabbing system, method and device based on machine vision and packaged pastries. BACKGROUND

[0002] At present, the pastry processing production line generally adopts an automatic production line for processing and production. When the pastries are sorted and packaged, a plurality of mechanical hands are usually arranged on the pastry conveying belt for identification and grabbing. The pastries on the tray are repeatedly identified and grabbed by the plurality of mechanical hands, so as to complete the grabbing and sorting of all pastries. However, each mechanical hand only relies on its own sensor to identify and grab the pastries, which results in a long time for the mechanical hand to identify and grab the pastries, affects the efficiency of the production of the pastry sorting and packaging, and the grabbing range of the mechanical hand and the conveying of the packaging container cannot be adjusted according to the real-time batch of pastry feeding, which is prone to cause the emptying of the packaging container. SUMMARY

[0003] In order to overcome the shortcomings of the prior art, the present application provides an automatic grabbing system based on machine vision to solve the problem that the prior art cannot adjust the grabbing range of the mechanical hand and the conveying of the packaging container according to the real-time batch of pastry feeding, and is prone to cause the emptying of the packaging container.

[0004] One scheme of the present application provides an automatic grabbing system based on machine vision, comprising:

[0005] A sorting conveying belt for conveying a tray carrying to-be-sorted articles;

[0006] A packaging conveying belt provided with a plurality of strips, and the plurality of strips are respectively arranged on both sides of the sorting conveying belt for conveying packaging containers;

[0007] A first vision detection module arranged above the sorting conveying belt, and the identification range of the first vision detection module covers the feeding area of the sorting conveying belt, for acquiring image data of a plurality of to-be-sorted articles on the tray;

[0008] A mechanical hand module provided with a plurality of mechanical hands, and the plurality of mechanical hand modules are arranged above the sorting conveying belt;

[0009] A control module in communication connection with the first vision detection module and the mechanical hand module;

[0010] The control module acquires the image data to generate a motion instruction and sends the motion instruction to the mechanical hand module, and the mechanical hand module executes the motion instruction to grab and distribute the to-be-sorted articles into the packaging containers.

[0011] In the scheme, the first visual detection module is arranged separately from the manipulator module, which reduces the recognition performance requirement of the manipulator module, and only needs to configure a processor in the first visual detection module and the control module for processing data or outputting action instructions, thereby reducing the equipment cost when multiple manipulator modules are arranged, and the coordination degree of the grabbing process is better through unified planning, so that the missing grabbing of the to-be-sorted articles or the emptying of the packaging containers is avoided, thereby improving the efficiency of sorting and packaging; the first visual detection module captures the to-be-sorted articles on the tray when the tray enters the feeding area of the sorting conveyor belt, to obtain image data of a plurality of to-be-sorted articles on the tray, and the image data is imported into the control module, the position coordinate information of each to-be-sorted article is obtained by the control module, and the action instruction is generated based on the position coordinate information and sent to the manipulator module, so that the action instruction of grabbing the to-be-sorted articles from the sorting conveyor belt to the packaging container on the packaging conveyor belt is completed when the manipulator module executes the action instruction;

[0012] Specifically, the action instruction is a movement instruction and a grabbing trigger instruction obtained by the control module after displacement compensation calculation based on the running speed of the sorting conveyor belt, the packaging conveyor belt and the manipulator module, and the method of overall recognition and same regulation in the scheme can adjust the grabbing range of the manipulator and the conveying of the packaging container according to the real-time feeding condition of the to-be-sorted articles in batches, so as to effectively avoid the emptying of the packaging container.

[0013] In one preferred scheme of the application, the packaging feeding module is arranged above the feeding area of the packaging conveyor belt and is used to supply the packaging containers to the packaging conveyor belt; the packaging feeding module is in communication connection with the control module.

[0014] And / or, the control module is in communication connection with the controller of the packaging conveyor belt.

[0015] In the scheme, after the packaging feeding module is in communication connection with the control module, the control module can adjust the feeding frequency of the packaging feeding module according to the real-time feeding quantity of the to-be-sorted articles, so as to avoid the emptying of the packaging containers caused by oversupply and reduce the waste of materials;

[0016] And after the control module is in communication connection with the controller of the packaging conveyor belt, the control module can suspend the conveying of the packaging containers of part of the packaging conveyor belt according to the real-time feeding quantity of the to-be-sorted articles, so as to sort the to-be-sorted articles into the packaging containers of the remaining packaging conveyor belt which is not suspended, thereby reducing unnecessary energy waste and avoiding the emptying of the packaging containers.

[0017] In one preferred scheme of the present application, the system further comprises a second visual detection module, which is arranged on both sides of the first visual detection module and covers the feeding area of the packaging conveying belt, and is used to obtain the position data of the packaging containers on the packaging conveying belt.

[0018] In this scheme, the second visual detection module covers the feeding area of the packaging conveying belt and is used to obtain the position data of the packaging containers on the packaging conveying belt. By combining the position data of the packaging containers and the real-time conveying speed of the packaging conveying belt, the position of the packaging containers at any time can be calculated by compensation, and the to-be-sorted items can be accurately transferred into the packaging containers.

[0019] In one preferred scheme of the present application, the first visual detection module comprises a shooting assembly and an image processing model, the image processing model is used to process the images shot by the shooting assembly; the image processing model comprises a noise reduction model and an edge detection model.

[0020] The noise reduction model is constructed based on at least one of a mean filter algorithm, a Gaussian filter algorithm, a bilateral filter algorithm or a wavelet denoising algorithm;

[0021] The edge detection model is constructed based on at least one of a Roberts operator, a Sobel operator, a Prewitt operator or a Canny edge detection algorithm.

[0022] In this scheme, the mean filter algorithm, the Gaussian filter algorithm, the bilateral filter algorithm or the wavelet denoising algorithm are all used to reduce the noise and interference in the images. By selecting one or more combinations, the image noise caused by environmental light or equipment factors can be reduced, while the loss of original features is minimized, the signal-to-noise ratio is improved, and subsequent image analysis and recognition are facilitated.

[0023] In addition, the Roberts operator, the Sobel operator, the Prewitt operator and the Canny edge detection algorithm are all algorithms for calculating the edge gradient in the image. By selecting one or more combinations, the edge of the to-be-sorted items and the edge of the tray can be accurately identified.

[0024] In one preferred scheme of the present application, the control module further comprises an efficiency evaluation model, which obtains a preset grabbing efficiency of the mechanical hand module and a preset conveying efficiency of the packaging conveying belt.

[0025] The efficiency evaluation model calculates real-time grabbing efficiency of the mechanical hand module based on the image data, and outputs a first regulation instruction if the real-time grabbing efficiency of the mechanical hand module is lower than a preset grabbing efficiency, and the mechanical hand module executes the first regulation instruction to suspend the grabbing work;

[0026] The efficiency evaluation model calculates real-time conveying quantity of the packaging conveying belt based on the image data, and outputs a second regulation instruction if the real-time conveying quantity of the packaging conveying belt is lower than a preset conveying quantity, and the packaging conveying belt executes the second regulation instruction to reduce the output quantity of the packaging container.

[0027] In the scheme, when the feeding quantity of the to-be-sorted articles is lower than the grabbing efficiency of the mechanical hand, the grabbing quantity of a single mechanical hand is reduced when multiple mechanical hands grab, and therefore, the first regulation instruction is used to suspend the grabbing work of part of the mechanical hands, so that the energy consumption of the mechanical hands in operation can be saved.

[0028] After the packaging conveying belt executes the second regulation instruction to reduce the output quantity of the packaging container, the mechanical hand module can be simultaneously adjusted to place the to-be-sorted articles on the packaging conveying belt on one side of the sorting conveying belt.

[0029] In one of the preferred schemes of the present application, the control module further comprises a sorting area division model, the sorting area division model marks coordinate points of the to-be-sorted articles on the tray to obtain coordinate point information of the to-be-sorted articles, the sorting area division model divides a plurality of sorting areas on the tray according to the coordinate point information of the to-be-sorted articles, and matches the sorting areas with adjacent packaging conveying belts.

[0030] In the scheme, by dividing a plurality of sorting areas, the mechanical hand module is allowed to sort the to-be-sorted articles in a single corresponding sorting area, so that the mutual influence of multiple mechanical hand modules in the sorting process can be avoided.

[0031] In one of the preferred schemes of the present application, the mechanical hand module comprises a gripper assembly, a displacement assembly, and a mechanical hand controller, the mechanical hand controller is electrically connected with the gripper assembly and the displacement assembly, and the gripper assembly moves between the sorting conveying belt and the packaging conveying belt through the displacement assembly.

[0032] In one of the preferred schemes of the present application, the displacement assembly further comprises a rotating assembly, the rotating assembly is rotatable along a rotating shaft, the gripper assembly is in transmission connection with the rotating assembly, and the rotating assembly is used to adjust the grabbing posture of the gripper assembly.

[0033] In the scheme, the rotating assembly is used to adjust the grasping posture of the gripper assembly, specifically, when the displacement assembly moves to above the to-be-sorted articles on the packaging conveying belt according to the action instruction, the rotating assembly makes corresponding rotation angle compensation according to the posture data of the to-be-sorted articles identified in the image data, so as to drive the rotating assembly to rotate to a compensation angle, so that the manipulator grasps the to-be-sorted articles in the compensation posture, and the manipulator returns to the preset posture, so that the to-be-sorted articles are loaded into the packaging container in the predetermined posture. Thus, it is ensured that the to-be-sorted articles grasped into the packaging container are in the predetermined posture, which facilitates the subsequent packaging process, and the arrangement of the to-be-sorted articles is more standardized and neat, and the aesthetic appearance is improved.

[0034] In one preferred scheme of the present application, an automatic grasping method based on machine vision is also pointed out, which can be used in the automatic grasping system based on machine vision in any of the above schemes, comprising:

[0035] Obtaining image data of a plurality of to-be-sorted articles in a tray;

[0036] Denoising and edge detection are performed on the image data, and the output is pretreatment data;

[0037] Based on the pretreatment data, position feature data of a plurality of to-be-sorted articles are extracted, and the output is position information;

[0038] The position information is input into a path planning model to generate an action instruction;

[0039] The action instruction is executed by a manipulator to grasp a plurality of to-be-sorted articles into a packaging container.

[0040] In the scheme, the grasping is planned uniformly, so that the coordination degree of the grasping process is better, and the missing of to-be-sorted articles or the vacancy of the packaging container is avoided, thereby improving the efficiency of sorting and packaging; in combination with the automatic grasping system based on machine vision, the first vision detection module is used to take pictures when the tray of to-be-sorted articles enters the feeding area of the sorting conveying belt, so as to obtain image data of a plurality of to-be-sorted articles on the tray, and the image data is input into the control module, the position coordinate information of each to-be-sorted article is analyzed and obtained by the control module, and the action instruction is generated based on the position coordinate information and sent to the manipulator module, and the action instruction is completed when the manipulator module executes the action instruction, that is, the to-be-sorted articles are grasped from the sorting conveying belt to the packaging container on the packaging conveying belt;

[0041] Specifically, the action instruction is a movement instruction and a grabbing trigger instruction obtained by the control module based on the displacement compensation calculation of the running speed of the sorting conveying belt, the packaging conveying belt and the manipulator module. According to the method of overall identification and same regulation in the scheme, the grabbing range of the manipulator and the conveying of the packaging container can be adjusted according to the feeding situation of the real-time batch of to-be-sorted articles, so as to effectively avoid the situation that the packaging container is empty.

[0042] In one preferred scheme of the present application, the method further comprises adjusting the grabbing posture of the manipulator, and the specific method comprises:

[0043] Based on the pre-processing data, the posture information of the to-be-sorted article is extracted;

[0044] The posture information is input into a posture compensation model to generate a posture adjustment instruction;

[0045] The posture adjustment instruction is executed by the manipulator, and the manipulator grabs the to-be-sorted article in a compensated posture;

[0046] The manipulator returns to a preset posture, so that the to-be-sorted article is loaded into the packaging container in a predetermined posture.

[0047] In one preferred scheme of the present application, an automatic grabbing device based on machine vision is also pointed out, comprising:

[0048] At least one processor; and,

[0049] The memory is in communication connection with the at least one processor; wherein the memory has instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to implement the automatic grabbing method based on machine vision of any one of the above schemes when executed.

[0050] In one preferred scheme of the present application, a packaging pastry is also pointed out, which is grabbed and transported to a packaging workshop by the automatic grabbing system based on machine vision of any one of the above schemes. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings from the structures shown in the drawings without creative labor.

[0052] Figure 1 The structure diagram of the automatic grabbing system based on machine vision of one embodiment of the present application is shown.

[0053] Figure 2 A flowchart illustrating an embodiment of the automatic grasping method based on machine vision according to the present invention;

[0054] Figure 3 A flowchart illustrating another embodiment of the automatic grasping method based on machine vision of the present invention;

[0055] Figure 4 This is a schematic diagram illustrating the architecture of an automatic grasping device based on machine vision, according to one embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0057] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative positional relationship and movement of the components in a specific posture. If the specific posture changes, the directional indications will also change accordingly.

[0058] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the use of "and / or" or "and / or" throughout the text includes three parallel solutions. For example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0059] In the embodiments of the present application, the to-be-sorted articles can be selected as different specifications of mooncakes or frozen pastries. The pastries and mooncakes are two different types of snacks in Chinese traditional food. The pastries refer to food made of flour as the main raw material, such as dumplings, steamed buns, and steamed buns. The mooncake is a special Mid-Autumn Festival food, which is usually composed of a skin and a filling. The skin has various types, such as the traditional Cantonese mooncake skin and the Suzhou mooncake skin. The filling also has various flavors, such as lotus paste, bean paste, and five nuts.

[0060] When the to-be-packaged articles are mooncakes or frozen pastries, the packaging containers are preferably half-opened cake or bread holders.

[0061] Please refer to Figure 1 One of the embodiments of the present application provides an automatic grabbing system 100 based on machine vision, comprising:

[0062] A sorting conveyor belt 110 is used to convey a tray carrying to-be-sorted articles.

[0063] A packaging conveyor belt 120 is provided with a plurality of strips, and the plurality of packaging conveyor belts 120 are respectively arranged on both sides of the sorting conveyor belt 110, for conveying packaging containers.

[0064] A first visual detection module 130 is arranged above the sorting conveyor belt 110, and the identification range of the first visual detection module 130 covers the feeding area of the sorting conveyor belt 110, for acquiring image data of a plurality of to-be-sorted articles on the tray.

[0065] A plurality of mechanical hand modules 140 are arranged above the sorting conveyor belt 110.

[0066] A control module 150 is in communication connection with the first visual detection module 130 and the mechanical hand module 140.

[0067] The control module 150 acquires the image data to generate action instructions and sends them to the mechanical hand module 140. The mechanical hand module 140 executes the action instructions to grab and distribute the to-be-sorted articles into the packaging containers.

[0068] In the embodiment, the first visual detection module 130 is arranged separately from the mechanical hand module 140, which reduces the recognition performance requirement of the mechanical hand module 140, and only needs to configure a processor in the first visual detection module 130 and the control module 150 for processing data or outputting action instructions, thereby reducing the equipment cost when multiple mechanical hand modules 140 are arranged. Meanwhile, the grabbing process is better coordinated through unified planning, so that the missing grabbing of the to-be-sorted articles or the emptying of the packaging containers is avoided, thereby improving the efficiency of sorting and packaging. The first visual detection module 130 photographs when the tray of the to-be-sorted articles enters the feeding area of the sorting conveyor belt 110, so as to obtain image data of a plurality of to-be-sorted articles on the tray, and the image data is imported into the control module 150. The control module 150 analyzes and obtains the position coordinate information of each to-be-sorted article, and generates an action instruction based on the position coordinate information and sends the action instruction to the mechanical hand module 140. When the mechanical hand module 140 executes the action instruction, the action instruction of grabbing the to-be-sorted articles from the sorting conveyor belt 110 to the packaging containers on the packaging conveyor belt 120 is completed.

[0069] Specifically, the action instruction is a movement instruction and a grabbing trigger instruction obtained by the control module 150 after displacement compensation calculation based on the running speeds of the sorting conveyor belt 110, the packaging conveyor belt 120 and the mechanical hand module 140. According to the method of overall recognition and unified control in the embodiment, the grabbing range of the mechanical hand and the conveying of the packaging containers can be adjusted according to the real-time feeding situation of the batch of to-be-sorted articles, so that the emptying of the packaging containers is effectively avoided.

[0070] In one preferred embodiment of the application, the system further comprises a first sensor and a plurality of second sensors. The first sensor is arranged below the sorting conveyor belt 110 and is used to obtain the conveying speed of the sorting conveyor belt 110. The second sensor is arranged below the packaging conveyor belt 120 and is used to obtain the conveying speed of the packaging conveyor belt 120.

[0071] In the embodiment, the recording data of the first sensor and the recording data of the second sensor realize time synchronization between devices based on time code (Timecode) technology, so as to facilitate marking and synchronizing the speed information of the sorting conveyor belt 110 and the plurality of packaging conveyor belts 120, avoid data deviation caused by asynchronous recording time, ensure that the displacement distance of the to-be-sorted articles on the sorting conveyor belt 110 and the displacement distance of the packaging containers on the packaging conveyor belt 120 are more accurate in calculation and cooperation, improve the efficiency and accuracy of the grabbing action, and avoid deviation caused by long-time operation.

[0072] In one of the preferred embodiments of the present application, the first sensor and the second sensor are both in communication connection with the control module 150; the control module 150 generates a grabbing track based on the transmission speed of the sorting conveyor belt 110, the transmission speed of the packaging conveyor belt 120 and the image data, and outputs as an action instruction.

[0073] In the present embodiment, after the first sensor and the second sensor are in communication connection with the control module 150, the control module 150 can calculate the actual displacement distance of the to-be-sorted items based on the real-time acquired transmission speed of the sorting conveyor belt 110, and calculate the actual displacement distance of the packaging containers based on the real-time acquired transmission speed of the packaging conveyor belt 120, so that the movement of the mechanical hand from the initial position to the coordinate position of the to-be-sorted items is more accurate.

[0074] In one of the preferred embodiments of the present application, the packaging material falling module 160 is arranged above the feeding area of the packaging conveyor belt 120, and is used for supplying packaging containers to the packaging conveyor belt 120; the packaging material falling module 160 is in communication connection with the control module 150.

[0075] And / or, the control module 150 is in communication connection with the controller of the packaging conveyor belt 120.

[0076] In the present embodiment, after the packaging material falling module 160 is in communication connection with the control module 150, the control module 150 can adjust the falling frequency of the packaging material falling module 160 according to the real-time feeding quantity of the to-be-sorted items, so as to reduce the emptying phenomenon caused by excess supply of packaging containers and reduce the waste of materials;

[0077] And after the control module 150 is in communication connection with the controller of the packaging conveyor belt 120, the control module 150 can suspend the conveying of the packaging containers of part of the packaging conveyor belt 120 according to the real-time feeding quantity of the to-be-sorted items, so as to sort the to-be-sorted items into the packaging containers of the remaining packaging conveyor belt 120 which is not suspended, thereby reducing unnecessary energy waste and avoiding the emptying phenomenon.

[0078] In one of the preferred embodiments of the present application, the system further comprises a second visual detection module 170, the second visual detection module 170 is arranged on both sides of the first visual detection module 130, and the identification range of the second visual detection module 170 covers the feeding area of the packaging conveyor belt 120, and is used for acquiring the position data of the packaging containers on the packaging conveyor belt 120.

[0079] In the embodiment, the second visual detection module 170 covers the feeding area of the packaging conveying belt 120, and is configured to acquire position data of the packaging containers on the packaging conveying belt 120. By combining the position data of the packaging containers and the real-time conveying speed of the packaging conveying belt 120, the position of the packaging containers at any time can be calculated by compensation, and the to-be-sorted items can be accurately transferred into the packaging containers.

[0080] In one preferred embodiment of the present application, the first visual detection module 130 comprises a shooting assembly and an image processing model configured to process images shot by the shooting assembly. The image processing model comprises a noise reduction model and an edge detection model.

[0081] The noise reduction model is constructed based on at least one of a mean filter algorithm, a Gaussian filter algorithm, a bilateral filter algorithm or a wavelet denoising algorithm.

[0082] The edge detection model is constructed based on at least one of a Roberts operator, a Sobel operator, a Prewitt operator or a Canny edge detection algorithm.

[0083] In the embodiment, the mean filter algorithm, the Gaussian filter algorithm, the bilateral filter algorithm or the wavelet denoising algorithm are all used to reduce noise and interference in images. By selecting one or more combinations thereof, the algorithms can be used to

[0084] In addition, the Roberts operator, the Sobel operator, the Prewitt operator and the Canny edge detection algorithm are all algorithms for calculating edge gradients in images. By selecting one or more combinations thereof, the algorithms can be used to accurately identify edges of to-be-sorted items and edges of trays.

[0085] In one preferred embodiment of the present application, the control module 150 further comprises an efficiency evaluation model configured to acquire a preset grabbing efficiency of the mechanical hand module 140 and a preset conveying efficiency of the packaging conveying belt 120.

[0086] The efficiency evaluation model is configured to calculate a real-time grabbing efficiency of the mechanical hand module 140 based on the image data. If the real-time grabbing efficiency of the mechanical hand module 140 is lower than the preset grabbing efficiency, the efficiency evaluation model outputs a first regulation and control instruction, and the mechanical hand module 140 executes the first regulation and control instruction to suspend the grabbing operation.

[0087] The efficiency evaluation model calculates a real-time conveying quantity of the packaging conveying belt 120 based on the image data, and outputs a second regulation instruction if the real-time conveying quantity of the packaging conveying belt 120 is lower than a preset conveying quantity. The packaging conveying belt 120 executes the second regulation instruction to reduce the output quantity of the packaging containers.

[0088] In the embodiment, when the feeding quantity of the to-be-sorted articles is lower than the grabbing efficiency of the manipulator, the grabbing quantity of a single manipulator is reduced when multiple manipulators grab, and thus the first regulation instruction is used to make part of the manipulators pause the grabbing work, so as to save the energy consumption of the manipulator.

[0089] After the packaging conveying belt 120 executes the second regulation instruction to reduce the output quantity of the packaging containers, the manipulator module 140 can be simultaneously adjusted to place the to-be-sorted articles on the packaging conveying belt 120 on one side of the sorting conveying belt 110.

[0090] In one preferred embodiment of the present application, the control module 150 further comprises a sorting area division model. The sorting area division model marks coordinate points of the to-be-sorted articles on the tray to obtain coordinate point information of the to-be-sorted articles. The sorting area division model divides a plurality of sorting areas on the tray according to the coordinate point information of the to-be-sorted articles, and matches the sorting areas with adjacent packaging conveying belts 120.

[0091] In the embodiment, by dividing a plurality of sorting areas, the manipulator module 140 can correspondingly sort the to-be-sorted articles in a single corresponding sorting area, so as to avoid mutual influence of multiple manipulator modules 140 in the sorting process.

[0092] In one preferred embodiment of the present application, the manipulator module 140 comprises a gripper assembly, a displacement assembly, and a manipulator controller. The manipulator controller is electrically connected with the gripper assembly and the displacement assembly. The gripper assembly moves between the sorting conveying belt 110 and the packaging conveying belt 120 through the displacement assembly.

[0093] In one preferred embodiment of the present application, the displacement assembly further comprises a rotating assembly. The rotating assembly can rotate along a rotating shaft. The gripper assembly is in transmission connection with the rotating assembly. The rotating assembly is used to adjust the grabbing posture of the gripper assembly.

[0094] In the embodiment, the rotating assembly is used to adjust the grasping posture of the gripper assembly, specifically, when the displacement assembly moves to above the to-be-sorted articles on the packaging conveying belt 120 according to the action instruction, the rotating assembly makes corresponding rotation angle compensation according to the posture data of the to-be-sorted articles identified in the image data, so as to drive the rotating of the gripper assembly to the compensation angle, so that the manipulator grasps the to-be-sorted articles in the compensation posture, and the manipulator returns to the preset posture, so that the to-be-sorted articles are loaded into the packaging container in the predetermined posture. Therefore, it is ensured that the to-be-sorted articles grasped into the packaging container are in the predetermined posture, which facilitates the subsequent packaging process, and the arrangement of the to-be-sorted articles is more standardized and neat, and the aesthetic appearance is improved.

[0095] In one preferred embodiment of the present application, an automatic grasping method based on machine vision is also pointed out, which can be used in the automatic grasping system 100 based on machine vision in any one of the above embodiments, and includes:

[0096] S10, acquiring image data of a plurality of to-be-sorted articles in a tray;

[0097] S20, denoising and edge detection are performed on the image data, and the output is pretreatment data;

[0098] S30, based on the pretreatment data, position feature data of a plurality of to-be-sorted articles are extracted, and the output is position information;

[0099] S40, inputting the position information into a path planning model to generate an action instruction;

[0100] S50, executing the action instruction by a manipulator to grasp a plurality of to-be-sorted articles into a packaging container.

[0101] In the embodiment, the grasping is performed by unified planning, so that the coordination degree of the grasping process is better, and the missing of to-be-sorted articles or the vacancy of the packaging container is avoided, thereby improving the efficiency of sorting and packaging; in combination with the automatic grasping system 100 based on machine vision, the first vision detection module 130 is used to take pictures when the tray of to-be-sorted articles enters the feeding area of the sorting conveying belt 110, so as to acquire image data of a plurality of to-be-sorted articles on the tray, and the image data is input into the control module 150, the position coordinate information of each to-be-sorted article is analyzed and acquired by the control module 150, and the action instruction is generated based on the position coordinate information and is sent to the manipulator module 140, and the action instruction is completed when the manipulator module 140 executes the action instruction, that is, the to-be-sorted articles are grasped from the sorting conveying belt 110 to the packaging container on the packaging conveying belt 120;

[0102] Specifically, the action instruction is a movement instruction obtained by the control module 150 based on the displacement compensation calculation of the running speed of the sorting conveyor belt 110, the packaging conveyor belt 120, and the manipulator module 140, and a grabbing trigger instruction, combined with the method of overall identification and same regulation in this embodiment, the grabbing range of the manipulator and the conveying of the packaging container can be adjusted according to the real-time batch of the feeding condition of the to-be-sorted articles, and the situation of empty packaging container can be effectively avoided.

[0103] In this embodiment, the path planning model is constructed based on a greedy algorithm; the core idea of the greedy algorithm is to select a local optimal solution each time, that is, to select the target closest to the current position, and through iteration of this process, the robot can gradually move and sort all targets.

[0104] In the application scenario of this embodiment, based on the preprocessed data, the coordinate point information of all to-be-sorted articles on the tray is obtained, and the current position of the manipulator module is taken as the starting point, the distance between the starting point and each ungrabbed to-be-sorted article is calculated, and the closest to-be-sorted article is found, the manipulator module is moved to the coordinate point of the closest to-be-sorted article to perform a grabbing and sorting action, and then the current position of the manipulator module is updated to the coordinate point of the just sorted to-be-sorted article, the above process of finding the closest to-be-sorted article and grabbing is repeated until all to-be-sorted articles are grabbed and sorted.

[0105] In one embodiment, the control module 150 includes a defective product classification model, and the following methods can be used to realize the selection of defective products while performing the unquantitative sorting task based on machine vision:

[0106] A defective product classification model is established using machine learning or deep learning methods, and the model is trained using image data of to-be-sorted articles to enable it to identify and classify defective products and distinguish them from normal products, such as temperature abnormalities, shape defects, or color differences. Appropriate threshold values are set based on the characteristics of defective products to determine whether they are defective. By setting reasonable parameters and algorithms in the vision system, products that may be defective are selected from the detected targets. Using machine learning methods, a defective product identification model is established by combining historical data and feature engineering, which learns and classifies the characteristics of defective products to accurately identify and select defective products. The sorting and selection operations are combined and performed simultaneously in the machine vision system, and the defective products in the to-be-sorted articles are separated by reasonable grabbing sequence and path planning. Further, in the automatic sorting process, a manual review link can be introduced to manually judge and confirm the products that may have doubts, thereby improving the accuracy and reliability of the selection.

[0107] Please refer to Figure 3In one preferred embodiment of the present application, the method further comprises adjusting the grasping posture of the robot arm, and the specific method comprises:

[0108] S51, based on the preprocessed data, extracting posture information of the to-be-sorted item;

[0109] S52, inputting the posture information into a posture compensation model to generate a posture adjustment instruction;

[0110] S53, executing the posture adjustment instruction by the robot arm, and the robot arm grasps the to-be-sorted item in a compensated posture;

[0111] S54, the robot arm returns to a preset posture, so that the to-be-sorted item is loaded into the packaging container in a predetermined posture.

[0112] In this embodiment, the posture compensation model performs feature recognition (such as contour analysis or text recognition) on the posture information, obtains the features of the to-be-sorted object (such as the pattern direction of the pattern, or the horizontal or vertical direction of the text), and then matches the extracted features with the pre-defined features to find the corresponding relationship between the features in the image and the pre-defined reference features; according to the feature matching result, the rotation angle posture of the object is calculated and the rotation angle difference with the reference feature is obtained, and appropriate compensation instructions (such as rotating the robot arm before grasping based on the rotation angle difference) are generated. Please refer to Figure 4 One embodiment of the present application provides a control device 200, comprising:

[0113] at least one processor 210; and a memory 220 connected in communication with the at least one processor 210; wherein the memory 220 stores instructions executable by the at least one processor 210, and the instructions are executed by the at least one processor 210 to enable the at least one processor 210 to implement the above-mentioned automatic grasping method based on machine vision. In this embodiment, the memory 220 stores a computer program 240. The processor 210 and the memory 220 are connected through a communication bus 230.

[0114] In one preferred embodiment of the present application, a packaging pastry is also provided, which is grasped and transported to a packaging workshop by the automatic grasping system 100 based on machine vision according to any one of the above-mentioned embodiments.

[0115] The above-mentioned only preferred embodiments of the present application, and not limit the patent scope of the present application, any equivalent structural transformation made under the inventive concept of the present application, and directly / indirectly applied in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A machine vision based automatic grasping system, characterized in that, The application relates to a sorting and packaging system. The system comprises: a sorting conveyor for conveying a tray carrying to-be-sorted items; a packaging conveyor arranged with a plurality of strips, and the plurality of strips are arranged on both sides of the sorting conveyor respectively, for conveying packaging containers; a first visual detection module arranged above the sorting conveyor, and the identification range of the first visual detection module covers the feeding area of the sorting conveyor, for acquiring image data of a plurality of to-be-sorted items on the tray; a plurality of mechanical hand modules arranged above the sorting conveyor; a control module in communication connection with the first visual detection module and the mechanical hand modules; the control module further comprises an efficiency evaluation model, which acquires a preset grabbing efficiency of the mechanical hand modules and a preset conveying efficiency of the packaging conveyor; the efficiency evaluation model calculates the real-time grabbing efficiency of the mechanical hand modules based on the image data, and if the real-time grabbing efficiency of the mechanical hand modules is lower than the preset grabbing efficiency, the efficiency evaluation model outputs a first regulation and control instruction to make part of the mechanical hands suspend the grabbing work; the efficiency evaluation model calculates the real-time conveying quantity of the packaging conveyor based on the image data, and if the real-time conveying quantity of the packaging conveyor is lower than the preset conveying quantity, the efficiency evaluation model outputs a second regulation and control instruction, and the packaging conveyor executes the second regulation and control instruction to reduce the output quantity of the packaging containers; 2. The machine vision-based automated grasping system of claim 1, wherein, wherein the control module acquires the image data to generate a motion instruction and sends the motion instruction to the mechanical hand modules, and the mechanical hand modules execute the motion instruction to grab and distribute the to-be-sorted items into the packaging containers. a packaging material falling module arranged above the feeding area of the packaging conveyor, for supplying the packaging conveyor with packaging containers; the packaging material falling module is in communication connection with the control module; 3. The machine vision-based automated grasping system of claim 2, wherein, and / or the control module is in communication connection with the controller of the packaging conveyor.

4. The machine vision-based automated grasping system of claim 1, wherein, The system further comprises a second visual detection module arranged on both sides of the first visual detection module, and the identification range of the second visual detection module covers the feeding area of the packaging conveyor, for acquiring position data of the packaging containers on the packaging conveyor. The first visual detection module comprises a shooting assembly and an image processing model, and the image processing model is used for processing images shot by the shooting assembly; the image processing model comprises a noise reduction model and an edge detection model; the noise reduction model is constructed based on at least one algorithm of a mean filtering algorithm, a Gaussian filtering algorithm, a bilateral filtering algorithm or a wavelet noise reduction algorithm; the edge detection model is constructed based on at least one algorithm of Roberts operator, Sobel operator, Prewitt operator or Canny edge detection.

5. The machine vision-based automated grasping system of claim 1, wherein, The control module further comprises a sorting area division model, which marks coordinate points of the to-be-sorted articles on the tray, obtains coordinate point information of the to-be-sorted articles, and divides a plurality of sorting areas on the tray according to the coordinate point information of the to-be-sorted articles, and matches the sorting areas with adjacent packaging conveying belts.

6. The machine vision-based automated grasping system of claim 1, wherein, The mechanical arm module comprises a gripper assembly, a displacement assembly, and a mechanical arm controller, the mechanical arm controller is electrically connected with the gripper assembly and the displacement assembly, and the gripper assembly moves between the sorting conveying belt and the packaging conveying belt through the displacement assembly.

7. The machine vision-based automated grasping system of claim 6, wherein, The displacement assembly further comprises a rotating assembly, the rotating assembly can rotate along a rotating shaft, the gripper assembly is in transmission connection with the rotating assembly, and the rotating assembly is used for adjusting the grasping posture of the gripper assembly.

8. A machine vision-based automatic grasping method, characterized by, The machine vision-based automatic grasping system according to any one of claims 1-7 comprises: acquiring image data of a plurality of to-be-sorted articles in a tray; performing denoising and edge detection on the image data to output pretreatment data; based on the pretreatment data, extracting position feature data of a plurality of to-be-sorted articles to output position information; inputting the position information into a path planning model to generate action instructions; grabbing a plurality of to-be-sorted articles to packaging containers through a mechanical arm according to the action instructions.

9. The machine vision-based automated grasping method of claim 8, wherein, The method further comprises adjusting the grasping posture of the mechanical arm, and the specific method comprises: based on the pretreatment data, extracting posture information of the to-be-sorted articles; inputting the posture information into a posture compensation model to generate posture adjustment instructions; grabbing the to-be-sorted articles by the mechanical arm in a compensated posture according to the posture adjustment instructions; the mechanical arm returns to a preset posture, and the to-be-sorted articles are loaded into the packaging containers in a predetermined posture.

10. A machine vision based automatic grasping device, characterized by, comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory has instructions stored thereon that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to implement the machine vision-based automatic grasping method according to claim 8 or 9 when executed.

Images