Disordered wire sorting method, device and system

Abstract

The present invention provides a method, device and system for sorting disordered wires, and relates to the field of automatic sorting technology. The method includes: collecting a material box image and point cloud data corresponding to the material box image, identifying the position of each wire in the material box image and the position of the identification code on the wire; based on the position of each wire and the position of each identification code, selecting qualified position wires that can be grabbed in the material box; wherein the qualified position wires are wires whose wire image and corresponding identification code image can be simultaneously visible in the material box image; based on the point cloud data corresponding to the qualified position wires, selecting the optimal position wires in the material box that will not cause grabbing collision; and controlling the grabbing robot to preferentially grab the optimal position wires in the material box. The present invention can enable the grabbing robot to quickly and accurately grab and sort the wires in the material box, thereby improving the automation degree and efficiency of wire sorting.

Description

Technical Field

[0001] The present invention relates to the field of automatic sorting technology, and in particular to a method, device and system for sorting disordered wires. Background Art

[0002] At present, automated assembly lines usually need to sort wires. The existing wire sorting methods usually adopt manual sorting or automatic machine sorting. However, manual sorting of wires is inefficient and has high labor costs. Since the wires in the material boxes of the assembly production line are usually placed in a messy manner, the accuracy of automatic wire sorting is reduced when machine sorting is adopted. When identification codes are affixed to the wires, the wires and the identification codes block each other, making it difficult to identify the corresponding relationship between the wires and the identification codes, and thus it is difficult to identify the wires that the robot can grasp, resulting in low wire sorting efficiency. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a method, device and system for sorting disordered wires, which can facilitate a grasping robot to quickly grasp and sort the wires in a discharge box, thereby improving the degree of automation and efficiency of wire sorting.

[0004] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0005] In the first aspect, an embodiment of the present invention provides a method for sorting disordered wires, comprising: collecting a material box image and point cloud data corresponding to the material box image, identifying the position of each wire in the material box image and the position of an identification code on the wire; screening out qualified position wires that can be grasped in the material box based on the position of each wire and the position of each identification code; wherein the qualified position wires are wires whose wire image and the corresponding identification code image can be simultaneously visible in the material box image; screening out the optimal position wires in the material box that will not cause grasping collision based on the point cloud data corresponding to the qualified position wires; and controlling the grasping robot to preferentially grasp the optimal position wires in the material box.

[0006] Further, an embodiment of the present invention provides a first possible implementation manner of the first aspect, wherein the step of screening out qualified position wires that can be grasped in the material box based on the position of each of the wires and the position of each of the identification codes comprises: establishing a wire 2D bounding box in the material box image based on the position of each of the wires, and establishing an identification code 2D bounding box in the material box image based on the position of each of the identification codes; screening out matching wire images and identification code images based on the pixel distance between the center point coordinates of each of the wire 2D bounding boxes and the center point coordinates of each of the identification code 2D bounding boxes; and using the wires corresponding to the screened matching wire images and identification code images as qualified position wires that can be grasped in the material box.

[0007] Further, an embodiment of the present invention provides a second possible implementation of the first aspect, wherein the step of screening out matching wire images and identification code images based on the pixel distance between the center point coordinates of each wire 2D bounding box and the center point coordinates of each identification code 2D bounding box comprises: when the pixel distance between the center point coordinates of the wire 2D bounding box and the center point coordinates of the identification code 2D bounding box satisfies a first formula, determining that the wire image in the wire 2D bounding box matches the identification code image in the identification code 2D bounding box; wherein the first formula is:

[0008]

[0009] is the pixel coordinate of the center point of the ith wire 2D bounding box in the bin image, is the pixel coordinate of the center point of the j-th identification code 2D bounding box in the material box image, and pixel_distance_threshold is a preset threshold.

[0010] Furthermore, an embodiment of the present invention provides a third possible implementation of the first aspect, wherein the preset threshold is related to the size of the wire and the pasting position of the identification code on the wire.

[0011] Further, an embodiment of the present invention provides a fourth possible implementation method of the first aspect, wherein the step of screening out the optimally positioned wire in the material box that will not cause a grasping collision based on the point cloud data corresponding to the qualified position wire includes: sorting the placement heights of the qualified position wires in the material box from large to small based on the point cloud data to obtain a placement height sorting result; obtaining a collision-free working convex hull area of ​​the grasping robot in the material box; determining whether the qualified position wire with the largest placement height is located within the collision-free working convex hull area, and if so, using the qualified position wire with the largest placement height as the optimally positioned wire.

[0012] Furthermore, an embodiment of the present invention provides a fifth possible implementation of the first aspect, wherein the method for sorting unordered wires also includes: establishing a 3D bounding box of a material box based on the point cloud data, and filtering noise points outside the material box in the point cloud data based on the 3D bounding box of the material box; obtaining the point cloud coordinates of the execution end of the grasping robot under the point cloud data and the working range of the grasping robot to determine the collision-free working convex hull area of ​​the grasping robot in the material box.

[0013] Further, an embodiment of the present invention provides a sixth possible implementation of the first aspect, wherein the disordered wire sorting method also includes: when the qualified position wire with the largest placement height is not within the collision-free working convex hull area, determining in turn whether the qualified position wire in the placement height sorting result is within the collision-free working convex hull area, until the optimal position wire with the largest placement height within the collision-free working convex hull area is obtained.

[0014] Furthermore, an embodiment of the present invention provides a seventh possible implementation manner of the first aspect, wherein the step of identifying the position of each wire in the material box image and the position of the identification code on the wire includes: acquiring a virtual material box image and virtual point cloud data based on a virtual camera, and establishing a virtual data set based on the virtual material box image and the virtual point cloud data; collecting image data and point cloud data of the material box under different lighting levels and different working scenes to establish a real data set; annotating the real data set with identification codes and wires, training a neural network model based on the virtual data set, and performing transfer learning on the trained neural network model based on the annotated real data set to obtain a target neural network model; identifying the material box image based on the target neural network model to obtain the position of each wire in the material box image and the position of the identification code on the wire.

[0015] In the second aspect, an embodiment of the present invention further provides an unordered wire sorting device, comprising: an identification module, used to collect a material box image and point cloud data corresponding to the material box image, and identify the position of each wire in the material box image and the position of the identification code on the wire; a first screening module, used to screen out qualified position wires that can be grasped in the material box based on the position of each wire and the position of each identification code; wherein the qualified position wires are wires whose wire image and corresponding identification code image can be simultaneously visible in the material box image; a second screening module, used to screen out the optimal position wires in the material box that will not cause grasping collision based on the point cloud data corresponding to the qualified position wires; a sorting module, used to control the grasping robot to preferentially grasp the optimal position wires in the material box.

[0016] In a third aspect, an embodiment of the present invention provides a disordered wire sorting system, comprising: a visual sensor, a grasping robot and a controller, the controller comprising a processor and a storage device; the visual sensor is used to collect a material box image and point cloud data corresponding to the material box image; the grasping robot is used to grasp the wire at the optimal position in the material box; the storage device stores a computer program, and the computer program, when executed by the processor, executes the method as described in any one of the first aspects.

[0017] The embodiment of the present invention provides a method, device and system for sorting disordered wires, the method comprising: collecting a material box image and point cloud data corresponding to the material box image, identifying the position of each wire in the material box image and the position of an identification code on the wire; screening out qualified position wires that can be grasped in the material box based on the position of each wire and the position of each identification code; wherein the qualified position wires are wires whose wire image and the corresponding identification code image can be simultaneously visible in the material box image; screening out the optimal position wires in the material box that will not cause grasping collision based on the point cloud data corresponding to the qualified position wires; and controlling the grasping robot to preferentially grasp the optimal position wires in the material box. The present invention can identify the position of the wires and the position of the identification code in the material box image, and can recognize the correspondence between the wires and the identification code, so as to screen out the wires and the identification codes pasted thereon, and at the same time expose the qualified position wires that can be grasped by the robot, and by screening out the optimal position wires that will not cause grasping collision from the qualified position wires according to the point cloud data corresponding to the qualified position wires, the grasping robot can quickly and accurately grasp and sort the wires in the material box, thereby improving the degree of automation and wire sorting efficiency.

[0018] Other features and advantages of the embodiments of the present invention will be described in the following description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned techniques of the embodiments of the present invention.

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A flow chart of a disordered wire sorting method provided by an embodiment of the present invention is shown;

[0022] Figure 2 A schematic diagram of a material box image provided by an embodiment of the present invention is shown;

[0023] Figure 3 A flowchart of optimal wire grabbing selection provided by an embodiment of the present invention is shown;

[0024] Figure 4 A schematic structural diagram of a disordered wire sorting device provided in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be described below in conjunction with the drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0026] This embodiment provides a method for sorting disordered wires, which can be applied to electronic devices such as computers. Figure 1 The flow chart of the disordered wire sorting method shown in the figure mainly includes the following steps:

[0027] Step S102, collecting a material box image and point cloud data corresponding to the material box image, identifying the position of each wire in the material box image and the position of the identification code on the wire.

[0028] Based on the image sensor, an image of a material box containing disordered wires is collected, which is recorded as a material box image. Based on the three-dimensional laser radar, three-dimensional point cloud data of the material box is scanned. The above-mentioned image sensor can be a camera or a depth camera, and the above-mentioned material box can be an RGB image or a depth image. In one embodiment, the above-mentioned image sensor and three-dimensional laser radar can be integrated into a 3D vision sensor so that the material box image and point cloud data can be collected at the same time. According to the calibration data of the image sensor and the three-dimensional laser radar, the three-dimensional point cloud coordinates corresponding to each pixel point in the material box image can be obtained.

[0029] In a specific embodiment, each wire in the above-mentioned material box is affixed with an identification code, which can be a barcode or a QR code. Multiple material box images can be collected in advance, and the material box images marked with the wire positions and the identification code positions are input into the neural network model as sample images for training. Based on the trained neural network model, the material box images collected on the automated assembly line are recognized to identify the position of each wire in the material box image (i.e., the vertex coordinates of the target box corresponding to the wire image) and the position of the identification code affixed to the wire (i.e., the vertex coordinates of the target box corresponding to the identification code image).

[0030] In another embodiment, in order to improve the accuracy of wire and identification code recognition, the present embodiment provides a specific implementation method for identifying the position of each wire in a material box image and the position of the identification code on the wire: a virtual material box image and virtual point cloud data are acquired based on a virtual camera, and a virtual data set is established based on the virtual material box image and the virtual point cloud data; image data and point cloud data of the material box under different lighting levels and different working scenes are collected to establish a real data set; identification codes and wires are labeled for the real data set, a neural network model is trained based on the virtual data set, and transfer learning of the trained neural network model is performed based on the labeled real data set to obtain a target neural network model; the material box image is recognized based on the target neural network model to obtain the position of each wire in the material box image and the position of the identification code on the wire.

[0031] Build a virtual data set and a real data set, simulate the material box scene in the virtual engine first, and obtain a large number of virtual material box images and virtual point cloud data through the virtual camera. Use the 3D camera to collect image data and point cloud data under different lighting conditions and different scenes, and annotate the identification codes and wires in the real data set. The neural network model uses image augmentation to train the virtual data set. After the neural network model is trained, the real data set is used for transfer learning. By training the neural network model based on the virtual data set and performing transfer learning on the trained neural network model based on the real data set, the obtained neural network model can handle the target detection tasks of identification codes and wires under various lighting conditions, improving the accuracy of wire and identification code recognition.

[0032] Step S104, based on the position of each wire and the position of each identification code, the qualified wires that can be grabbed in the material box are screened.

[0033] The above-mentioned qualified position wires are wires whose wire image and corresponding identification code image (the wire and the identification code pasted on the wire) can be simultaneously visible in the material box image, that is, the wire and the identification code pasted on the wire can be completely visible in the image in the material box, fully exposing the wire that can be grasped by the robot.

[0034] The wire can be any wire with an identification code attached. In a specific embodiment, the wire can be a wire with a plastic packaging bag, see Figure 2 The schematic diagram of the material box shown in the figure shows that the wires are randomly piled up in the material box. Figure 2As shown in the enlarged image of the wire on the left, the wire is coiled and placed in a plastic packaging bag. The identification codes are uniformly pasted at fixed positions on the plastic packaging bags. The plastic packaging bags containing the wire are randomly stacked in the material box. Since the wires in the lower part of the material box will be blocked by the wires above, only the identification codes of some wires are visible. The neural network model recognizes multiple wires and identification codes. In order to screen out the fully exposed wires as qualified position wires that can be captured, the fixed relative position of the wire and the identification code can be determined based on the position where the identification code is pasted on the wire. Based on the fixed relative position of the wire and the identification code, the position of each wire and the position of each identification code, each identifiable wire is matched with the identification code pasted thereon, so that the wire and the identification code pasted thereon are completely visible, which are recorded as qualified position wires.

[0035] Step S106, based on the point cloud data corresponding to the qualified position wires, the wires at the optimal position in the material box that will not cause a grasping collision are screened out.

[0036] The placement height of each qualified wire in the bin is determined based on the point cloud data corresponding to the bin image. The wire with a higher placement height and a horizontal posture (whether the wire is placed in a horizontal posture can be determined based on the relative position of the wire's point cloud data and the bin), is easier to grasp. Since wires close to the side wall of the bin are prone to collision during grasping, the qualified wires that are close to the center of the bin, have the highest placement height and are in a horizontal posture are taken as the optimal position wires, so that the grasping robot can effectively complete the wire grasping and sorting.

[0037] Step S108, controlling the grabbing robot to preferentially grab the wire at the optimal position in the material box.

[0038] The above-mentioned grasping robot is placed in the point cloud coordinate system corresponding to the above-mentioned point cloud data, and the point cloud coordinates of the wire in the optimal position are obtained according to the image coordinate conversion of the wire in the optimal position in the above-mentioned material box image. The execution end of the grasping robot is controlled to reach the point cloud coordinates of the wire in the optimal position to grasp the wire.

[0039] In one embodiment, the execution end of the grasping robot can be a suction cup. After determining the optimal position wire in the material box, the execution end suction cup of the grasping robot is controlled to reach the position of the identification code on the wire to grasp the wire. Repeat the above steps S102 to S108 so that the optimal position wire that is easy to grasp in the material box can be continuously identified, so that the wire under the material box is continuously exposed until the grasping robot has sorted all the wires in the material box.

[0040] The above-mentioned disordered wire sorting method provided in the present embodiment can identify the corresponding relationship between the wire and the identification code by identifying the position of the wire in the material box image and the position of the identification code, thereby screening out the wire and the identification code pasted thereon and exposing the qualified position wire that can be grasped by the robot at the same time, and screening out the optimal position wire that will not cause grasping collision from the qualified position wire according to the point cloud data corresponding to the qualified position wire, so that the grasping robot can quickly grasp and sort the wire in the material box, thereby improving the degree of automation and wire sorting efficiency.

[0041] In one embodiment, in order to accurately screen out qualified position wires, this embodiment provides an implementation method for screening qualified position wires that can be grabbed in the material box based on the position of each wire and the position of each identification code, which can be specifically performed with reference to the following steps (1) to (3):

[0042] Step (1): A 2D bounding box of the wires is established in the material box image based on the position of each wire, and a 2D bounding box of the identification code is established in the material box image based on the position of each identification code.

[0043] Based on the bounding box algorithm, a two-dimensional minimum bounding box is established for each wire in the identified material box image, and the established minimum bounding box of the wire is recorded as the wire 2D bounding box. Based on the bounding box algorithm, a two-dimensional minimum bounding box is established for each identification code in the identified material box image, and the established minimum bounding box of the identification code is recorded as the identification code 2D bounding box.

[0044] Step (2): Based on the pixel distance between the center point coordinates of each wire 2D bounding box and the center point coordinates of each identification code 2D bounding box, matching wire images and identification code images are screened out.

[0045] The image in the wire 2D bounding box is recorded as the wire image, and the image in the identification code 2D bounding box is recorded as the identification code image. The center point coordinates of the wire 2D bounding box are determined according to the vertex coordinates of the wire 2D bounding box, and the center point coordinates of the identification code 2D bounding box are determined according to the vertex coordinates of the identification code 2D bounding box. The pixel distance between the center points is calculated according to the center point coordinates of each wire 2D bounding box and the center point coordinates of each identification code 2D bounding box, and it is determined whether the pixel distance satisfies the fixed relative position of the wire and the identification code. If so, the wire 2D bounding box and the identification code 2D bounding box that meet the conditions are determined to be the matching wire image and identification code image (that is, the identification code in the identification code image is pasted on the wire in the wire image).

[0046] In a specific implementation, when the pixel distance between the center point coordinates of the wire 2D bounding box and the center point coordinates of the identification code 2D bounding box satisfies a first formula, it is determined that the wire image in the wire 2D bounding box matches the identification code image in the identification code 2D bounding box; wherein the first formula is:

[0047]

[0048] is the pixel coordinate of the center point of the ith wire 2D bounding box in the bin image, is the pixel coordinate of the center point of the 2D bounding box of the jth identification code in the bin image, and pixel_distance_threshold is the preset threshold.

[0049] When the i-th wire 2D bounding box and the j-th identification code 2D bounding box satisfy the above first formula, it is determined that the i-th wire 2D bounding box matches the j-th identification code 2D bounding box, that is, the identification code in the j-th identification code 2D bounding box is pasted on the wire in the i-th wire 2D bounding box.

[0050] In a specific embodiment, the preset threshold pixel_distance_threshold is related to the size of the wire and the pasting position of the identification code on the wire. When the identification code is pasted at the center of the wire, the preset threshold value is smaller. When the center point coordinates of the wire 2D bounding box and the center point coordinates of the identification code 2D bounding box tend to coincide, it is determined that the first formula is satisfied. When the pasting position of the identification code is far away from the center of the wire, the preset threshold value is larger. For example, the value of the preset threshold pixel_distance_threshold can be 40 to 60 pixels, and the preferred value is 50 pixels.

[0051] Step (3): The screened matching wire images and wires corresponding to the identification code images are used as qualified wires that can be grabbed in the material box.

[0052] Since the screened matching wire images and identification code images are images of the same wire, when the wire and identification code of the same wire are completely visible in the material box image, it indicates that the wire is completely exposed above the material box, and the grasping robot can grasp the wire and record it as a qualified position wire.

[0053] In one embodiment, in order to accurately select the wires at the optimal position, this embodiment provides an implementation method for selecting the wires at the optimal position in the material box that will not cause a grab collision based on the point cloud data corresponding to the qualified wires. Specifically, the following steps 1) to 3) can be referred to for execution:

[0054] Step 1): Based on the point cloud data, the placement heights of the qualified position wires in the material box are sorted from large to small to obtain the placement height sorting result.

[0055] For each qualified wire that is fully exposed and can be grasped, the wire is sorted according to the point cloud coordinates of each qualified wire according to its placement height in the material box. The higher the placement height and the closer to the material box entrance, the easier it is to be grasped.

[0056] Step 2): Obtain the collision-free working convex hull area of ​​the grasping robot in the material box.

[0057] In one implementation, the coordinates of the collision-free working convex hull area in the material box input by the user are received to obtain the collision-free working convex hull area of ​​the grasping robot in the material box.

[0058] In another embodiment, a 3D bounding box of a material box is established based on point cloud data, and noise points outside the material box in the point cloud data are filtered based on the 3D bounding box of the material box; the point cloud coordinates of the execution end of the grasping robot under the point cloud data and the working range of the grasping robot are obtained to determine the collision-free working convex hull area of ​​the grasping robot in the material box. The point cloud coordinates of the material box are obtained, and the minimum bounding box of the material box is established to obtain the 3D bounding box of the material box. According to the working range of the grasping robot, the execution end of the grasping robot and the position of the 3D bounding box of the material box, the convex hull area that the grasping robot can reach without collision in the material box is determined. By calculating the convex hull area that the robot can reach without collision in the material box, the collision probability of the robot grasping wires in a deeper material box can be reduced, thereby improving the success rate of wire grasping.

[0059] Step 3): Determine whether the qualified position wire with the largest placement height is located within the collision-free working convex hull area. If so, take the qualified position wire with the largest placement height as the optimal position wire.

[0060] First, determine whether the qualified position wire with the largest placement height is located in the collision-free working convex hull area. If it is determined that the qualified position wire with the largest placement height is located in the collision-free working convex hull area, the qualified position wire with the largest placement height is used as the optimal position wire that is easiest for the grasping robot to grasp.

[0061] When the qualified wire at the highest placement height is not within the collision-free working convex hull area, determine in turn whether the qualified wires at the placement height sorting results are within the collision-free working convex hull area, until the optimal wire at the highest placement height within the collision-free working convex hull area is obtained. When the qualified wire at the highest placement height is not within the collision-free working convex hull area, determine whether the next qualified wire at the placement height sorting results is within the collision-free working convex hull area. If so, use the next qualified wire at the placement height as the optimal wire at the placement height that is easiest to grab and has no collision. If not, continue to determine whether the next qualified wire at the placement height is within the collision-free working convex hull area, until the optimal wire at the highest placement height within the collision-free working convex hull area is obtained.

[0062] The above-mentioned disordered wire sorting method provided in this embodiment can be applicable to detection under different lighting and different scenes by training the neural network model with a virtual data set generated by a virtual engine and performing migration training on the neural network model with a real data set, thereby improving the detection accuracy of the neural network model in detecting wires and identification codes. It is not affected by lighting conditions, wire types, or wire colors, has high generalization, and improves the degree of automation of wire assembly by screening out the wires in the optimal position in the material box.

[0063] Based on the above embodiment, this embodiment provides an example of automatically sorting disordered wires in a material box by applying the above disordered wire sorting method, which can be specifically performed with reference to the following steps:

[0064] The material box image and the corresponding point cloud data are collected based on the 3D vision sensor, and the 2D bounding box and 3D bounding box are established according to the size of the material box to select the optimal grab wire in the material box.

[0065] The identification code attached to the wire is a barcode, the 2D bounding box includes a wire bounding box and a barcode bounding box, and the 3D bounding box is a material box 3D bounding box, see Figure 3 The optimal grasping wire selection flow chart shown in the figure performs target detection of wires and barcodes on the material box image, and filters out the grasping samples that can be grasped in the image coordinate system of the material box image according to the pixel distance between the center points of the bounding boxes of the wires and the barcodes on the material box image (i.e., the 2D image), and records them as qualified position wires.

[0066] The point cloud data noise points outside the 3D bounding box of the material box are filtered according to the point cloud coordinates of the 3D bounding box of the material box, and the collision-free working convex hull area in the material box is calculated according to the working range of the grasping robot, the robot execution end and the 3D bounding box of the material box.

[0067] For the grasping samples that can be grasped, sort them according to the placement height on the point cloud data, and give priority to the wires with high placement positions in the material box as grasping samples. Determine whether the grasping samples are in the collision-free convex hull area. If so, use them as the optimal grasping wire (i.e., the optimal position wire). If not, return and select the next grasping sample according to the wire height sorting result until the optimal position wire with the largest placement height in the collision-free working convex hull area is obtained. Control the grasping robot to reach the optimal position wire to grasp the wire.

[0068] Corresponding to the disordered wire sorting method provided in the above embodiment, the embodiment of the present invention provides a disordered wire sorting device, see Figure 4 The schematic diagram of the structure of a disordered wire sorting device is shown, and the device includes the following modules:

[0069] The identification module 41 is used to collect the material box image and the point cloud data corresponding to the material box image, and identify the position of each wire in the material box image and the position of the identification code on the wire.

[0070] The first screening module 42 is used to screen out qualified wires that can be grabbed in the material box based on the positions of each wire and the positions of each identification code; wherein the qualified wires are wires whose wire images and corresponding identification code images can be simultaneously visible in the material box image.

[0071] The second screening module 43 is used to screen out the wires in the optimal position in the material box without causing a grabbing collision based on the point cloud data corresponding to the qualified position wires.

[0072] The sorting module 44 is used to control the grasping robot to preferentially grasp the wires at the optimal position in the material box.

[0073] The above-mentioned disordered wire sorting device provided in the present embodiment can identify the corresponding relationship between the wire and the identification code by identifying the position of the wire in the material box image and the position of the identification code, thereby screening out the wire and the identification code pasted thereon and exposing the qualified position wire that can be grasped by the robot at the same time, and by screening out the optimal position wire that will not cause grasping collision from the qualified position wire according to the point cloud data corresponding to the qualified position wire, it can be convenient for the grasping robot to quickly grasp and sort the wire in the material box, thereby improving the degree of automation and wire sorting efficiency.

[0074] In one embodiment, the first screening module 42 is used to establish a wire 2D bounding box in the material box image based on the position of each wire, and to establish an identification code 2D bounding box in the material box image based on the position of each identification code; based on the pixel distance between the center point coordinates of each wire 2D bounding box and the center point coordinates of each identification code 2D bounding box, to screen out matching wire images and identification code images; and use the wires corresponding to the screened matching wire images and identification code images as qualified position wires that can be captured in the material box.

[0075] In one embodiment, the first screening module 42 is used to determine that the wire image in the wire 2D bounding box matches the identification code image in the identification code 2D bounding box when the pixel distance between the center point coordinates of the wire 2D bounding box and the center point coordinates of the identification code 2D bounding box satisfies a first formula; wherein the first formula is:

[0076]

[0077] is the pixel coordinate of the center point of the ith wire 2D bounding box in the bin image, is the pixel coordinate of the center point of the 2D bounding box of the jth identification code in the bin image, and pixel_distance_threshold is the preset threshold.

[0078] In one embodiment, the preset threshold is related to the size of the wire and the pasting position of the identification code on the wire.

[0079] In one embodiment, the second screening module 43 is used to sort the placement heights of qualified position wires in a material box from large to small based on point cloud data to obtain a placement height sorting result; obtain the collision-free working convex hull area of ​​the grasping robot in the material box; determine whether the qualified position wire with the largest placement height is located in the collision-free working convex hull area, and if so, use the qualified position wire with the largest placement height as the optimal position wire.

[0080] In one embodiment, the second screening module 43 is used to establish a 3D bounding box of the material box based on the point cloud data, and filter out noise points outside the material box in the point cloud data based on the 3D bounding box of the material box; obtain the point cloud coordinates of the execution end of the grasping robot under the point cloud data and the working range of the grasping robot to determine the collision-free working convex hull area of ​​the grasping robot in the material box.

[0081] In one embodiment, the second screening module 43 is used to determine in sequence whether the qualified position wires in the placement height sorting results are located within the collision-free working convex hull area when the qualified position wire with the largest placement height is not within the collision-free working convex hull area, until the optimal position wire with the largest placement height within the collision-free working convex hull area is obtained.

[0082] In one embodiment, the above-mentioned recognition module 41 is used to obtain a virtual material box image and virtual point cloud data based on a virtual camera, and establish a virtual data set based on the virtual material box image and virtual point cloud data; collect image data and point cloud data of the material box under different lighting levels and different working scenes to establish a real data set; annotate the real data set with identification codes and wires, train the neural network model based on the virtual data set, and perform transfer learning on the trained neural network model based on the annotated real data set to obtain a target neural network model; identify the material box image based on the target neural network model to obtain the position of each wire in the material box image and the position of the identification code on the wire.

[0083] The above-mentioned disordered wire sorting device provided in this embodiment can be applicable to detection under different lighting and different scenes by training the neural network model with a virtual data set generated by a virtual engine and performing migration training on the neural network model with a real data set, thereby improving the detection accuracy of the neural network model in detecting wires and identification codes. It is not affected by lighting conditions, wire types, or wire colors, has high generalization, and improves the degree of automation of wire assembly by screening out the wires in the optimal position in the material box.

[0084] The implementation principle and technical effects of the device provided in this embodiment are the same as those of the aforementioned embodiments. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0085] Corresponding to the methods and devices provided in the aforementioned embodiments, an embodiment of the present invention further provides an unordered wire sorting system, characterized in that it includes: a visual sensor, a grasping robot and a controller, the controller includes a processor and a storage device; the visual sensor is used to collect material box images and point cloud data corresponding to the material box images; the grasping robot is used to grasp the wires at the optimal position in the material box; a computer program is stored on the storage device, and the computer program executes the steps of the method provided in the aforementioned embodiment when it is run by the processor.

[0086] An embodiment of the present invention provides a computer-readable medium, wherein the computer-readable medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method described in the above embodiment.

[0087] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned embodiment, and will not be repeated here.

[0088] The computer program product of the disordered wire sorting method, device and system provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0089] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0090] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0091] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0092] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for sorting disordered wires, characterized in that: include: Collecting a material box image and point cloud data corresponding to the material box image, identifying the position of each wire in the material box image and the position of the identification code on the wire; Based on the position of each of the wires and the position of each of the identification codes, the qualified wires in the material box that can be grabbed are screened out; wherein the qualified wires are wires whose wire images and corresponding identification code images can be simultaneously visible in the material box image; Filter out the wire at the optimal position in the material box that will not cause a grasping collision based on the point cloud data corresponding to the qualified position wire; Controlling the grasping robot to preferentially grasp the wire at the optimal position in the material box; The step of screening out qualified wires that can be grabbed in the material box based on the positions of the wires and the positions of the identification codes comprises: Establishing a wire 2D bounding box in the material box image based on the position of each wire, and establishing an identification code 2D bounding box in the material box image based on the position of each identification code; Based on the pixel distance between the center point coordinates of each of the wire 2D bounding boxes and the center point coordinates of each of the identification code 2D bounding boxes, filtering out matching wire images and identification code images; The screened matching wire images and wires corresponding to the identification code images are used as qualified wires that can be grabbed in the material box.

2. The method according to claim 1, characterized in that The step of screening out matching wire images and identification code images based on the pixel distance between the center point coordinates of each wire 2D bounding box and the center point coordinates of each identification code 2D bounding box comprises: When the pixel distance between the center point coordinates of the wire 2D bounding box and the center point coordinates of the identification code 2D bounding box satisfies a first formula, it is determined that the wire image in the wire 2D bounding box matches the identification code image in the identification code 2D bounding box; wherein the first formula is: is the pixel coordinate of the center point of the ith wire 2D bounding box in the bin image, is the pixel coordinate of the center point of the j-th identification code 2D bounding box in the material box image, and pixel_distance_threshold is a preset threshold.

3. The method according to claim 2, characterized in that The preset threshold is related to the size of the wire and the pasting position of the identification code on the wire.

4. The method according to claim 1, characterized in that: The step of selecting the wire at the optimal position in the material box that will not cause a grasping collision based on the point cloud data corresponding to the qualified position wire includes: Based on the point cloud data, the placement heights of the qualified position wires in the material box are sorted from large to small to obtain a placement height sorting result; Obtaining a collision-free working convex hull area of ​​the grasping robot in the material box; It is determined whether the qualified position wire with the largest placement height is located within the collision-free working convex hull area. If so, the qualified position wire with the largest placement height is used as the optimal position wire.

5. The method according to claim 4, characterized in that Also includes: Establishing a 3D bounding box of the material box based on the point cloud data, and filtering noise points outside the material box in the point cloud data based on the 3D bounding box of the material box; The point cloud coordinates of the execution end of the grasping robot under the point cloud data and the working range of the grasping robot are obtained to determine the collision-free working convex hull area of ​​the grasping robot in the material box.

6. The method according to claim 4, characterized in that Also includes: When the qualified position wire with the largest placement height is not within the collision-free working convex hull area, determine in turn whether the qualified position wires in the placement height sorting results are within the collision-free working convex hull area, until the optimal position wire with the largest placement height within the collision-free working convex hull area is obtained.

7. The method according to claim 1, characterized in that The step of identifying the position of each wire in the material box image and the position of the identification code on the wire includes: Acquire a virtual material box image and virtual point cloud data based on a virtual camera, and establish a virtual data set based on the virtual material box image and the virtual point cloud data; Collect image data and point cloud data of the material box under different lighting levels and different working scenes to establish a real data set; Performing identification code labeling and wire material labeling on the real data set, training a neural network model based on the virtual data set, and performing transfer learning on the trained neural network model based on the labeled real data set to obtain a target neural network model; The material box image is recognized based on the target neural network model to obtain the position of each wire in the material box image and the position of the identification code on the wire.

8. A disordered wire sorting device, characterized in that: include: An identification module, used to collect a material box image and point cloud data corresponding to the material box image, and identify the position of each wire in the material box image and the position of the identification code on the wire; A first screening module is used to screen out qualified position wires that can be grabbed in the material box based on the position of each wire and the position of each identification code; wherein the qualified position wires are wires whose wire images and corresponding identification code images can be simultaneously visible in the material box image; A second screening module is used to screen out the wires in the optimal position in the material box that will not cause a grasping collision based on the point cloud data corresponding to the qualified position wires; A sorting module, used for controlling the grabbing robot to preferentially grab the wires at the optimal position in the material box; The second screening module is used to establish a wire 2D bounding box in the material box image based on the position of each wire, and to establish an identification code 2D bounding box in the material box image based on the position of each identification code; based on the pixel distance between the center point coordinates of each wire 2D bounding box and the center point coordinates of each identification code 2D bounding box, screen out matching wire images and identification code images; and use the wires corresponding to the screened matching wire images and identification code images as qualified position wires that can be captured in the material box.

9. A disordered wire sorting system, characterized in that: include: A visual sensor, a grasping robot and a controller, wherein the controller includes a processor and a storage device; The visual sensor is used to collect the material box image and the point cloud data corresponding to the material box image; The grabbing robot is used to grab the wire at the optimal position in the material box; The storage device stores a computer program, which, when executed by the processor, executes the method according to any one of claims 1 to 7.

Images