An object separator based on a 2D camera and an object separation method

Abstract

The present invention discloses an object separator based on a 2D camera, comprising: an object conveyor belt arranged at the lower part, a plurality of 2D cameras at different heights and / or angles arranged above the object conveyor belt, a data processor connected to the plurality of 2D cameras, and a conveyor belt control system. The present invention also discloses a method for separating objects based on a 2D camera, comprising: a preprocessing part and an on-line processing part, wherein the preprocessing part includes a camera erection step and a camera calibration step, and the on-line processing part includes a corner point detection step of an object, a same object recognition step across cameras, and an object size and position calculation step, realizing the single-piece separation of fully automated multi-side-by-side objects, improving the object sorting efficiency, reducing the erection cost of visual devices in the single-piece separation system, and solving the problems of high hardware cost of the visual system in the current single-piece separation system, inaccurate object positioning, and low practicability.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and particularly to an object separator based on a 2D camera and an object separation method. Background Art

[0002] In recent years, vision detection technology has been widely applied in the field of object separation, including automatic object handling, sorting, automatic identification and grasping in logistics warehousing, etc. Object sorting is a core link in e-commerce. With the rapid development of China's e-commerce platforms and the increasing speed of transportation, the business volume of the logistics industry has increased sharply. In order to improve the processing efficiency of objects, in the logistics sorting at each transfer station and express delivery point, it is usually necessary to drive conveyor belts and other conveying devices with a power source to transport a large number of package stacks to different positions for single-piece separation, so as to facilitate subsequent scanning, identification and sorting of objects.

[0003] First, at present, the sorting operation sites in the e-commerce and express delivery industries usually adopt the method of manual separation to separate a large number of incoming objects. The task volume is heavy, the labor intensity is high, and the efficiency is not high.

[0004] Second, some researchers have proposed single-object separation methods and systems based on computer vision technology to replace manual single-object separation. However, when obtaining the position information of objects, most of these methods require additional hardware devices to assist in addition to the image acquisition device.

[0005] The prior art, such as the technical solution disclosed in Chinese Patent Application No. 201910162842.X, requires multiple photoelectric sensors to assist in addition to camera installation. The camera is used to judge whether there is a side-by-side phenomenon of packages, and then multiple photoelectric sensors are used to locate the side-by-side packages, and a speed difference is given to different conveyor belts under the side-by-side packages to achieve the effect of package separation.

[0006] The prior art has high requirements for the layout position of photoelectric sensors, complex layout, inaccurate positioning, and difficulty in dealing with the situation of multiple packages side by side.

[0007] Third, some people have also tried to only use a vision acquisition device to locate objects. Most of these methods use multiple 3D cameras to splice the object images collected by multiple 3D cameras to cover the entire field of view. After positioning the objects in the field of view, the position information is sent to the separation device and the side-by-side device to complete single-piece separation. In actual engineering applications, it is necessary to obtain high-quality object images at a conveyor belt speed of 0-2 m / s. Therefore, the requirements for parameters such as the transmission interface of the camera, camera delay, image resolution, and frames per second transmitted are relatively high. 3D cameras that meet the requirements are expensive, and the field of view blind area is large when a single 3D camera collects images. It is necessary to deploy a vision device with multiple 3D cameras, and the cost is high, which is not conducive to large-scale installation on the production line.

[0008] Fourth, some people have also tried to further study by combining the second and third technologies. For example, Chinese Patent Application No. 201911168246.9 discloses a side-by-side wrapping separation device and method, including: a wrapping dispersion device that first conveys the packages to the dispersion device. The packages reaching the dispersion device from the conveying line are dispersed by the angular difference in the conveying direction, leaving a gap between adjacent packages. Subsequently, it is a detection and recognition device. The 3D camera acquires the package images, stitches the images captured by 4 3D cameras, removes the overlapping parts in the photos through visual algorithms, obtains the package images in the entire field of view, and establishes a coordinate system in the image. The edge detection algorithm is used to extract the coordinate information of the package contours in the image. According to the relative positions, sizes and other information of each package, the first package at the front is selected. The belt where the first package is located is the first belt set, and the remaining packages are the second packages. The belt where the second packages are located is the second belt set. The first belt set is adjusted to run faster, and the second belt set is adjusted to run slower or stop through the belt differential control module to achieve the effect of parallel separation of packages. The packages after parallel separation are conveyed to the single-piece separation module to achieve the side-by-side separation of packages.

[0009] However, the existing technology needs to use multiple 3D cameras to stitch and obtain the complete field of view size. Multiple 3D cameras are expensive. Using image stitching to remove the overlapping parts of each camera's photos wastes part of the camera's field of view. In addition, currently, image stitching will cause alignment deviation in the overlapping area, which affects the output of the package position. Moreover, general depth cameras have poor detection effects on short packages, are greatly affected by external environments such as light, have poor robustness, and the installation conditions are relatively harsh, which are not suitable for large-scale installation on the assembly line. Also, because the edge detection algorithm is used to obtain the package position, the traditional edge detection algorithm has low accuracy, which will cause errors in the output of the package position. The existing method has complex processing for parallel packages, takes a large amount of time, and it is easy to stack packages when processing a large number of parallel packages in a timely manner, and has a poor separation effect on packages with similar sizes. The above problems lead to a decrease in the practicality of this method.

[0010] Therefore, there is currently no set of efficient and high-separation-precision package separation machine and package separation method. Summary of the Invention

[0011] In order to solve the problems existing in the prior art, the present invention provides an object separation machine and an object separation method based on a 2D camera as follows, to replace manual labor to achieve efficient single-piece separation of objects. Through the rotation target detection technology, the objects conveyed on the conveyor belt are detected, the real-time positions of the objects are obtained, and finally the separation belt machine realizes the separation, distance pulling and queuing of the objects. It realizes the single-piece separation of fully automated multi-side-by-side objects, improves the efficiency of object sorting and detection, reduces the installation cost of the vision device in the single-piece separation system, and solves the problems of high hardware cost of the vision system in the current single-piece separation system, inaccurate object positioning and low practicality.

[0012] On one hand, the present invention provides an object separator based on a 2D camera, comprising:

[0013] An object conveyor belt arranged at the lower part, a plurality of 2D cameras at different heights and / or angles arranged above the object conveyor belt, a data processor connected to the plurality of 2D cameras, and a conveyor belt control system; wherein, the 2D cameras are erected at a certain height difference and in a regular manner that can cover the entire field of view size of a single object; the data processor acquires multiple images respectively collected by multiple 2D cameras, associates and matches the objects between two 2D cameras with a height difference by using the feature information of the objects in the collected multiple images; calculates the true size of the object by using the camera imaging principle for the imaging sizes of the same object in cameras at different heights, and outputs accurate object position information; the conveyor belt control system is used to receive the position information of the object and control the object conveyor belt to separate the objects.

[0014] According to a second aspect of the present invention, there is provided a method for separating objects based on a 2D camera, comprising:

[0015] A preprocessing part and an on-line processing part, wherein the preprocessing part includes a camera erection step and a camera calibration step, and the on-line processing part includes an object corner point detection step, a same object recognition step across cameras, and an actual space size and position calculation step of the object;

[0016] The camera erection step is used to erect multiple 2D cameras. The camera calibration step is used to calibrate the multiple 2D cameras to obtain the mapping relationship between the pixel image coordinate systems of the object images of different cameras. The object corner point detection step is used to obtain the corner point information of the object and further obtain the image information of the object. The same object recognition across cameras uses a re-identification algorithm to accurately identify multiple objects in the associated matching area, and finally assigns the same number to the same object in different images. The actual space size and position calculation of the object uses the difference in the image positions of the same object in two cameras with a certain height to correct the error caused by the perspective effect of the object imaging (the object appears smaller when farther away and larger when closer), and then calculates the actual space size and position information of the object, and outputs it to the conveyor belt control system to realize the single-piece separation of the objects.

[0017] Preferably, the camera erection step includes:

[0018] Set up three 2D cameras, namely the entrance camera, the midpoint camera, and the exit camera. Among them, the entrance camera is installed directly above the entrance of the object conveyor belt, with a height of a, and the optical axis of the entrance camera is placed at an angle to the vertical direction; the midpoint camera is installed directly above the midpoint of the field of view size, with a height of b, and the optical axis of the midpoint camera is placed vertically and overlooking the plane of the object conveyor belt; the exit camera is installed directly above the exit, with a height of a, and the optical axis of the exit camera is placed at an angle to the vertical direction.

[0019] Preferably, the camera calibration steps include: establishing a pixel coordinate system for the images captured by two cameras. When the height of the actual object is determined, this mapping relationship remains unchanged at any position at this height. Two calibrations at different heights are performed between the 2D cameras at different heights to obtain two mapping relationships, so as to obtain the positions of objects at any height between different heights; the camera calibration includes coordinate transformation of the images formed by the 2D cameras with a height difference in the same actual space field of view area, obtaining the pixel image coordinates of the same actual space position coordinates between different cameras, and finally obtaining the pixel point mapping relationship of the two 2D cameras to the same actual space coordinate, which is regarded as completing one camera calibration; through the aforementioned camera calibration steps, two calibrations are completed, including the mapping relationship of pixel points between the 2D cameras between the 0-height plane and the maximum-height plane. The 0-height mapping relationship is regarded as the first mapping relationship, and the maximum-height plane mapping relationship is regarded as the second mapping relationship; at the same time, an image coordinate system is established with the upper left corner of the 2D camera as the origin, and then an imaging plane coordinate system is established according to the field of view size covered by the camera and the camera resolution to obtain the corresponding relationship between the pixel point length and the actual physical length of the object.

[0020] Preferably, the corner detection steps of the object include: adding a rotation factor to the conventional target detection algorithm to accurately detect irregularly placed objects and special objects such as shaped parts, and finally obtaining the coordinates of multiple corner points on the upper surface of the object and the position of the object in the image. The rotation factor includes using rotation yolov5, R 2 CNN, RRPN, Glidingvertex, P-RSDet or SCRDet rotation target detection models or lightweight deep learning detection models to implement the rotation target test.

[0021] Preferably, the cross-camera same object recognition steps include: setting an associated matching area and object re-identification.

[0022] Preferably, the setting of the associated matching area includes:

[0023] Obtain the detection box result of the first object image of the midpoint camera;

[0024] Obtain the first association box of the second object image of the entrance camera or the exit camera under the first mapping relationship calibrated at the 0-height plane and the second association box of the second object image of the entrance camera or the exit camera under the second mapping relationship calibrated at the maximum height plane according to the first mapping relationship and the second mapping relationship respectively obtained from the calibration between the midpoint camera and the entrance camera or the exit camera in the camera installation strategy and the camera calibration strategy;

[0025] Frame all the areas between the first association box and the second association box and set them as the association matching area;

[0026] Repeat the above steps to obtain the detection box results of the midpoint camera for each object, and calculate the association matching area of the corresponding object in the entrance camera or the exit camera.

[0027] Preferably, the object re-identification includes: obtaining a re-identification algorithm model after training the object image data, and using the surface feature information of the object to perform association matching on the association matching area of the first object image of each object obtained by the midpoint camera in the entrance camera or the exit camera, to obtain the identification of the same object across cameras, and output the same number of the same object across cameras.

[0028] Preferably, the steps of calculating the actual spatial size and position of the object utilize the differences in the image positions of the same object in two cameras with a certain height difference to correct the error caused by the imaging of the object being smaller in the distance and larger nearby, and then calculate the actual spatial size and position information of the object. According to the actual spatial size and position information of the object, the single-piece separation of the object is performed, including:

[0029] Calculate the different pixel lengths and widths of the same object imaged as smaller in the distance and larger nearby in the midpoint camera and the entrance camera or the exit camera respectively according to the multiple corner coordinates of the same object in the midpoint camera and the multiple corner coordinates of the entrance camera or the exit camera;

[0030] Calculate the true size and height information of each object according to the pinhole imaging principle of the camera, determine the true coordinates of each object in the imaging plane coordinate system to eliminate the error, and determine the true size of each object and convert it into the coordinates of the imaging plane coordinate system.

[0031] Preferably, calculating the position information of the object includes:

[0032] Taking the midpoint camera as the origin, taking the imaging plane coordinate system as the X and Y axes, and taking the vertical direction from the midpoint camera to the calibrated 0-height plane as the Z axis, establish a camera coordinate system;

[0033] Taking the origin of the camera coordinate system as the origin of the world coordinate system, a world coordinate system is established. At this time, the camera coordinate system is equal to the world coordinate system;

[0034] Convert the coordinates of the object in the imaging plane coordinate system and the camera focal length parameters in the world coordinate system, and calculate to obtain the corresponding coordinates in the world coordinate system.

[0035] The object separator and object separation method provided by the present invention have the following beneficial effects:

[0036] According to the characteristics that the same object has different imaging sizes for a 2D camera with a height difference, the actual size of the object is calculated based on the pinhole imaging principle of the camera, and then the world coordinate of the object is obtained. This coordinate is input into the single-piece separation module to achieve quasi-real-time and high-precision single-piece separation. This method and system for single-piece separation of objects by multiple 2D cameras based on corner detection can use 2D cameras to replace 3D cameras to detect the actual size of objects and output the world coordinates of the objects, solving the problems of high equipment costs in engineering applications and poor detection effects of 3D cameras on short objects. The camera installation of the present invention is relatively simple, and the multi-angle lens can effectively reduce the occlusion blind area between objects and improve the detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic structural diagram of a preferred embodiment of the object separator of the present invention.

[0038] Figure 2 It is a schematic diagram of the principle of the layout method of the preferred embodiment of the entrance camera in the object separator of the present invention.

[0039] Figure 3 It is a schematic diagram of the principle of the layout method of the preferred embodiment of the midpoint camera in the object separator of the present invention.

[0040] Figure 4 It is a schematic diagram of the principle of the layout method of the preferred embodiment of the exit camera in the object separator of the present invention.

[0041] Figure 5 It is a flowchart of a preferred embodiment of the object single-piece separation method of the present invention.

[0042] FIG. 6(a) is a schematic diagram of the traditional camera setting method in the preferred embodiment of the object separation method of the present invention.

[0043] FIG. 6(b) is a schematic diagram of the preferred camera setting method in the preferred embodiment of the object separation method of the present invention.

[0044] FIG. 6(c) shows the change in the blind area caused by the stacking of multiple objects when the object is moving.

[0045] Figure 7Schematic diagram of camera calibration in the preferred embodiment of the object separation method described in the present invention.

[0046] Figure 8 Schematic diagram of the camera calibration method in the preferred embodiment of the object separation method described in the present invention.

[0047] Figure 9 Schematic diagram of the camera calibration method process in the preferred embodiment of the object separation method described in the present invention.

[0048] Figure 10(a) is a schematic diagram of the principle of object corner points determined by the non-rotation target detection algorithm in the preferred embodiment of the object separation method described in the present invention.

[0049] Figure 10(b) is a schematic diagram of the principle of object corner points determined by the rotation target detection algorithm in the preferred embodiment of the object separation method described in the present invention.

[0050] Figure 11 Schematic diagram of the working principle of the object corner point detection module of the rotation target detection algorithm for implementing objects in the preferred embodiment of the object separation method described in the present invention.

[0051] Figure 12 The flowchart of the rotation target detection algorithm for objects in the preferred embodiment of the object separation method described in the present invention is shown.

[0052] Figure 13 Schematic diagram of the principle of identifying the same object across cameras in the preferred embodiment of the object separation method described in the present invention is shown.

[0053] Figure 14 The flowchart of the steps for identifying the same object across cameras in the preferred embodiment of the object separation method described in the present invention is shown.

[0054] Figure 15 Schematic diagram of the principle of object re-identification in the preferred embodiment of the object separation method described in the present invention is shown.

[0055] Figure 16 Schematic diagram of the principle of the ReID algorithm for object re-identification to identify the same target across cameras in the preferred embodiment of the object separation method described in the present invention is shown.

[0056] Figure 17 The flowchart of calculating the actual size and position of the object in the preferred embodiment of the object separation method described in the present invention is shown. Detailed implementation mode

[0057] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in conjunction with the accompanying drawings of the specification and specific implementation modes.

[0058] The method provided by the present invention can be implemented in the following terminal environment, and the terminal may include one or more of the following components: a processor, a memory, and a display screen. Among them, at least one instruction is stored in the memory, and the instruction is loaded and executed by the processor to implement the method described in the following embodiments.

[0059] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts within the entire terminal, and by running or executing instructions, programs, codes, code sets, or instruction sets stored in the memory, as well as calling data stored in the memory, it executes various functions of the terminal and processes data.

[0060] The memory may include a Random Access Memory (RAM), and may also include a Read-Only Memory (ROM). The memory can be used to store instructions, programs, codes, code sets, or instruction sets.

[0061] The display screen is used to display the user interfaces of various applications.

[0062] In addition, those skilled in the art can understand that the structure of the above terminal does not constitute a limitation on the terminal. The terminal may include more or fewer components, or combine certain components, or have different component arrangements. For example, the terminal may also include components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, and a power supply, which will not be elaborated here.

[0063] It should be noted that the ultimate technical purpose of this embodiment is to output the world coordinate system coordinates of each package. However, the first and second package image information initially collected by the 2D camera first establishes a pixel image coordinate system. Therefore, there is a coordinate system conversion process. The specific conversion process is: pixel image coordinate system → imaging plane coordinate system → (camera) camera coordinate system → world coordinate system.

[0064] Among them:

[0065] Pixel image coordinate system (u-v): A rectangular coordinate system with the upper left corner of the image as the origin and pixels as the unit. u and v represent the row and column numbers of the pixels.

[0066] Imaging plane coordinate system (o-xy): A two-dimensional coordinate system with the geometric center O of the image as the origin, and the x and y axes are parallel to the edge lines of the image frame respectively. The unit is the physical size corresponding to each pixel point in the actual space range of the image, such as 1 millimeter / pixel, and x and y represent the physical distances from the origin in the horizontal and vertical directions.

[0067] (Camera) Camera coordinate system (o e -x e y ez e ):With the optical center of the camera or camera as the origin, the z-axis coincides with the optical axis, that is, the z-axis points in the front direction of the camera (i.e., perpendicular to the imaging plane), and the positive directions of the x-axis and y-axis are parallel to the imaging plane coordinate system.

[0068] World coordinate system (o w -x w y w z w ):Since the camera can be placed at any position in the environment, a reference coordinate system is selected in the environment to describe the position of the camera and to describe the position of any object in the environment. This coordinate system is called the world coordinate system.

[0069] As Figure 1 shown, this preferred embodiment takes the logistics application as the scenario, and correspondingly provides a 2D camera parcel separator, including:

[0070] A logistics conveyor belt is provided at the lower part of the separator, and a plurality of 2D cameras with different heights and / or angles are provided above the logistics conveyor belt, a data processor connected to the plurality of 2D cameras, and a belt control system. Among them, multiple 2D cameras are installed at a certain height difference and in a regular manner that can cover the entire field of view of a single parcel, reducing the blind area of the field of view caused by occlusion between parcels; the data processor acquires multiple images respectively collected by the multiple 2D cameras, and uses the characteristic information of the parcels to associate and match the logistics parcels between the two 2D cameras with a height difference; for the imaging of the same parcel by cameras at different heights, the true size of the logistics parcel is calculated using the camera imaging principle, and the accurate parcel position information is output. The conveyor belt control system is used to receive the position information of the parcel and control the logistics conveyor belt to separate the parcels.

[0071] Obtaining the true size and position information of the parcel: The calculation of the parcel size and position uses the difference in the image positions of the same parcel in two cameras with a certain height to correct the error caused by the perspective imaging of the parcel being smaller when far and larger when near, and then calculates the actual size of the parcel and converts it into the world coordinate system.

[0072] Function of the belt control system: Output the position information of the parcel in the world coordinate system to the belt control system to achieve single-piece separation of the parcels.

[0073] The 2D camera forms a certain field of view size and monitoring range on the logistics conveyor belt, where the field of view size represents the area of the belt control system on the logistics conveyor belt, and the monitoring range is the area photographed by the 2D camera.

[0074] The logistics conveyor belt is of N×M specification, which is 4×7 specification in this embodiment.

[0075] The camera setup strategy includes: at least two 2D cameras with a certain height difference, mounted above the logistics package conveyor, with a downward or oblique view; the 2D cameras have complementary fields of view, covering the entire area where package location information needs to be output;

[0076] More preferably, there are three 2D cameras, namely an entrance camera 1 , a midpoint camera 2 and an exit camera 3 .

[0077] like Figure 2 As shown, in this embodiment, the entrance camera 1 is installed directly above the entrance of the logistics conveyor belt, with a height of 1.7m, and the optical axis of the entrance camera 1 is tilted 35° from the vertical direction.

[0078] like Figure 3 As shown, the midpoint camera 2 is installed just above the midpoint of the field of view, at a height of 1.9 m. The optical axis of the midpoint camera 2 is consistent with the vertical direction and is placed in a downward direction.

[0079] like Figure 4 As shown, the exit camera 3 is installed directly above the exit, and the installation height is the same as that of the entrance camera, which is 1.7m. The optical axis of the exit camera 3 is tilted at -35° to the vertical direction.

[0080] As a preferred embodiment, it also includes a single-piece separation module composed of a side device. The belt control system will control the belts in different areas to run at different speeds for parallel separation. After separation, the packages with intervals in a single queue are output through the side device.

[0081] As a preferred embodiment, it also includes an adhesion separation device, which is composed of a scattering belt mechanism and is used to pull a certain gap between multiple logistics packages to separate the adhered packages.

[0082] like Figure 5 As shown, another aspect of the present invention is to provide a package separation method based on a 2D camera, comprising:

[0083] Preprocessing and online processing: The preprocessing part includes camera setup and calibration, while the online processing part includes package corner detection, cross-camera identification of identical packages, and calculation of the package's actual spatial size and position.

[0084] The camera setup steps are used to set up multiple 2D cameras. The camera calibration steps are used to calibrate multiple 2D cameras to obtain the mapping relationship between the pixel image coordinate systems of different cameras in the wrapped image. The corner detection steps of the package are used to obtain the corner information of the package and further obtain the image information of the package. The cross-camera same package recognition steps use a re-identification algorithm to accurately identify multiple packages in the associated matching area, and finally assign the same number to the same package in different images. The actual spatial dimensions and positions of the package are calculated using the differences in the positions of the same package in the images of two cameras with a certain height, correct the error caused by the perspective imaging of the package being smaller in the distance and larger nearby, and then calculate the actual spatial dimensions and position information of the package, and output it to the belt control system to achieve single-piece separation of the packages.

[0085] Among them, the camera setup steps include: setting three 2D cameras, namely the entrance camera 1, the midpoint camera 2, and the exit camera 3; among them, the entrance camera 1 is set directly above the entrance of the logistics conveyor belt, with a height of a, and the optical axis of the entrance camera 1 is placed at an angle β with the vertical direction; the midpoint camera 2 is set directly above the midpoint of the field of view size, with a height of b, and the optical axis of the midpoint camera 2 is consistent with the vertical direction and is placed in a top view; the exit camera 3 is set directly above the exit, with a height of a, and the optical axis of the exit camera 3 is placed at an angle -β with the vertical direction. In this embodiment, b is 1.9m and a is 1.7m. Of course, those skilled in the art can adjust the height values accordingly according to the architectural characteristics of the logistics site, which are all within the protection scope of the present invention.

[0086] In this embodiment, a package with a maximum height of 70cm is selected. In this embodiment, when the tallest package just enters the 1.6m×2m field of view size, the three cameras can also obtain a complete package image.

[0087] As shown in Figures 6(a) and 6(b), they are schematic comparison diagrams of whether a complete package image can be obtained under the traditional camera setup method and the camera setup method of this embodiment.

[0088] As shown in Figure 6(a), under the traditional camera setup method, when the camera monitoring range can only cover the 2m×1.6m field of view size of the 0 plane, a complete image of the edge package cannot be obtained.

[0089] As shown in Figure 6(b), in the camera setup method of this embodiment, when the camera monitoring range can cover the 2m×1.6m field of view size of the 70cm plane of the tallest package, a complete image of the edge package can be obtained.

[0090] As shown in Table 1, there will be a field of view blind area when the camera captures images. The blind area refers to the area that can never be observed by two cameras during the movement of the package on the conveyor belt.

[0091] Table 1 Classification of possible field of view blind areas

[0092]

[0093] As can be seen from Table 1, the possible blind spots in the field of view include two categories:

[0094] The first category is the blind spot caused by a single package. When only the midpoint camera exists and the package is located at the midpoint, the single-package blind spot is as shown in the schematic diagram Figure 1 shown above. In this embodiment, the left blind spot A and the upper blind spot B can be observed by the entrance camera, and the right blind spot C and the lower blind spot D can be observed by the exit camera. When the package is at different traveling positions, the midpoint camera can observe areas A, B, C, and D respectively. Therefore, there is no blind spot caused by a single package in the embodiment of the present invention.

[0095] The second category is the area change of the multi-package blind spot during movement. The specific situation is shown in Table 2.

[0096] Table 2 Area change of the blind spot caused by multi-package stacking during package movement

[0097]

[0098] For the specific blind spot area, please refer to Figure 6(c).

[0099] As a preferred embodiment, when the package moves on the conveyor belt, the blind spots caused by multi-packages include the types of parallel package occlusion and "high-low-high" package occlusion. Among them, the blind spot can only be captured by the midpoint camera 2. If a package is detected in this area in the image of camera 2, it is given a small empirical height (5 cm in this embodiment) to eliminate the perspective error.

[0100] As Figure 7 shown, the purpose of camera calibration: Since the position and size of the same object are different in different cameras, calibration will obtain the mapping relationship between pixel points of different cameras, so as to calculate the position of the same package in the images of different cameras in the subsequent process.

[0101] As Figure 8 shown, the method of camera calibration: Establish a pixel image coordinate system for the images captured by the two cameras. When the height of the actual object is determined, this mapping relationship remains unchanged at any position at this height. Two calibrations are required between the two cameras in this solution, namely the 70 cm height calibration and the 0 cm height calibration. The purpose is to use the mapping relationships obtained from the 70 cm and 0 cm calibrations to find the positions of packages at any height between 0 and 70 cm.

[0102] The camera calibration strategy includes: establishing a pixel coordinate system for the images captured by two cameras. When the height of the actual object is determined, this mapping relationship remains unchanged at any position at this height. Two calibrations at different heights are performed between the 2D cameras at different heights to obtain two mapping relationships, so as to obtain the positions of objects at any height between different heights; the camera calibration steps include coordinate transformation of the images formed by the 2D cameras with a height difference in the same actual space visual field area, obtaining the pixel image coordinates of the same actual space position coordinates between different cameras, and finally obtaining the pixel point mapping relationship of the two 2D cameras to the same actual space coordinate, which is regarded as completing one camera calibration; the method completes two camera calibrations, including the mapping relationship of pixel points between the 2D cameras between the 0-height plane and the maximum-height plane. The 0-height mapping relationship is regarded as the first mapping relationship, and the maximum-height plane mapping relationship is regarded as the second mapping relationship; at the same time, an image coordinate system is established with the upper left corner of the 2D camera as the origin, and then an imaging plane coordinate system is established according to the visual field size covered by the camera and the camera resolution to obtain the corresponding relationship between the pixel point length and the actual physical length of the object.

[0103] See Figure 9 , in this embodiment, the process of camera calibration includes:

[0104] 1. Place a 70-cm standard cardboard box within the visual field so that all three cameras can capture the box;

[0105] 2. Respectively obtain the pixel point coordinates of the eight corner points on the upper and lower surfaces of the standard cardboard box in the images captured by Camera 1 and Camera 2;

[0106] 3. Perform two operations on the coordinate values of the upper surface and the lower surface respectively to obtain the mapping relationship H1 at the 70-cm height and the mapping relationship H2 at the 0-cm height between the two cameras;

[0107] 4. Replace it with an uncalibrated camera, and the three-camera calibration work can be completed after the above calibration steps are performed three times.

[0108] As shown in Figure 10(a), the existing target detection algorithm can only obtain the horizontal maximum circumscribed rectangle of the target, and cannot accurately obtain the actual size of each package, and there is a problem of overlapping detection frames, resulting in ineffective single-piece separation.

[0109] See Figure 10(b). As a preferred implementation, the corner point detection steps of the package include: adding a rotation factor to the conventional target detection algorithm, and being able to accurately detect special packages such as irregularly placed packages and shaped parts, and finally obtaining the coordinates of the four corner points on the upper surface of the package and the position of the package in the image. This embodiment uses rotated yolo5, R 2A rotational object detection model of CNN, RRPN, Gliding vertex, P-RSDet or SCRDet. Apply the wrapped corner detection step to detect the packages in the images captured by three cameras respectively, and obtain the accurate detection frame of each package.

[0110] Figure 11 It is the working principle diagram of the package corner detection module for implementing the package corner detection step. This module obtains the size and position of the package in different camera images for subsequent calculation of the actual position of the package.

[0111] Figure 12 The following is the flowchart of the package corner detection step, including:

[0112] Use the camera calling strategy of the midpoint camera 2 in cooperation with other cameras to judge the actual position where the package is located, including:

[0113] If the package is located in the left half of the entire field of view size, as Figure 11 shown by package 1, then call the entrance camera 1 and the midpoint camera 2;

[0114] If the package is located in the middle part of the entire field of view size, as Figure 11 shown by package 2, then call the entrance camera 1 and the midpoint camera 2, and the exit camera 3 and the midpoint camera 2 respectively;

[0115] If the package is located in the right half of the entire field of view size, as Figure 11 shown by package 3, then call the exit camera 3 and the midpoint camera 2;

[0116] Then obtain the position information of the packages with the same number in different camera images.

[0117] Most of the packages transported on the conveyor belt are solid color yellow boxes with a single feature. If the traditional global image matching method is used, it is very difficult to find effective feature points of different package targets for correlation matching, so it is very difficult to realize the recognition of the same package between different cameras. Therefore, the following method is used to set the correlation matching area:

[0118] Obtain the detection frame result of the first package image of the midpoint camera 2;

[0119] According to the first mapping relationship and the second mapping relationship respectively obtained by the calibration between the midpoint camera 2 and the entrance camera 1 or the exit camera 3 in the 0-height plane and the maximum height plane in the camera erection strategy and the camera calibration strategy, obtain the first associated frame of the second package image under the first mapping relationship of the 0-height calibration of the entrance camera 1 or the exit camera 3 and the second associated frame of the second package image under the second mapping relationship of the maximum height plane calibration;

[0120] Frame all the areas between the first associated box and the second associated box and set them as the associated matching area;

[0121] Repeat the above steps to obtain the detection box results of the midpoint camera 2 for each package, and calculate the associated matching area of the corresponding package in the entrance camera 1 or the exit camera 3;

[0122] After training the package image data, obtain a re-identification algorithm model. Use the surface feature information of the logistics package to perform associated matching on the associated matching area of the first package image of each package obtained by the midpoint camera 2 in the entrance camera 1 or the exit camera 3, obtain the identification of the same package across cameras, and output the same number of the package across cameras.

[0123] Figure 13 As shown in the schematic diagram of the same package identification across cameras. As a preferred implementation, the principle of the same package identification step across cameras is: adopt a local matching method for the same package identification. For example, when a package is detected by the midpoint camera 2, obtain a certain area of the image in the entrance camera 1 through the mapping relationship output by camera calibration, and search for and identify the same package in this area, so as to achieve the purpose of the same package identification.

[0124] Figure 14 As shown in the flowchart of the same package identification across cameras. As a preferred implementation, it includes:

[0125] 1. Obtain the mapping relationship H1 of the 70 cm plane and the mapping relationship H2 of the 0 cm plane output in the camera calibration step, and the image pixel coordinates of the four corner points of the top surface of the package captured by the midpoint camera 2 in the midpoint camera 2 output by the corner point detection of the package in the corner point detection step of the package;

[0126] 2. Obtain the image area R1 of the package in the entrance camera 1 or the image area R1' in the exit camera 3 according to the mapping relationship H1;

[0127] Obtain the image area R2 of the package in the entrance camera 1 or the image area R2' in the exit camera 3 according to the mapping relationship H2;

[0128] Obtain the union of area R1 and R2 or the union of R1' and R2', so as to obtain the final associated matching area;

[0129] 3. Use the re-identification algorithm to accurately identify multiple packages in the associated matching area, and finally assign the same number to the same package in different images, as Figure 15 shown.

[0130] The error caused by global matching can be greatly reduced by associating matching areas. However, the larger the package, the greater the error of the associated matching area. Sometimes, one large package and multiple small packages may appear in the associated matching area, making accurate identification impossible. Therefore, we use Figure 16 The ReID algorithm shown in the figure can identify the same target across cameras.

[0131] Directly calculating the parcel's position in the 2D image, when converted to the world coordinate system for output, will result in errors in the output of the parcel's position due to the "distant smaller, near larger" imaging. Therefore, the positional differences between the images of the same parcel taken from two cameras at a certain height difference are used to correct for this error. The actual dimensions of the parcel are then calculated and converted to the world coordinate system for output to the belt control system to separate the parcels individually.

[0132] The process of calculating the package size and location is as follows Figure 17 As shown, the different pixel lengths and widths of the same package imaged in the midpoint camera 2 and the entrance camera 1 or exit camera 3 due to the different distances and nearness are calculated based on the corner point coordinates of the same package in the midpoint camera 2 and the entrance camera 1 or exit camera 3 respectively; the true size and height information of a single package are calculated based on the camera pinhole imaging principle, and the true coordinates of the single package in the imaging plane coordinate system are determined to eliminate errors. The true size of the package is determined and converted into coordinates in the imaging plane coordinate system.

[0133] The process includes: taking the position of the same-numbered package in the image of midpoint camera 2, the position of the same-numbered package in the image of entrance camera 1 (or exit camera 3), and the height difference of the camera setup as input for joint calculation, obtaining the actual size of the package and the coordinates of the world coordinate system, and outputting the position of the package on the conveyor belt to the belt control system.

[0134] As a preferred embodiment, the position of the package with the same number in the image of the midpoint camera 2, the position of the package with the same number in the image of the entrance camera 1 or the exit camera 3, and the height difference of the camera installation are used as inputs to perform a joint calculation to obtain the actual size of the package; the world coordinate system coordinates of the package are determined; the world coordinate system coordinates of the package on the conveyor belt are output to the belt control system; the pinhole imaging principle is used to calculate the real coordinates of the single package formed by the entrance camera 1 or the exit camera 3 and the midpoint camera 2 in the imaging plane coordinate system to eliminate errors, and the actual spatial size and position of the package are determined.

[0135] The process of establishing the world coordinate system includes:

[0136] Establish a camera coordinate system with the midpoint camera 2 as the origin, the imaging plane coordinate system as the X and Y axes, and the vertical direction from the midpoint camera 2 to the calibrated 0 height plane as the Z axis;

[0137] Taking the origin of the camera coordinate system as the origin of the world coordinate system, a world coordinate system is established. At this time, the camera coordinate system is equal to the world coordinate system.

[0138] Convert the coordinates of the imaging plane coordinate system and the camera focal length parameters in the world coordinate system, and calculate to obtain the corresponding world coordinate system coordinates.

[0139] In this embodiment, based on the characteristics that the imaging sizes of the same logistics package are different when imaged by 2D cameras with a height difference, the actual size of the logistics package is calculated according to the pinhole imaging principle of the camera, and then the world coordinate of the package is obtained. This coordinate is input into the single-piece separation module to achieve quasi-real-time and high-precision single-piece separation. The method and system for single-piece separation of logistics packages using multiple 2D cameras based on rotational object detection can use 2D cameras to replace 3D cameras to detect the actual size of the package and output the world coordinates of the package, solving the problems of high equipment cost in engineering applications and poor detection effect of 3D cameras on short packages. The camera setup of this method is relatively simple, and the multi-angle lenses can effectively reduce the occlusion blind area between packages and improve the detection effect.

[0140] The horizontal position detection error of the device of the present invention is within 5 cm, and the position detection resolution in the vertical height direction reaches 0.1 cm. For thin and short packages, if a short package is occluded during the image formation process, the default package height is 5 cm. For irregularly shaped packages, the detection algorithm is used to increase the number of detected corner points on the package surface. For each of the multiple corner points, the pinhole imaging principle is used to calculate the three-dimensional size and the position of the package to increase the accuracy, or the mean value is obtained multiple times for the determination of coordinates and associated regions to be as accurate as possible.

[0141] The solution of the present invention can extend the application scenario of the object to the detection of typical parameters such as the volume, position, and / or mass of any relatively movable object, including measuring the volume of live pigs in the breeding industry through the cooperation of a conveyor belt and 2D cameras to obtain the growth status and the decision for market release. Of course, it can also be extended to any detection scenario with relative displacement from the 2D camera, all within the protection scope of the present invention.

[0142] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they know the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the present invention. Obviously, those skilled in the art can make various changes and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these modifications and variations.

Claims

1. A method for separating objects based on a 2D camera, characterized in that, An object separator based on a 2D camera, wherein the object separator based on the 2D camera comprises: An object conveyor belt is arranged at the bottom, multiple 2D cameras at different heights and / or angles are arranged above the object conveyor belt, a data processor connected to the multiple 2D cameras, and a conveyor belt control system; wherein the 2D cameras are set up at a certain height difference and in a regular manner to cover the entire field of view of a single object; the data processor obtains multiple images captured by the multiple 2D cameras, and uses the feature information of the objects in the captured multiple images to associate and match objects between two 2D cameras with different heights; the camera imaging principle is used to calculate the actual size of the object when imaging the same object in cameras at different heights, and output accurate object position information; the conveyor belt control system is used to receive the object position information and control the object conveyor belt to separate the objects; The object separation method comprises: A preprocessing part and an online processing part, wherein the preprocessing part includes a camera setup step and a camera calibration step, and the online processing part includes an object corner point detection step, a cross-camera identical object recognition step, and an object's actual spatial size and position calculation step; The camera setting step is used to set up multiple 2D cameras, the camera calibration step is used to calibrate the multiple 2D cameras to obtain the mapping relationship between the pixel image coordinate systems of the object image of different cameras, the object corner detection step is used to obtain the corner point information of the object and then obtain the image information of the object, the cross-camera same object recognition step uses a re-recognition algorithm to accurately recognize multiple objects in the associated matching area, and finally assigns the same number to the same object in different images, and the actual spatial size and position calculation step of the object uses the difference in the image position of the same object in two cameras at a certain height to correct the error caused by the imaging of the object due to the small size at a distance and the large size at a close distance, and then calculates the actual spatial size and position information of the object and outputs it to the conveyor control system to achieve single-piece separation of the object; The corner point detection steps of the object include: adding a rotation factor to the conventional object detection algorithm to accurately detect irregularly placed objects and special objects with irregular shapes, and finally obtaining the coordinates of multiple corner points on the upper surface of the object and the position of the object in the image. The rotation factor includes using a rotated object detection model such as rotated yolov5, R 2 CNN, RRPN, Gliding vertex, P-RSDet or SCRDet, or a lightweight deep learning detection model to implement the rotated object test.

2. The object separation method based on a 2D camera according to claim 1, characterized in that The camera setting step includes: Three 2D cameras are set up, namely an entrance camera (1), a midpoint camera (2) and an exit camera (3); wherein the entrance camera (1) is set up directly above the entrance of the object conveyor belt, with a height of a, and the optical axis of the entrance camera (1) is tilted at a certain angle to the vertical direction; the midpoint camera (2) is set up directly above the midpoint of the field of view, with a height of b, and the optical axis of the midpoint camera (2) is placed vertically perpendicular to the plane of the object conveyor belt; the exit camera (3) is set up directly above the exit, with a height of a, and the optical axis of the exit camera (3) is tilted at a certain angle to the vertical direction.

3. The object separation method based on a 2D camera according to claim 2, wherein The camera calibration steps include: establishing a pixel coordinate system for the images captured by the two cameras. When the height of the actual object is determined, the mapping relationship remains unchanged at any position at this height. Two calibrations at different heights are performed between the 2D cameras at different heights to obtain two mapping relationships, so as to obtain the positions of objects at any height between different heights; the camera calibration steps include coordinate transformation of the images formed by the 2D cameras with a height difference in the same actual space field of view area, obtaining the pixel image coordinates of the same actual space position coordinates between different cameras, and finally obtaining the pixel point mapping relationship of the two 2D cameras to the same actual space coordinate, which is regarded as completing one camera calibration; through the aforementioned camera calibration steps, two calibrations are completed, including the mapping relationship of pixel points between the 2D cameras between the 0-height plane and the maximum-height plane. The 0-height mapping relationship is regarded as the first mapping relationship, and the maximum-height plane mapping relationship is regarded as the second mapping relationship; at the same time, an image coordinate system is established with the upper left corner of the 2D camera as the origin, and then an imaging plane coordinate system is established according to the field of view size covered by the camera and the camera resolution to obtain the corresponding relationship between the pixel point length and the actual physical length of the object.

4. The method for separating objects based on a 2D camera according to claim 1 or 2, characterized in that The steps for identifying the same object across cameras include: setting an associated matching area and object re-identification.

5. The object separation method based on a 2D camera according to claim 4, wherein The setting of the associated matching area includes: Obtaining the detection box result of the first object image of the midpoint camera (2); According to the first mapping relationship and the second mapping relationship respectively obtained by the calibration between the midpoint camera (2) and the entrance camera (1) or the exit camera (3) in the 0-height plane and the maximum-height plane in the camera installation strategy and the camera calibration strategy, obtaining the first associated box of the second object image of the entrance camera (1) or the exit camera (3) under the first mapping relationship of the 0-height calibration and the second associated box of the second object image of the entrance camera (1) or the exit camera (3) under the second mapping relationship of the maximum-height plane calibration; Framing all the areas between the first associated box and the second associated box and setting them as the associated matching area; Repeating the above steps to obtain the detection box results of the midpoint camera (2) for each object, and calculating the associated matching area of the corresponding object in the entrance camera (1) or the exit camera (3).

6. The object separation method based on a 2D camera according to claim 5, characterized in that, The object re-identification includes: obtaining a re-identification algorithm model after training the object image data, and using the surface feature information of the object to perform associated matching on the associated matching area of the first object image of each object obtained by the midpoint camera (2) in the entrance camera (1) or the exit camera (3) to obtain the identification of the same object across cameras, and outputting the same number of the same object across cameras.

7. The method for separating an object based on a 2D camera according to claim 1 or 2, characterized in that, The steps for calculating the actual space size and position of the object utilize the difference in the positions of the same object in the images of two cameras with a certain height difference to correct the error caused by the perspective effect of the object (appearing smaller when far and larger when near), and then calculate the actual space size and position information of the object. According to the actual space size and position information of the object, single-piece separation of the object is performed, including: Calculate the different pixel lengths and widths of the same object imaged in the midpoint camera (2) and the entrance camera (1) or the exit camera (3) due to the perspective effect of objects appearing smaller in the distance and larger up close, based on the multiple corner coordinates of the same object in the midpoint camera (2) and the multiple corner coordinates of the entrance camera (1) or the exit camera (3) respectively. Calculate the true size and height information of each object according to the pinhole imaging principle of the camera, determine the true coordinates of each object in the imaging plane coordinate system to eliminate errors, determine the true size of each object and convert it into the coordinates of the imaging plane coordinate system.

8. The object separation method based on a 2D camera according to claim 7, characterized in that, Calculating the position information of the object includes: Taking the midpoint camera (2) as the origin, using the imaging plane coordinate system as the X and Y axes, and taking the vertical direction from the midpoint camera (2) to the calibrated 0-height plane as the Z axis, establish a camera coordinate system. Taking the origin of the camera coordinate system as the origin of the world coordinate system, establish a world coordinate system. At this time, the camera coordinate system is equal to the world coordinate system. Convert the coordinates of the object in the imaging plane coordinate system and the camera focal length parameters in the world coordinate system, and calculate to obtain the corresponding world coordinate system coordinates.

Images