Material removing system based on executing mechanism and camera

Abstract

The invention discloses a material removing system based on an executing mechanism and a camera. The material removing system comprises an encoder, a belt line, a controller, materials, the camera and the executing mechanism. The encoder is connected with the belt line through friction or directly connected to a driving wheel or a driven wheel of the belt line, and when the belt line rotates, the encoder is driven to rotate, and the encoder generates pulse signals. The controller is connected with the encoder and used for receiving the pulse signals generated by the encoder and recording the pulse value of the materials moving to the portion below the executing mechanism through the belt line, and the pulse difference between the executing mechanism and the camera is obtained. The camera comprises an X-ray camera and a line scanning camera, and the X-ray camera and the line scanning camera are both connected with the encoder, receive pulse signals generated by the encoder and enable materials to be imaged in the camera through the pulse signals; the device has the beneficial effects that whether the materials are qualified or not is judged by identifying whether contents appear at the material seals or not, and the device is linked with an execution mechanism to remove unqualified materials.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual detection and rejection, and in particular relates to a material rejection system based on an actuator and a camera. Background Art

[0002] With the continuous improvement of industrial automation and intelligence, visual inspection and rejection technology is increasingly used in high-speed production lines. This technology mainly uses visual sensors to perform real-time inspection of products on the production line, and removes unqualified products through actuators based on the inspection results. The core of this technology lies in accurately and quickly judging product quality, and precisely controlling the actuators to remove products at the appropriate time.

[0003] However, current methods mainly focus on detecting defects on the surface of materials through industrial cameras, and are unable to identify defects such as oil or material stuck in the sealing of materials. Figure 1 and Figure 2 As shown, Figure 1 The image of the contents taken by the X-ray camera, Figure 2 This is an image of the outer packaging taken by a line scan camera. It is not possible to determine whether there is material stuck in the packaging seal only through the outer packaging image. Only the image of the contents corresponding to the outer packaging seal area can clearly show whether there is material stuck. Summary of the invention

[0004] The purpose of the present invention is to provide a material rejection system based on an actuator and a camera, which determines whether the material is qualified by identifying whether the contents appear in the sealing area of ​​the outer packaging, and works in conjunction with the actuator to reject unqualified materials.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a material rejection system based on an actuator and a camera, comprising an encoder, a belt line, a controller, a material, a camera, and an actuator;

[0006] The encoder is connected to the belt line through friction, or directly connected to the driving wheel or driven wheel of the belt line. When the belt line rotates, the encoder is driven to rotate, and the encoder generates a pulse signal;

[0007] The controller is connected to the encoder and is used to receive the pulse signal generated by the encoder, record the pulse value of the material moving under the actuator through the belt line, and obtain the pulse difference between the actuator and the camera;

[0008] The camera includes a line scan camera and an X-ray camera, and both the line scan camera and the X-ray camera are connected to an encoder, receive a pulse signal generated by the encoder, and image the material in the camera through the pulse signal; the X-ray camera obtains an image of the content of the material, and the line scan camera obtains an image of the outer packaging of the material;

[0009] The actuator is connected to the controller and is used to remove materials that do not meet the standards based on the coordinates corresponding to the material position, the pulse value corresponding to the material position, and the pulse difference between the actuator and the camera when the material on the belt line is visually detected.

[0010] As a preferred technical solution of the present invention, it also includes performing an image alignment operation on the content image and the outer packaging image; using a conventional image segmentation method to confirm the packaging area of ​​the outer packaging image, and then using a sealing positioning method to confirm the sealing area of ​​the outer packaging; using a conventional image segmentation method to confirm the content area of ​​the content image; and judging whether the content in the content image appears in the sealing area by finding the intersection of the sealing area and the content area.

[0011] As a preferred technical solution of the present invention, when performing image alignment operation on the content image and the outer packaging image, the registered image of the content image and the registered image of the outer packaging image are obtained and unified in the belt line coordinate system. The registration is based on affine transformation, including a series of cropping, scaling, translation, and filling operations. The above transformation types are combined to obtain the homogeneous matrix of the affine transformation model:

[0012]

[0013] in is the translation matrix, is the scaling matrix, is the rotation matrix, is the pixel coordinate of the mark point in the content image, is the pixel coordinates of the same mark point in the outer packaging image, du, dv, z 1 ,z 2 ,θ represents five affine transformation parameters.

[0014] As a preferred technical solution of the present invention, the outer packaging image seal positioning method includes an algorithm implementation for positioning the packaging seal area, and returns the seal position by inputting the original image, outer packaging area information and seal size and shape parameters; wherein the outer packaging area information is obtained using a conventional image segmentation algorithm.

[0015] As a preferred technical solution of the present invention, the method for positioning the sealing area of ​​the package comprises the following steps:

[0016] Step 1: Prepare a seal classification training data set; for the case where the seal is located on the short side of the package, first rotate each package sample so that the long side of the package is horizontal, and divide it into four categories according to the direction of the seal: the front seal of the package faces left, the front seal of the package faces right, the back seal of the package faces left, and the back seal of the package faces right; the number of samples in each category is 50 to 100; for the case where the seal is located on the long side of the package, similarly rotate each package sample so that the short side of the package is horizontal, and divide it into four categories: the front seal of the package faces left, the front seal of the package faces right, the back seal of the package faces left, and the back seal of the package faces right, and the number of samples in each category is 50 to 100;

[0017] Step 2: training a seal classification recognizer; based on the training data set prepared in step 1, a conventional image classification algorithm is used to classify and train the seal positions; the trained classification recognizer can recognize four types of seal orientations, corresponding to the front seal of the package facing left, the front seal of the package facing right, the back seal of the package facing left, and the back seal of the package facing right;

[0018] Step 3: seal identification; using the trained seal classification identifier, firstly, according to the outer packaging area information, the minimum bounding rectangle thumbnail of the outer packaging is intercepted, and the center point coordinates, length, width and rotation angle of the minimum bounding rectangle thumbnail are recorded. The minimum bounding rectangle thumbnail to be identified is rotated so that the long side or short side of the package is horizontal according to the actual situation, and then the rotated image is input into the seal classification identifier for identification; according to the seal classification identification result and the recorded center point coordinates, length, width and rotation angle of the minimum bounding rectangle thumbnail, the specific orientation of the package in the original outer packaging image is determined; finally, according to the input seal size and shape parameters, with the outer edge of the seal orientation as the starting position, along the direction perpendicular to the rotation angle of the minimum bounding rectangle, the area that meets the seal size and shape is intercepted as the seal area.

[0019] As a preferred technical solution of the present invention, by finding the intersection of the sealing area and the content area, it is judged whether the content in the content image appears in the sealing area. The specific implementation method is as follows: after determining the sealing area of ​​the package, the sealing area is set to white with a pixel value of 255, and the remaining image areas are set to black pixel values ​​of 0, so as to obtain a sealing area mask image; after determining the content area using a conventional image segmentation algorithm, the content area is set to white with a pixel value of 255, and the remaining image areas are set to black pixel values ​​of 0, so as to obtain a content area mask image; the pixels at corresponding positions of the sealing area mask image and the content area mask image are calculated, and only at the pixel positions where both the two input mask images are white 255, the result image is set to 255, otherwise it is set to 0, and the content mask image at the sealing location is obtained in this way; if the area of ​​the white area in the content mask image at the sealing location is greater than the set threshold, the judgment result is wrong, and if it is less than the threshold, it is correct, and the information is transmitted to the controller.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] By combining X-ray penetration to photograph the contents and line scanning light to determine the sealing area, it is possible to identify whether the contents appear at the seal of the outer packaging, determine whether the material is qualified, and work with the actuator to remove unqualified materials. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 An image of the contents taken by the X-ray camera of the present invention;

[0023] Figure 2 An image of the outer packaging taken by the line scan camera of the present invention;

[0024] Figure 3 The figure is an overall flow chart of material detection and rejection according to the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Example 1

[0027] See also Figure 1-Figure 3 , which is the first embodiment of the present invention, and which provides a material rejection system based on an actuator and a camera, including an encoder, a belt line, a camera, a material, a controller, and an actuator;

[0028] The encoder is connected to the belt line through friction, or directly connected to the driving wheel or driven wheel of the belt line. When the belt line rotates, it drives the encoder to rotate, and the encoder generates a pulse signal.

[0029] The controller is connected to the encoder to receive the pulse signal generated by the encoder, record the pulse value of the material moving under the actuator through the belt line, and obtain the pulse difference between the actuator and the camera;

[0030] The camera includes a line scan camera and an X-ray camera, and both the line scan camera and the X-ray camera are connected to the encoder, receive the pulse signal generated by the encoder, and image the material in the camera through the pulse signal; the content image of the material is obtained by the X-ray camera, and the outer packaging image of the material is obtained by the line scan camera;

[0031] The actuator is connected to the controller and is used to remove materials that do not meet the standards based on the coordinates corresponding to the material position, the pulse value corresponding to the material position, and the pulse difference between the actuator and the camera when the material on the belt line is detected by the conventional image segmentation algorithm.

[0032] In this embodiment, preferably, it also includes an image alignment operation for the content image and the outer packaging image: obtaining the registration image of the content image and the registration image of the outer packaging image, unifying them in the belt line coordinate system, and the registration is based on affine transformation, including a series of operations such as cropping, scaling, translation, and filling. The above transformation types are combined to obtain the homogeneous matrix of the affine transformation model:

[0033]

[0034] in is the translation matrix, is the scaling matrix, is the rotation matrix, is the pixel coordinate of the mark point in the content image, is the pixel coordinates of the same mark point in the outer packaging image, du, dv, z 1 ,z 2 ,θ represents five affine transformation parameters;

[0035] The outer packaging area is confirmed by using image segmentation on the outer packaging image, and the sealing position area is confirmed by using the sealing positioning method; the content area is determined by using image segmentation on the content image, and the content area is output;

[0036] By finding the intersection of the sealing area and the content area, it is determined whether there is content in the content image at the sealing position. The specific process is as follows: the sealing area is set to white (pixel value 255) and the rest of the image area is set to black pixel value (0) to obtain the sealing area mask image; then the content area is set to white (pixel value 255) and the rest of the image area is set to black pixel value (0) to obtain the sealing area mask image; the pixels at the corresponding positions of the two images are ANDed to obtain the mask image of the content at the sealing position. If the area of ​​the white area in the mask image of the content at the sealing position is greater than the set threshold, the judgment result is wrong, and if it is less than the threshold, it is correct, and the information is passed to the controller.

[0037] In this embodiment, preferably, the method for locating the seal position of the outer packaging image includes an algorithm for locating the package seal, and returns the seal position by inputting the original image, the outer packaging area information, and the seal size and shape parameters; and includes the following steps:

[0038] Step 1: Prepare a seal classification training data set. For the case where the seal is located on the short side of the package, first rotate each package sample so that the long side of the package is horizontal, and divide it into four categories according to the direction of the seal: the front seal of the package faces left, the front seal of the package faces right, the back seal of the package faces left, and the back seal of the package faces right. The number of samples in each category is 50 to 100. For the case where the seal is located on the long side of the package, similarly rotate each package sample so that the short side of the package is horizontal, and divide it into four categories: the front seal of the package faces left, the front seal of the package faces right, the back seal of the package faces left, and the back seal of the package faces right, and the number of samples in each category is 50 to 100.

[0039] Step 2: Train the seal classification recognizer. Based on the training data set prepared in step 1, a conventional image classification algorithm is used to classify and train the seal position. The trained classification recognizer can recognize four types of seal orientations, corresponding to the front seal facing left, the front seal facing right, the back seal facing left, and the back seal facing right.

[0040] Step 3: Seal identification. Using the trained seal classification identifier, firstly intercept the smallest bounding rectangle of the outer packaging according to the outer packaging area information, record the center point coordinates, length, width and rotation angle of the smallest bounding rectangle, rotate the smallest bounding rectangle to be identified, so that the long side or short side of the package is horizontal according to the actual situation, and then input the rotated image into the seal classification identifier for identification. According to the seal classification identification results and the recorded center point coordinates, length, width and rotation angle of the smallest bounding rectangle, determine the specific orientation of the package in the original outer packaging image, and finally, according to the passed seal width parameter, take the outer edge of the seal orientation as the starting position, and intercept the area that meets the seal size and shape along the direction perpendicular to the rotation angle of the smallest bounding rectangle as the seal area.

[0041] Example 2

[0042] See also Figure 1-Figure 3 , which is the second embodiment of the present invention, and this embodiment provides the following technical solutions:

[0043] Food packaging inspection agencies based on X-ray machines include:

[0044] Use a calibration plate to align the image of the contents taken by the X-ray machine and the image of the outer packaging taken by the line scan camera to unify the coordinate system;

[0045] By setting the belt line moving direction as the Y direction and the direction perpendicular to the belt line moving direction as the X direction, a calibration system of the belt line and the camera image coordinate system is established:

[0046] Based on the belt line and camera calibration system, the pulse signal generated by the encoder and the Y value corresponding to each pixel of the calibration plate in the camera field of view are used, and the zero point of the camera coordinate system is set to a corner point of the calibration plate. The image coordinates of the X-ray camera are converted into the line scan camera image coordinate system through the calibration plate; the specific method is as follows:

[0047] A corner point of the calibration plate is selected as the 0 point position. The X-ray camera and the line scan camera shoot the same calibration plate. After shooting, the X-ray camera and the line scan camera can obtain the position of the same corner point in their respective image coordinate systems. The two images are registered to ensure that the shape, size and position of the same package in different images are consistent. The registration is based on affine transformation, which includes a series of operations such as caching, splicing, scaling, translation, rotation, filling, and cropping.

[0048] Specifically include:

[0049] Cache images. Since the package may be at the edge of the image, which may cause missed detection, the previous frame of the X-ray camera and line scan camera needs to be cached.

[0050] Stitching images: crop a fixed area from the cached previous frame and stitch it with the current frame. The size of the fixed area is a manually specified parameter, which is generally related to the package size and is generally half the size of the previous frame. When taking pictures, it is impossible to ensure that each package appears completely in one picture, so image stitching is required to stitch part of the previous picture onto the top of the next picture to ensure that each package appears completely at least once. Since the Y-axis coordinate of the package position can be converted into a pulse value, the position of each package is unique, and the repeated complete packages in the picture can be eliminated according to the pulse value coordinate of the center point of the package.

[0051] Scaling, translation, and rotation of images. Since the parameters of the X-ray camera and the line scan camera (camera parameters, installation height, installation position, etc.) are inconsistent, their coordinate system bases are different and the same package cannot be aligned. By calculating the four corner points of the input, the following transformation matrix is ​​obtained:

[0052] Combining the above transformation types gives the homogeneous matrix of the affine transformation model:

[0053]

[0054] in is the translation matrix, is the scaling matrix, is the rotation matrix, is the pixel coordinate of the mark point in the content image, is the pixel coordinates of the same mark point in the outer packaging image, du, dv, z 1 ,z 2 ,θ represents five affine transformation parameters;

[0055] Fill (or crop) the image. Since the aligned X-ray camera and line scan camera images have different resolutions, the line scan camera needs to be used as a reference. If the resolution of the filled X-ray camera image is smaller, the X-ray camera image is filled with a black border, otherwise it is cropped.

[0056] So far, we can use this method to unify the detection results of all images into a camera coordinate system; and there is a strict functional relationship between the Y value of the image coordinate system and the encoder pulse value. The number of pulses = the Y direction value of the image position / the number of imaging rows for each pulse + the number of pulses per picture * the number of pictures. The X direction can be mapped to the position of the calibration plate.

[0057] Based on the belt line and camera calibration system, the pulse signal generated by the encoder is transmitted to the controller, and the corner point selected by the calibration plate is moved to the actuator through the belt line to record the pulse value, and the pulse difference between the actuator and the camera is obtained. The calibration plate is moved under the ruler, and the laser pen is moved to align each line of the calibration plate, and the position of the laser pen is recorded. Since there is a strict corresponding relationship between the position of the laser pen and the position of the actuator, the actuator and the calibration plate's X coordinate system are aligned by aligning the zero point in the X direction;

[0058] By calibrating the upper and lower cameras and the X-ray machine, their respective affine transformation matrices are generated so that the images they capture can be converted into coordinates under the actuator coordinates;

[0059] Perform image segmentation on the outer packaging image captured by the line scan image to determine the regional information of the packaging; use a conventional image classification algorithm to determine the sealing direction, thereby determining the sealing area;

[0060] The sealing area of ​​any shape is represented by the sealing contour point set. After obtaining the sealing direction using the conventional image classification algorithm, the position of the sealing area of ​​any shape can be determined in the original image based on the outer packaging area information.

[0061] Since areas with different tissue densities and absorption rates will present different grayscales or colors, an X-ray image of the inspected object is formed; the density of the contents of the material packaging is relatively large, so a darker gray area will be formed; then conventional image segmentation algorithms are used to extract the content area.

[0062] Find the intersection of the content area and the sealing area. If its area is larger than the preset threshold, it is judged as an unqualified product.

[0063] The controller calculates the Y value of the camera coordinate based on the pulse difference between the actuator and the camera, and obtains the pulse number of the unqualified product reaching the bottom of the blowing device; finally, when the product reaches the bottom of the blowing device, based on the correspondence between the coordinate system of the actuator and the X coordinate system of the calibration plate, the corresponding blowing device is calculated after obtaining the X coordinate of the product position, and the blowing device performs air blowing to remove the unqualified products.

[0064] Although the embodiments of the present invention have been shown and described, as detailed above, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A material rejection system based on an actuator and a camera, characterized in that: Including encoder, belt line, controller, material, camera, actuator; The encoder is connected to the belt line through friction, or directly connected to the driving wheel or driven wheel of the belt line. When the belt line rotates, the encoder is driven to rotate, and the encoder generates a pulse signal; The controller is connected to the encoder and is used to receive the pulse signal generated by the encoder, record the pulse value of the material moving under the actuator through the belt line, and obtain the pulse difference between the actuator and the camera; The camera includes an X-ray camera and a line scan camera, and both the X-ray camera and the line scan camera are connected to an encoder, receive a pulse signal generated by the encoder, and image the material in the camera through the pulse signal; the X-ray camera obtains an image of the content of the material, and the line scan camera obtains an image of the outer packaging of the material; The actuator is connected to the controller and is used to remove materials that do not meet the standards based on the coordinates corresponding to the material position, the pulse value corresponding to the material position, and the pulse difference between the actuator and the camera when the material on the belt line is detected by a conventional image segmentation algorithm.

2. A material rejection system based on an actuator and a camera according to claim 1, characterized in that: It also includes performing an image alignment operation on the content image and the outer packaging image; using conventional image segmentation on the outer packaging image to confirm the outer packaging image area, and then using the outer packaging seal positioning method to confirm the outer packaging seal area; using conventional image segmentation on the content image to confirm the content area; By solving the intersection of the outer packaging sealing area and the content area, it is determined whether there is content in the sealing area.

3. A material rejection system based on an actuator and a camera according to claim 2, characterized in that: When performing image alignment on the content image and the outer packaging image, obtain the registered image of the content image and the registered image of the outer packaging image, and unify them in the belt line coordinate system. The registration is based on affine transformation, which includes a series of cropping, scaling, translation, and filling operations. The above transformation types are combined to obtain the homogeneous matrix of the affine transformation model: in is the translation matrix, is the scaling matrix, is the rotation matrix, is the pixel coordinate of the mark point in the content image, are the pixel coordinates of the same mark point in the outer packaging image, du, dv, z1, z2, and θ represent five affine transformation parameters.

4. A material rejection system based on an actuator and a camera according to claim 2, characterized in that: By finding the intersection of the outer packaging sealing area and the content area, it is determined whether there is content at the sealing position. The specific implementation method is as follows: Construct a blank image of the same size as the registration image of the outer packaging image, set the sealed area in the blank image to white with a pixel value of 255, and set the rest of the image area to black pixel value 0, so as to obtain a sealed area mask image; construct a blank image of the same size as the registration image of the content image, set the content area in the blank image to white with a pixel value of 255, and set the rest of the image area to black pixel value 0, so as to obtain a content mask image; perform AND operation on the pixels at the corresponding positions of the sealed area mask image and the content mask image, so as to obtain a mask image of the content at the sealed position; If the area of ​​the mask image of the contents at the seal is larger than the set threshold, the result is judged as wrong, if it is smaller, it is correct, and the information is transmitted to the controller.

5. The material rejection system based on an actuator and a camera according to claim 2, characterized in that: The outer packaging seal positioning method includes the algorithm implementation of packaging seal area positioning. By inputting the original image, outer packaging area information, and seal size and shape parameters, the seal position is returned.

6. A material rejection system based on an actuator and a camera according to claim 5, characterized in that: The outer packaging sealing positioning method comprises the following steps: Step 1: Prepare a seal classification training data set; for the case where the seal is located on the short side of the package, first rotate each package sample so that the long side of the package is horizontal, and divide it into four categories according to the direction of the seal: the front seal of the package faces left, the front seal of the package faces right, the back seal of the package faces left, and the back seal of the package faces right; the number of samples in each category is 50 to 100; for the case where the seal is located on the long side of the package, similarly rotate each package sample so that the short side of the package is horizontal, and divide it into four categories: the front seal of the package faces left, the front seal of the package faces right, the back seal of the package faces left, and the back seal of the package faces right, and the number of samples in each category is 50 to 100; Step 2: training a seal classification identifier; based on the training data set prepared in step 1, a conventional image classification algorithm is used to classify and train the seal positions; The trained classification recognizer can recognize four types of seal orientations, corresponding to the front seal facing left, the front seal facing right, the back seal facing left, and the back seal facing right. Step 3: seal identification; using the trained seal classification identifier, firstly, the smallest circumscribed rectangle of the outer packaging is captured according to the outer packaging area information, and the smallest circumscribed rectangle to be identified is rotated so that the long side or short side of the packaging is horizontal according to the actual situation, and then the rotated image is input into the seal classification identifier for identification; the specific orientation of the package in the original outer packaging image is determined according to the identification result, and finally, according to the input seal size and shape parameters, the area that meets the seal size and shape is captured along the seal orientation as the seal area.

Images