A grain sorting method based on machine vision

Abstract

The present invention discloses a grain sorting method based on machine vision, comprising the following steps: S1: creating a grain template; S2: camera calibration; S3: scanning the grains on the mother film with the camera to obtain a scanned image of the grains, wherein the scanned image includes the mechanical coordinates of the grains; S4: matching the mechanical coordinates of the grains with the grid coordinates one by one to obtain the type information of the grains; the grid coordinates refer to coordinates with the center point of the mother film as the origin, and the grid coordinates mark the type of grains at each position of the mother film; S5: classifying the grains and moving the grains of the same type information to the same sub-film. The present invention provides a grain sorting method based on machine vision, which automatically completes the scanning and judgment of the grains on the mother film based on the guidance of machine vision, thereby improving the accuracy and efficiency of grain classification and reducing the error rate.

Description

Technical Field

[0001] The present invention relates to the field of grain classification, and in particular to a grain sorting method based on machine vision. Background Art

[0002] As LED technology continues to evolve, the demand for and quality of LEDs in production is becoming increasingly stringent, placing higher demands on the quality of LED die. During LED wafer processing, the wafer must be cut into multiple dies and then sorted by type. After sorting, dies of the same type must be moved to the same sub-film.

[0003] In the prior art, grain sorting and handling are performed individually, requiring manual identification of the grain type, marking its location, and then transporting it to the corresponding sub-film. Existing grain sorting and handling methods are inefficient and prone to human error. Summary of the Invention

[0004] The present invention aims to address, at least to some extent, one of the problems in the related art. To this end, the present invention provides a machine vision-based grain sorting method that automatically scans and identifies grains on a matrix film using machine vision guidance, thereby improving the accuracy and efficiency of grain sorting and reducing the error rate.

[0005] In order to achieve the above objectives, the present application adopts the following technical solution: a grain sorting method based on machine vision, comprising the following steps:

[0006] S1: Create grain template;

[0007] S2: camera calibration;

[0008] S3: The camera scans the grains on the mother film to obtain a scanned image of the grains, wherein the scanned image includes the mechanical coordinates of the grains;

[0009] S4: Match the mechanical coordinates of the grains to the grid coordinates one by one to obtain the type information of the grains; the grid coordinates refer to the coordinates with the center point of the mother film as the origin, and the grid coordinates mark the type of grains at each position of the mother film;

[0010] S5: Classify the grains and move the grains with the same type of information into the same sub-membrane.

[0011] Furthermore, creating the grain template in step S1 includes creating M grain templates, each grain template correspondingly storing characteristics of a grain type; M is an integer greater than 0.

[0012] Furthermore, in step S3, when the camera scans the grains on the mother film, the camera is fixed, and the mother film moving component drives the mother film to move in a plane at the same distance from the camera to obtain complete mother film surface scanning information.

[0013] Furthermore, step S4 also includes performing AOI inspection on the scanned image to obtain defect information of the grains.

[0014] Furthermore, the method further includes S6: detecting the grains fixed in the sub-membrane.

[0015] Furthermore, there are two cameras, and the fields of view of the two cameras correspond to the mother film and the daughter film respectively.

[0016] Furthermore, in step S5, the die is transported using a transport assembly, wherein the transport assembly includes a suction nozzle;

[0017] During the handling process, both cameras and the handling assembly are connected to the control unit. The camera corresponding to the mother film monitors the position of the suction nozzle in the handling assembly in real time to ensure that the suction nozzle is aligned with the center of the die to be handled. The control unit then controls the handling assembly to suck the die.

[0018] During the transport process, the camera corresponding to the sub-film monitors the position of the suction nozzle in the transport component in real time to ensure that the grain in the suction nozzle is aligned with the position to be transported. The control unit then controls the transport component to place the grain.

[0019] Furthermore, the mother membrane and the daughter membrane are arranged relative to each other.

[0020] Furthermore, two cameras are symmetrically and parallelly arranged between the mother film and the daughter film, a lens barrel is provided in the camera, a 45° reflector is provided in front of the lens barrel, and the angle between the mirror surface of the 45° reflector and the axis of the lens barrel is 45°; and the mirror surface of the 45° reflector faces the outside of the center of the line connecting the two lens barrels.

[0021] The above technical solution provided by the embodiment of the present application has the following advantages compared with the existing technology: the present application uses a camera to scan the grains on the mother film to obtain a scanned image of the grains. Since the grid coordinates of the grains themselves have stored the type of grains at the corresponding positions, the mechanical coordinates of the grains and the grid coordinates are matched one by one to obtain the type information of the grains at each position in the mechanical coordinates of the grains, and the subsequent transportation of the grains can be completed based on the type information; the present application is based on the guidance of machine vision to automatically complete the scanning and judgment of the grains on the mother film, thereby improving the accuracy and efficiency of grain classification and reducing the misjudgment rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] In the attached figure:

[0025] Figure 1 Schematic diagram of the process of the grain sorting method based on machine vision in this application;

[0026] Figure 2 Schematic diagram of the front view of the camera in Example 1;

[0027] Figure 3 Schematic diagram of the back of the camera in Example 2;

[0028] Reference numerals: 101, fixed base; 102, lifting adjustment assembly; 107, camera; 108, lens barrel; 109, 45° reflector; 110, air path control system; 111, first field of view; 112, second field of view. DETAILED DESCRIPTION

[0029] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, the specific embodiments of the present invention are now described in detail with reference to the accompanying drawings. In the following description, it should be understood that the directions or positional relationships indicated by "front", "back", "up", "down", "left", "right", "longitudinal", "horizontal", "vertical", "horizontal", "top", "bottom", "inside", "outside", "head", "tail", etc. are based on the directions or positional relationships shown in the accompanying drawings and are constructed and operated in specific directions. They are only for the convenience of describing the technical solution and do not indicate that the mechanisms or components referred to must have specific directions. Therefore, they should not be understood as limiting the present invention.

[0030] It should also be noted that, unless otherwise clearly specified and limited, terms such as "installed", "connected", "connected", "fixed", and "set" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integrated connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. When an element is referred to as being "on" or "under" another element, the element can be "directly" or "indirectly" located on the other element, or there may be one or more intervening elements. The terms "first", "second", "third", etc. are only for the convenience of describing the present technical solution, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second", "third", etc. may explicitly or implicitly include one or more of such features. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0031] In the following description, specific details such as particular system structures and techniques are provided for purposes of illustration, not limitation, to facilitate a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, mechanisms, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0032] Please see the attached Figure 1 , the present application provides a grain sorting method based on machine vision, comprising the following steps:

[0033] S1: Create grain template;

[0034] S2: camera calibration;

[0035] S3: The camera scans the grains on the mother film to obtain a scanned image of the grains, which includes the mechanical coordinates of the grains;

[0036] S4: Match the mechanical coordinates of the grains to the grid coordinates one by one to obtain the type information of the grains; the grid coordinates refer to the coordinates with the center point of the mother film as the origin, and the grid coordinates mark the type of grains at each position of the mother film;

[0037] S5: Classify the grains and move the grains with the same type of information into the same sub-membrane.

[0038] This application uses a camera to scan the grains on the mother film to obtain a scanned image of the grains. Since the grid coordinates of the grains themselves have stored the type of grains at the corresponding position, the mechanical coordinates of the grains and the grid coordinates are matched one by one to obtain the type information of the grains at each position in the mechanical coordinates of the grains, and the subsequent grain transportation can be completed based on the type information; this application is based on the guidance of machine vision to automatically complete the scanning and judgment of the grains on the mother film, thereby improving the accuracy and efficiency of grain classification and reducing the misjudgment rate.

[0039] Example 1

[0040] The present application provides a grain sorting method based on machine vision, comprising the following steps:

[0041] S1: Creating a grain template; specifically, creating M grain templates, each of which stores characteristics of a grain type, where M is an integer greater than 0. Specifically, characteristics corresponding to all existing grain types can be stored in the grain templates to ensure that the camera can identify the types of all grains in the mother film based on the scanning results. The specific grain type can refer to the grain grade or other characteristic types.

[0042] S2: Camera calibration; Specifically, the camera calibration method in the prior art can be used to calibrate the camera. The camera calibration process involves conversions between the world coordinate system, the camera coordinate system, the image coordinate system, and the grain mechanical coordinate system. Among them, the world coordinate system is the absolute coordinate system of the objective three-dimensional world, the camera coordinate system refers to the optical center of the camera as the coordinate origin, the X-axis and Y-axis are parallel to the X-axis and Y-axis of the image coordinate system respectively, and the optical axis of the camera is the Z-axis. The image coordinate system refers to the center of the image plane as the coordinate origin, and the X-axis and Y-axis are parallel to the two vertical edges of the image plane respectively. The grain mechanical coordinate system takes the upper left corner vertex of the image plane as the origin, and the X-axis and Y-axis are parallel to the X-axis and Y-axis of the image coordinate system respectively.

[0043] S3: The camera scans the grains on the mother film to obtain a scanned image of the grains, which includes the mechanical coordinates of the grains;

[0044] Among them, during the mother film scanning process, the camera's field of view cannot cover the entire area of the mother film. This application keeps the camera fixed, and the mother film moving component drives the mother film to move in a plane at the same distance from the camera, ensuring that the camera can obtain images of various areas in the mother film, and splice multiple images to form a complete mother film image for output; the output mother film image includes the mechanical coordinates of the grain, which refers to the various coordinate values of the grain in the grain mechanical coordinate system.

[0045] S4: Align the mechanical coordinates of the grains with the grid coordinates to obtain the image coordinates and type information of the grains. Grid coordinates refer to the coordinate values of the grains within the grid coordinate system, where the origin is the center of the master film, and the X-axis and Y-axis are parallel to the two vertical edges of the image plane. Material coordinates are provided by the master film supplier. The corresponding data of the master film includes the coordinate values of the grains at various positions in the master film within the grid coordinate system, as well as the type information of the grains, which can be grain grade information.

[0046] The scanned image output by the camera in the above steps contains the mechanical coordinates of the grains. By matching the mechanical coordinates of the grains with the grid coordinates, the grain type under the material coordinates can be matched to the grain mechanical coordinates, and then the grain mechanical coordinates and grain type of each grain in the scanned image can be obtained.

[0047] After obtaining the grain type, this application also needs to use AOI (Automated Optical Inspection) equipment to perform AOI inspection on the scanned image to obtain defect information of each grain. The defects of the grains cannot be observed by the human eye and can only be obtained with the help of AOI equipment. This application performs AOI inspection on the scanned image to avoid inspecting the grains on the mother film one by one. If the grains on the mother film are inspected, it is necessary to drive the AOI equipment to align with each grain, which makes the inspection time longer and is not conducive to the efficiency of grain classification and transportation.

[0048] The present application also includes a control unit that is connected to the camera, AOI equipment, and subsequent handling components to control the operation of the camera, AOI equipment, and handling components. The AOI equipment outputs the inspection results to the control unit, which annotates the defects of each grain at the mechanical coordinates of each grain in the scanned image.

[0049] S5: Classify the grains and move the grains with the same type of information into the same sub-membrane.

[0050] In this step, a crystal grain transport component is used to transport the crystal grains. The transport component includes a suction nozzle, which is connected to a vacuum pipe. The crystal grains are sucked and placed by controlling the vacuum pipe.

[0051] The present application includes two cameras, corresponding to the field of view of the mother film and the field of view of the daughter film respectively. The camera corresponding to the mother film has two functions: first, before the grains are transported, the camera scans the mother film to obtain the position and type of the grains in the mother film; second, when the grains are transported, the camera is used to monitor whether the suction nozzle is aligned with the grains to be transported, ensuring that the suction nozzle can accurately align with the center position of the grains to be transported. During the transport process, the two cameras and the transport component are both connected to the control unit for communication. The camera corresponding to the mother film monitors the position of the suction nozzle in the transport component in real time to ensure that the suction nozzle is aligned with the center of the grains to be transported. The control unit then controls the transport component to absorb the grains.

[0052] The camera associated with the sub-film functions as follows: When the nozzle brings the die to the sub-film, the camera monitors whether the die reaches the placement location, ensuring that the center of the die held by the nozzle is aligned with the placement location. During the transport process, the camera associated with the sub-film monitors the position of the nozzle in the transport assembly in real time. Once the die in the nozzle is aligned with the placement location, the control unit controls the transport assembly to place the die.

[0053] Specifically, during the grain transportation process, grains of the same type marked with the same defect can be placed in a set area in the sub-membrane. One sub-membrane is used to place grains of the same type. Even grains of the same type may have different defects during the AOI inspection process. For example, some grains are intact, while some grains have worn surfaces. After determining the same type of grains in the mother film, the control unit controls the transportation component to transport the intact grains of this type to the top of the corresponding sub-membrane, and transport the grains with worn surfaces to the bottom of the sub-membrane, and feed back the transportation information to the control unit through the second camera. The control unit records the placement position in each sub-membrane to facilitate subsequent retrieval of the grains in the sub-membrane.

[0054] In this application, the mother film and the daughter film are arranged opposite to each other, and two cameras are symmetrically and parallelly arranged between the mother film and the daughter film. A lens barrel is provided in the camera, and a 45° reflector is provided in front of the lens barrel. The angle between the mirror surface of the 45° reflector and the axis of the lens barrel is 45°; and the mirror surface of the 45° reflector faces the outside of the center of the line connecting the two lens barrels.

[0055] S6: The grains fixed in the sub-membrane are inspected. The inspection here mainly checks whether the grains are firmly placed in the sub-membrane to prevent the grains from falling off.

[0056] Example 2

[0057] In this embodiment, the daughter film and the mother film are two opposite planes, so two cameras need to be set up, corresponding to the field of view of the mother film and the field of view of the daughter film respectively. The two cameras are symmetrically and parallelly set between the mother film and the daughter film, as shown in the attached figure. Figure 2-3As shown, the complete vision system formed by the two cameras includes:

[0058] Two cameras 107 are arranged symmetrically and in parallel,

[0059] Two lens barrels 108 respectively located at the front ends of the two cameras 107;

[0060] Two 45° reflectors 109 are respectively located in front of the lens barrel 108 , wherein the angle between the mirror surface of the 45° reflector 109 and the axis of the lens barrel 108 is 45°; and the mirror surface of the 45° reflector 109 faces the outside of the center of the line connecting the two lens barrels 108 .

[0061] In the present application, the camera 107, the lens barrel 108 and the 45° reflector 109 are all symmetrically arranged in order to form two mutually parallel viewing planes, which respectively correspond to the absorption and placement of the grains. Specifically, a grain absorption position can be set in one of the viewing planes, and a grain placement position can be set in the other viewing plane. Through the observation of the camera 107 and the lens barrel 108, it can be ensured that the positions where the grains are absorbed and placed are without deviation, and then the mechanical errors of the conveying device can be corrected to ensure that the grains can be efficiently and accurately conveyed from the grain absorption position to the grain placement position, thereby reducing the grain alignment error and improving the conveying efficiency.

[0062] In this application, the 45° reflector 109 refracts the field of view of the camera 107 to form a 90° angle, and rotates the field of view originally parallel to the barrel surface by 90° to form a field of view perpendicular to the barrel surface. At the same time, it is also necessary to satisfy the requirement that the angle between the mirror surface of the 45° reflector 109 and the axis of the barrel 108 is 45°; and the mirror surface of the 45° reflector 109 faces the outside of the center of the line connecting the two barrels 108. This application uses the 45° reflector 109 placed in front of the barrel 108 to enable the two parallel cameras 107 to observe the field of view of the two sides parallel to the axis of the barrel 108. As shown in the attached figure Figure 2 As shown, the present application can place the die suction position in the first viewing surface 111 and set the die placement position in the second viewing surface 112 .

[0063] As a specific embodiment, the present application further includes a fixed base 101, which is fixed to the die handling device. Two symmetrically and parallelly arranged cameras 107 are disposed above the fixed base 101. The visual mechanism in the present application is used to achieve alignment between die suction and die placement during die handling. It is part of the handling device and can also be used as a separate visual monitoring system.

[0064] In order to increase the use range of the visual mechanism and to adjust the field of view of the camera 107 during transportation, an adjustment module is provided between the camera 107 and the fixed base 101 in this application. The adjustment module includes a lifting adjustment component 102 and a front-to-back adjustment component, wherein the lifting adjustment component 102 is used to adjust the height of the camera 107 in the vertical direction, that is, to adjust the upper and lower positions of the center of the field of view of the camera 107, and the front-to-back adjustment component is used to adjust the front and back positions of the camera 107 in the horizontal direction. The front and back positions here refer to the distance between the camera 107 and the 45° reflector 109, that is, to adjust the focal length of the field of view of the camera 107.

[0065] The present application further provides an air control system 110 above the camera 107 , and the air control system 110 is used to drive the entire visual mechanism to move in three-dimensional space, so that the visual mechanism can move to the position for crystal grain absorption and placement.

[0066] It is understandable that the above embodiments only express the preferred implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the patent scope of the present invention. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present invention, the above technical features can be freely combined, and several deformations and improvements can be made, all of which fall within the scope of protection of the present invention. Therefore, all equivalent changes and modifications made to the scope of the claims of the present invention should fall within the scope of coverage of the claims of the present invention.

Claims

1. A grain sorting method based on machine vision, characterized in that: The steps include: S1: Create grain template; S2: Camera calibration; wherein, there are two cameras, and the fields of view of the two cameras correspond to the mother film and the daughter film respectively; S3: The camera scans the grain on the mother film to obtain a scanned image of the grain, and the scanned image includes the mechanical coordinates of the grain; S4: Matching the mechanical coordinates of the grains with the grid coordinates one by one to obtain the type information of the grains; the grid coordinates refer to the coordinates with the center point of the master film as the origin, and the grid coordinates mark the type of the grains at each position of the master film; wherein the mechanical coordinates of the grains are the coordinate values of the grains in the mechanical coordinate system of the grains, and the mechanical coordinate system of the grains takes the upper left corner vertex of the image plane as the origin, and the X-axis and Y-axis are respectively parallel to the two vertical sides of the image plane; S5: Classify the grains and move the grains with the same type of information into the same sub-membrane; The step S5 uses a transport assembly to transport the grains, the transport assembly including a suction nozzle; classifying the grains and transporting the grains with the same type of information to the same sub-film includes: during the transport process, the two cameras and the transport assembly are in communication with the control unit, the camera corresponding to the mother film monitors the position of the suction nozzle in the transport assembly in real time to ensure that the suction nozzle is aligned with the center of the grain to be transported, and the control unit then controls the transport assembly to suck the grains; During the transport process, the camera corresponding to the sub-film monitors the position of the suction nozzle in the transport component in real time to ensure that the grain in the suction nozzle is aligned with the position to be transported. The control unit then controls the transport component to place the grain.

2. The grain sorting method based on machine vision according to claim 1, characterized in that: Creating a grain template in step S1 includes creating M grain templates, each of which stores characteristics of a grain type; M is an integer greater than 0.

3. The grain sorting method based on machine vision according to claim 1, characterized in that: When the camera scans the grains on the mother film in step S3, the camera is fixed and the mother film moving component drives the mother film to move in a plane at the same distance from the camera to obtain complete mother film surface scanning information.

4. The grain sorting method based on machine vision according to claim 1, characterized in that: Step S4 also includes performing AOI inspection on the scanned image to obtain defect information of the grains.

5. The grain sorting method based on machine vision according to claim 1, characterized in that: The process also includes S6: inspecting the crystal grains fixed in the sub-membrane; wherein the inspection is to check whether the crystal grains are firmly placed in the sub-membrane.

6. The grain sorting method based on machine vision according to claim 1, characterized in that: The mother membrane and the daughter membrane are arranged opposite to each other.

7. The method for grain sorting based on machine vision according to claim 6, characterized in that: Two cameras are symmetrically and parallelly arranged between the mother film and the daughter film. A lens barrel is provided in the camera, and a 45° reflector is provided in front of the lens barrel. The angle between the mirror surface of the 45° reflector and the axis of the lens barrel is 45°; and the mirror surface of the 45° reflector faces the outside of the center of the line connecting the two lens barrels.

Images