Machine vision system with computer-generated virtual reference

Abstract

A machine vision system using a computer generated virtual reference is disclosed. When the accurate measure of the virtual reference is known, the machine vision system can measure the difference in absolute scale. In a machine vision system, a virtual reference is generated based on computer mapping. The computer-generated virtual reference is further processed, thereby achieving high realistic precision. The processing may include combining multiple portions of the image from the manufactured object or generating, by calculation, an image that looks like a measured object. When the computer-generated virtual reference is based on the model graph, it does not include inaccuracies caused by manufacturing tolerances.

Description

[0001] This application is a divisional application of application number 2018800810397, filed on December 14, 2018, with the title “Machine vision system with computer generated virtual reference”. BACKGROUND

[0002] The following disclosure relates to machine vision systems. In particular, the disclosure relates to measurement machine vision systems used in quality control or other similar tasks requiring measurement of objects. More specifically, the disclosure relates to a machine vision system using a reference object. Computer controlled machine vision systems are used in various applications. One typical application is quality control of objects manufactured in manufacturing. By imaging the manufactured object using one or more cameras, various properties of the object can be measured. The measurement can involve measuring the entire object or some selected features of the object. Thus, depending on the selected features, the measurement can be one-, two- or three-dimensional, or even combined dimensions can be performed. In addition to dimensions and shapes, other properties can be measured, such as color, roughness or other such features. The measurement of the machine vision system is typically performed by comparing the manufactured object to a model object. The result obtained typically gives the relative difference of the measured object to the model object used.

[0003] Measuring three-dimensional coordinates requires only two cameras, since three-dimensional coordinates can be calculated from two two-dimensional images, provided that the measured point is visible in both images. However, typically the number of cameras is larger. This is due to the fact that a larger number of cameras increases the coverage and accuracy of the measurement. The cameras are typically positioned so that they can see all features of the measured object, or at least as many features as possible. Correspondingly, not all cameras can see the measured features. In addition to increasing the number of cameras, many other concepts are known, such as precise calibration and image processing algorithms, to improve the measurement accuracy. In addition, the camera positions can be planned for a specific object, or more accurate cameras or specific lighting is used to improve the quality of the images acquired from the desired features.

[0004] Measurement machine vision systems are particularly good at recognizing features under test similarly under different conditions. Thus, when measuring an object, features such as edges and holes will be similarly detected even if the conditions change. From this, even small changes in the position or shape of an object can be accurately measured. Although the measurements derived from the acquired images are accurate, they cannot be compared to measurements taken with other measurement tools, such as coordinate measuring machines. This is because it is difficult to measure the absolute position of, for example, an edge using conventional machine vision systems and methods. While it is possible to accurately measure the relative changes in size, position, or other changes of an object under test, it is difficult to measure the same changes in absolute scale, rather than relative differences.

[0005] In conventional solutions, these measurements are sometimes supplemented by accurately measuring the position of the object under test or by placing the object under test in a measurement jig so that the exact position is known. When the position is known exactly, at least some absolute measures of the object to be measured can be measured. One method is to make a reference object ("golden object") that is manufactured as accurately as possible to meet the nominal dimensions of the object. Another method is to accurately measure the reference part using an absolute reference measurement system and add the measured difference to the reference part value, resulting in a result comparable to the absolute scale.

[0006] However, these methods can be problematic if different types of objects or a large number of objects need to be measured. If the object under test needs to be accurately positioned before the measurement can be made, the measurements will be too slow. Correspondingly, if different types of objects need to be measured, different types of jigs or other positioning means that can need to be changed between measurements can also be needed. All of these mechanical methods are expensive and prone to wear. Even if the use of mechanical jigs is avoided by appropriate mathematical or optical positioning methods, it is still necessary to purchase and use inexpensive absolute reference measurement systems. SUMMARY

[0007] A machine vision system using a computer-generated virtual reference object is disclosed. When the exact measures of the virtual reference object are known, the machine vision system is able to measure differences on an absolute scale. In the machine vision system, the virtual reference object is produced on the basis of computer graphics. The computer-generated virtual reference object is further processed in order to achieve high photorealistic accuracy. The processing can involve combining multiple parts from images of manufactured real objects or generating images that look like real objects under test by calculation. When the computer-generated virtual reference object is based on a model drawing, it does not include inaccuracies caused by manufacturing tolerances, but it includes all the features and properties of the object as it was designed.

[0008] In one aspect, a method for measuring an object is disclosed. The method includes receiving a computer generated three-dimensional virtual reference, wherein the received virtual reference is generated based on a model drawing of the object, and the virtual reference includes precise coordinates of the object, wherein the precise coordinates include coordinates of at least one discrete point; acquiring at least two images of the object, wherein the acquired at least two images are acquired using at least two different vision sensors; determining three-dimensional positions of the at least one discrete point on the object based on the acquired images, wherein the determined three-dimensional positions are in the same coordinate system as the computer generated three-dimensional virtual reference; determining corresponding discrete point coordinates on the virtual reference; and calculating absolute scale positions of the at least one discrete point on the object based on the determined positions of the at least one discrete point on the acquired images and the corresponding precise coordinates on the virtual reference.

[0009] The method as described above facilitates measuring the absolute scale of an object without having a precisely manufactured reference. The above method eliminates all problems associated with wear and tear of the reference. The above method also helps eliminate inaccuracies caused by measurement tolerances. This provides improved measurement quality while reducing costs.

[0010] In one embodiment, the method further includes generating the computer generated virtual reference based on the model drawing of the object. In one embodiment, the generating further includes receiving additional information including at least one of: lighting setup information, object material information, object color information, or vision sensor parameters. The vision sensor parameters include camera coordinates and orientation. In another embodiment, the generating further includes acquiring at least one image of the manufactured object; generating a projection view based on the model drawing, wherein the projection view corresponds to a direction in which the at least one image of the manufactured object is acquired; and aligning at least a portion of the at least one acquired image on the generated projection view.

[0011] In another embodiment, the generating further includes generating a realistic image of the object based on the model drawing and the received additional information.

[0012] In one embodiment, the above method is implemented as a computer program including computer program code configured to perform the above method when the computer program is executed in a computing device.

[0013] In one embodiment, a controller comprising at least one processor and at least one memory is disclosed. The at least one processor is configured to perform the method as described above. In another embodiment, a machine vision system is disclosed. The machine vision system comprises: a housing; a camera system comprising a plurality of cameras positioned inside the housing; an illumination system comprising a plurality of illumination devices positioned inside the housing; and a controller as described above, wherein the controller is connected to the machine vision system and configured to perform the method as described above.

[0014] Using a computer-generated virtual reference object provides a number of benefits. Computer generation can be performed without involving manufacturing tolerances, thus, the computer-generated virtual reference object matches the plan for the object exactly without the need to manufacture an expensive reference object and accurately measure the manufactured reference object. Thus, using a computer-generated virtual reference object increases the accuracy of the measurement.

[0015] Another benefit of using a computer-generated virtual reference object is that when the exact measure of the absolute scale is known, the difference observed in the measurement can be easily calculated on the absolute scale. Another benefit of this is that when the reference object does not need to be measured, inaccuracies caused by measurement tolerances can also be avoided.

[0016] Traditional real reference objects are also prone to mechanical wear and tear and other issues caused by exposure to the manufacturing site environment. For example, the person performing the measurement can knock the reference object, which can cause scratches and other mechanical defects. Furthermore, sometimes dust and other impurities can make the object appear different. In some cases, even exposure to sunlight can be a source of visible changes to the reference object. All of these light exposure related defects can be avoided by using a computer-generated virtual reference object. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are incorporated in and form a part of the specification, illustrate various embodiments and together with the description serve to explain the principles of the machine vision system. In the drawings: Figure 1 is an example of a machine vision system; Figure 2 is an example of a method of a machine vision system; Figure 3 is an example of a method for generating a computer-generated virtual reference object; Figure 4 is an example of a method for generating a computer-generated virtual reference object. DETAILED DESCRIPTION

[0018] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings.

[0019] In the following, a first measurement comprising a computer-generated virtual reference will be discussed. Subsequently, two alternative methods for preparing a computer-generated virtual reference will be discussed. A computer-generated virtual reference should be understood as one or more reference images (also referred to as reference views), which are computer-generated views of the virtual reference. Thus, a computer-generated virtual reference should not be understood as one reference view, but as a collection of one or more reference views, which show the object from various angles and possibly with different parameters. Furthermore, a computer-generated reference model can also have several reference views from the same angle, but with different lighting or other settings. Furthermore, the skilled person will understand that a computer-generated virtual reference does not have to be a complete object. It is sufficient as long as the features to be measured (region of interest) are covered.

[0020] In the following description, a machine vision system comprising a plurality of cameras is discussed. However, the expression camera is only used to provide an understanding, as a conventional digital camera is usually suitable for this purpose. Instead of a conventional camera, other types of vision sensors can also be used, which are capable of producing images suitable for comparison. These vision sensors include different types of specialized cameras, such as thermal cameras, scanners, digital X-ray imaging devices, bendable imaging units, three-dimensional cameras, etc.

[0021] In Figure 1 a block diagram showing an example of a machine vision system is disclosed. In Figure 1 a measurement station 102 is disclosed. The measurement station 102 comprises four cameras 100a-d and three lighting devices 101a-c. The number of cameras and lighting devices is not limited to four and three, but can be freely chosen. Usually, the number of cameras and lighting devices is higher. The cameras and lighting devices can be attached to one or more frames, which are further attached to the measurement station 102. Instead of frames, the cameras and lighting devices can also be directly attached to the walls of the measurement station 102. The cameras are calibrated to a chosen coordinate system using conventional calibration methods, and if needed, the lighting devices are also calibrated to the chosen coordinate system.

[0022] The measurement station 102 further comprises a conveyor 104 for carrying the object 103 to be measured inside the measurement station. The conveyor is only an example. The object to be measured can also be carried by using other means, such as an industrial robot, or it can be placed by a person performing the measurement.

[0023] In this specification, the ambient light is assumed to be the lighting conditions of the hall or house in which the measurement station is located. The ambient light can be natural light from windows or lighting devices in the house. Advantageously, the measurement station 102 can be closed so that the ambient light does not disturb the measurement, however, this is not essential. For example, if the measurement benefits from a precisely defined lighting arrangement, the ambient light can be compensated. Using a powerful lighting arrangement, even some leaked ambient light can cause some variation in the measurement conditions, still the measurement station 102 can be used. If a conveyor is used, the measurement station 102 can be closed, for example, by using a door or curtain at the conveyor opening. If the object to be measured is placed on the measurement platform by a person, it is easy to make a sealed measurement station in which the ambient light is completely eliminated. If the ambient light cannot be completely eliminated, additional lighting devices for compensating the ambient light can be used.

[0024] The measurement station 102 is connected to the controller 105 using a network connection 108. The network connection can be wired or wireless. The controller can be arranged at the measurement station or it can be in a remote location. If the controller 105 is located at the measurement station 102, the controller can be operated remotely, for example, from a control room for controlling a plurality of systems of a manufacturing site. The controller 105 comprises at least one processor 106 and at least one memory 107. The processor is configured to execute computer program code in order to perform the measurement. The at least one memory 107 is configured to store the computer program code and related data, for example, acquired measurement images and reference views. The controller 105 is typically connected to further computing devices, for example, for possible long-term storage of measurement images and measurement conditions.

[0025] The measurement station 102 can be used as described in the examples below with reference to Figures 2-4 . Figure 2 Examples of methods using a computer generated virtual reference are disclosed. Figure 3 and Figure 4 Two examples of methods for generating a reference view are disclosed. The benefit of using a computer generated virtual reference is that the dimensions of the reference are precisely scaled. This is due to the fact that there is no manufacturing of the virtual reference, thus there is no inaccuracy caused by manufacturing tolerances. The person skilled in the art will understand that the presented examples are merely examples and other similar principles can be used in measurements utilizing a computer generated virtual reference.

[0026] In Figure 2 , an example of a method is disclosed. In the method, a measurement station, such as Figure 1of a measurement station or similar measurement station. In the measurement, first, at least one computer generated virtual reference is received in step 200. The computer generated virtual reference can be received in the form of a two-dimensional projection view, which can easily be compared to images acquired using a regular camera. However, the model can also be three-dimensional, so that a two-dimensional projection view can be created from the three-dimensional model, or first the three-dimensional positions of the discrete features of the comparison are calculated, which are then compared to the computer generated reference. Similar principles can also be used for cameras containing two or more lenses. These cameras are often referred to as three-dimensional cameras or stereo cameras. In this method, only the necessary views of the computer generated virtual reference need to be received. For example, when only one feature is measured, it can be enough to have only one reference. However, there are usually multiple reference views corresponding to different camera views and possible object orientations.

[0027] The computer generated virtual reference can be associated with optional related settings, which are also received in step 201. For example, if the computer generated virtual reference contains a specific lighting setting, for example, using a subset of the available lighting devices, when the optional settings are received, the lights of the subset can be activated. The computer generated virtual reference can be generated using specific lighting devices, and it is beneficial to use the same lighting devices when measuring, because this will provide a better correspondence between the compared images.

[0028] After receiving the reference views and possible optional settings, the measurement station is ready to receive the first object to be measured in step 202. This can be done, for example, using a conveyor belt, a measurement person, a robotic device or any other means for placing the object to the measurement platform of the measurement station.

[0029] In step 203, the object is measured by acquiring a plurality of images. Subsequently, in step 204, the images are compared to the corresponding reference views. The comparison can be similar to a regular comparison, where the result achieved is a relative difference. However, since the exact measure of the computer generated virtual reference is known, it is also possible to calculate the absolute measure of the measured object.

[0030] In this application, an absolute scale or absolute measure means an arrangement, in which the measure can be expressed in precise units, such as nanometers, millimeters or meters in the metric system.

[0031] When measuring an object, the position of discrete points on the object is determined and compared to a computer generated virtual reference. In this application, discrete points mean points or features on the object. These points or features include, for example, holes, grooves, edges, corners and similar points that have a precise position on the object and that precise position has been determined by a designer. Traditionally, machine vision systems use point clouds that are projected on the object. These point clouds are not discrete points on the object because their position is unknown with respect to the object and they are not part of the object.

[0032] When deriving the discrete points from the computer generated virtual reference, it is known that they are in the ideal correct position because there is no deviation due to manufacturing tolerances.

[0033] In the following, two different methods for providing a computer generated virtual reference are disclosed. However, the following methods should be considered as method examples and any other method for generating a computer generated virtual reference can be used.

[0034] In Figure 3 In the following, two different methods for providing a computer generated virtual reference are disclosed. However, the following methods should be considered as method examples and any other method for generating a computer generated virtual reference can be used.

[0035] In step 300, the starting point for generating the computer generated virtual reference is receiving a model drawing of the object to be measured. Typically, the drawing is a CAD drawing or other similar computer aided design tool drawing. The object to be measured can be any object that can be measured using a computer controlled machine vision system. Typically, these mechanical parts are used as parts of cars, mobile phones, home appliances and any other device that contains mechanical parts. The model drawing includes the coordinates of the object. The coordinates of the drawing can be two dimensional or three dimensional. The coordinates define the dimensions in a coordinate system that corresponds to the coordinate system used in the measurement exactly.

[0036] Subsequently, in step 301, the object manufactured according to the model drawing is received. The object does not have to be a so called golden object but a normal object that has inaccuracies of the manufacturing process. Therefore, it is known that the object is not ideal and has deviations from the ideal dimensions shown in the drawing.

[0037] In step 302, a projection model is generated using the drawing of the received object. The projection is typically a two dimensional view of the object. In order to generate the projection view, the viewing direction and other parameters related to the vision sensor have to be known. These parameters include, for example, the vision sensor position, rotation angle, focal length, lens errors and other parameters that can be necessary to generate a view that corresponds to a camera image. When using the same parameters for taking the image and for producing the view, the end result is in the same coordinate system and can be compared to each other without any further processing. However, further image processing can be used in some applications.

[0038] The process further comprises, in step 303, acquiring one or more images corresponding to the generated projection pattern. The acquired images preferably comprise the object in conditions corresponding to the actual measurement conditions.

[0039] Finally, the acquired images and the projection model are combined to select at least a part of the images and to accurately align the selected images on the projection model. This alignment can be fully automatic, however, it is also possible to manually align the necessary parts by hand. To be sufficiently accurate, the acquired images and the projection model can be scaled so that the alignment process can be performed very accurately. The result of the combination can be used as a computer generated virtual reference.

[0040] Measurement settings are not discussed above, however, they can be stored together with the computer generated virtual reference. Thus, the same illumination can be used at the measurement as was used for acquiring the images to generate the computer generated virtual reference.

[0041] In Figure 4 Another method for generating a computer generated virtual reference is disclosed. The method starts by receiving a model of the object to be measured in step 400. This step is at least partly similar to step 300 of Figure 3 The received model can be a set of points indicated by coordinates that are interconnected to each other in order to create the object.

[0042] In order to provide the computer generated virtual reference, further object properties are also received in step 401. These further properties comprise for example the manufacturing material, the color of the object, etc. Thus, it is possible to determine how the object reflects light and how it appears on the camera.

[0043] As mentioned above, a typical measurement station usually comprises a plurality of independently controllable lighting devices. It is also possible to control the intensity, wavelength and other properties of the light. In order to generate a computer generated reference model that can be compared to the images taken by the camera, it is beneficial to know which lighting setup is used. Thus, it is beneficial to receive the lighting setup to be used in step 402.

[0044] When generating the computer generated virtual reference, it is necessary to know the viewing angle. Thus, as in the method of Figure 3 It is necessary to know the information of the camera that is used to image the object to be measured. This can for example be achieved by receiving a camera identification in step 403 that can be used to retrieve the camera position and orientation.

[0045] When the model drawing and the additional properties are known, a computer generated virtual reference can be generated at step 404. The computer generated virtual reference should be as realistic as possible visually. This can be achieved using rendering techniques such as scan line rendering or ray tracing, which are methods for visual surface determination. Ray tracing is a technique for generating images by tracing the path of light through the pixels in the image plane and simulating the effect of its meeting a virtual object, the generated image can be used as the computer generated virtual reference. Ray tracing is capable of producing a high degree of visual realism and is suitable for the purpose. Ray tracing techniques are computationally demanding. However, since the computer generated virtual reference and the image representing the computer generated virtual reference can be pre-computed, the expensive computation can be done at a computing center or similar, without the need for it to be done by the measuring station. Thus, a high accuracy of the model object can be achieved compared to conventional solutions, without the need for increased computational power.

[0046] In the above examples, the camera and the lighting device can be conventional cameras and lighting devices, however, without being limited thereto. Thus, dedicated cameras and dedicated lighting devices designed for specific wavelengths can be used. Thus, the light does not need to be visible to the human eye. Thus, the skilled person will understand that the computer generated virtual reference is provided as visible to the camera. For example, in some measurements, it can be useful to use ultraviolet or infrared wavelengths, which are not visible to the human eye. Similar principles apply to other visual sensor types, such as scanners or other measuring devices producing images or measurements which can be compared to the computer generated virtual reference. Thus, the expression "image" should be understood broadly to cover images generated with various image generating devices which can view the object differently.

[0047] In the above examples, the use of the computer generated virtual reference is disclosed. In the measurement, the computer generated virtual reference is used like a conventional reference.

[0048] When measuring the three-dimensional discrete points of the computer generated virtual reference, for each point k x 3 dimensional matrix V wherein k represents the number of measured points. The real object which is measured has a similar matrix R However, these matrices are not in the same position or even in the same coordinate system without positioning. Thus, the difference cannot be calculated directly by calculating R-V. However, since the purpose is to obtain an absolute scale result in the form of R it is natural within the available measurement tolerances, for example, by using the following or any other suitable similar method, to be sufficient to obtain R .

[0049] The problem of positioning can be solved by using conventional positioning methods. One example is the conventional 3-2-1 method. Another commonly used option is to use a best fit method. These positioning methods are merely examples, and other methods are available. The available methods can have different accuracies, and this needs to be taken into account when selecting a method.

[0050] To provide a better understanding, a method of calculating coordinates is disclosed in the following paragraphs. Let the coordinates of a single measurement point or feature on the acquired two-dimensional image be: V k = { x 1vk ,y 1vk ,x 2vk ,y 2vk ,… x nvk ,y nvk} where n is the number of the camera and k is the number of the measurement point. Thus, it is possible to form k x 2n dimensional matrix V where k is also the number of the measurement point and n is the number of the camera. Accordingly, a matrix of the same dimensionality can be generated D where each row is in the form: {x 1dk ,y 1dk ,x 2dk ,y 2dk ,… x ndk ,y ndk} where n is also the number of the camera and each value depicts the difference between the real measurement point and the virtual measurement point in the two-dimensional plane. Accordingly, the coordinates of the measured two-dimensional coordinates are shown in matrix form M = V + D on a graph, where M has the same dimensionality as V and D and each row has the same form { x 1mk ,y 1mk ,x 2mk ,y 2mk ,… x nmk ,y nmk}. The absolute scale three-dimensional matrix R is k x 3 matrix, where each row Xk, Yk and Zk is a function of the rows of the matrix M R k = {X k ​Y k Z k} = f k (x) 1mk y 1mk x 2mk y 2mk , ... x nmk y nmk ) therefore, R = f( M ) The above method can be implemented as computer software including computer program code that executes in a computing device capable of communicating with external devices. When the software executes in the computing device, it is configured to perform the above-described inventive method. The software is implemented on a computer-readable medium such that it can be provided to a computing device, such as... Figure 1 The controller 105.

[0051] As described above, components of the exemplary embodiments may include computer-readable media or memory for storing instructions for programming according to the teachings of the present invention and for storing data structures, tables, records and / or other data described herein. Computer-readable media may include any suitable medium that participates in providing instructions to a processor for execution. Common forms of computer-readable media may include, for example, floppy disks, floppy disks, hard disks, magnetic tape, any other suitable magnetic media, CD-ROMs, CD±Rs, CD±RWs, DVDs, DVD-RAMs, DVD±RWs, DVD±Rs, HD DVDs, HD DVD-Rs, HD DVD-RWs, HD DVD-RAMs, Blu-ray discs, any other suitable optical media, RAM, PROMs, EPROMs, FLASH-EPROMs, any other suitable memory chips or cassette tapes, carrier waves, or any other suitable medium from which a computer can read.

[0052] Those skilled in the art will understand that, with advancements in technology, the basic idea of ​​machine vision systems can be implemented in various ways. Therefore, machine vision systems and their embodiments are not limited to the examples described above; rather, they may vary within the scope of the claims.

Claims

1. A method for measuring an object, comprising: receiving (200) a computer generated three-dimensional virtual reference, wherein the received virtual reference is generated based on a model drawing of the object, and the virtual reference comprises accurate coordinates of the object, wherein the accurate coordinates comprise coordinates of at least one discrete point; acquiring (203) at least two images of the object, wherein the acquired at least two images are acquired with at least two different cameras; determining (204) three-dimensional positions of the at least one discrete point on the object based on the acquired images, wherein the determined three-dimensional positions are in the same coordinate system as the computer generated three-dimensional virtual reference; determining (204) corresponding discrete point coordinates on the virtual reference; and calculating (205) absolute scale positions of the at least one discrete point on the object based on the determined positions of the at least one discrete point on the acquired images and the corresponding accurate coordinates on the virtual reference, wherein the method further comprises generating (300-304) the computer generated virtual reference based on the model drawing of the object, wherein the method further comprises: acquiring at least one image of a manufactured object; generating a projection view based on the model drawing, wherein the projection view corresponds to a direction of acquiring at least one image of a manufactured object; and aligning at least a portion of the acquired at least one image on the generated projection view.

2. The method of claim 1, wherein, The generating further comprises receiving (201) additional information, the additional information comprising at least one of: lighting setting information, object material information, object color information, or camera parameters.

3. The method of claim 2, wherein, The camera parameters comprise camera coordinates and camera orientation.

4. The method of any one of claims 2 or 3, wherein, The generating further comprises generating a realistic image of the object based on the model drawing and the received additional information.

5. A computer program comprising computer program code configured to perform the method according to any one of the preceding claims 1 to 4 when the computer program is executed on a computing device.

6. A controller comprising at least one processor (106) and at least one memory (107), wherein, The at least one processor is configured to perform the method according to any one of the preceding claims 1 to 4.

7. A machine vision system, comprising: a housing (102); a camera system comprising a plurality of cameras (100a ~ 100d) inside the housing (102); a lighting system comprising a plurality of lighting devices (101a ~ 101c) inside the housing (102); and a controller (105), wherein the controller is configured to perform the method according to any one of the preceding claims 1 to 4.

Images