Flexible and intuitive configuration system for automatic vision inspection systems

Abstract

The present application mainly discloses a flexible and intuitive configuration system, which is realized by an electronic device coupled with a master control electronic device of an automatic visual inspection system, thereby being information-linked with N cameras and N mechanical arms of the automatic visual inspection system. When a setting procedure is performed on the automatic visual inspection system, an arm setting operation is first performed on each of the mechanical arms, and a camera setting operation is performed on the camera held by the mechanical arm, and then the camera is controlled to shoot a specified object. In the process, the flexible and intuitive setting system correspondingly records the configuration parameters of the camera and the mechanical arm, and uploads multiple object images of the object to a remote electronic device, so that the remote electronic device generates an object defect identification model by using the multiple object images. Finally, the object defect identification model is installed in the master control electronic device.

Description

TECHNICAL FIELD

[0001] The present invention relates to the technical field of machine vision, and in particular to a flexible and intuitive configuration system for an automated visual inspection system. BACKGROUND

[0002] Generally speaking, a semi-finished product or a finished product manufactured by industrial technology must undergo quality inspection to determine whether it meets the industrial standards, wherein the quality inspection includes cosmetic inspection and functional testing. Practical experience shows that when a product inspector performs cosmetic inspection on an industrial product, he or she often misses or misidentifies defects. Therefore, an automated visual inspection system using machine vision technology is developed and introduced into an automated production line to replace human labor for performing cosmetic inspection on industrial semi-finished products or finished products. For example, Taiwan Utility Model Patent No. TWM355371U discloses an automated visual inspection system for detecting defects on a nut or a gear, and the automated visual inspection system mainly includes a light source, at least one camera, and a processing device (or an electronic device).

[0003] Generally speaking, an industrial product must undergo abnormality inspection on multiple angles of the product itself before it is shipped. At the same time, the market trend of industrial products is moving towards small quantities and great varieties, and the production cycle is short. Therefore, the appearance of different models of industrial products changes rapidly, and the setting frequency of the automated visual inspection system on the production line also needs to be changed frequently by personnel intervention due to the rapid change in the appearance of the products. For example, when a gear is inspected for defects, a product inspector checks several surfaces of the gear according to relevant inspection regulations to determine whether the gear has defects such as adhered foreign matter, dents, scratches, discoloration, and no threading (i.e., no internal threads in the screw hole). Therefore, in order for the automated visual inspection system to correctly detect defects on a batch of gears conveyed by a conveyor belt, the system manufacturer usually sets a database and a defect recognition software in the electronic device. It is worth noting that the database pre-stores a plurality of standardized defect gear images and a golden sample image, and these pre-classified gear images are processed into a plurality of reference defect feature images having adhered foreign matter, dents, scratches, discoloration, and no threading. With this setting, after the camera captures at least one gear image from a gear, the defect recognition software performs a feature extraction process on the gear image to obtain a gear feature image. Then, the defect recognition software compares the gear feature image with at least one reference defect feature image stored in the database to determine whether the currently inspected gear has defects defined by the relevant inspection regulations.

[0004] Engineers familiar with the development and manufacture of automatic visual inspection systems applied to defect detection know that in order to enable the automatic visual inspection system to make correct defect detection and identification of gear of model A, the engineers must not only calibrate the setting height, setting angle, camera distance, aperture size and other camera parameters of each camera, but also calibrate the light range and light intensity of the light source. More complexly, when the quality inspectors of the gear manufacturer find new types of defects in the production process, they may not be able to use the previously calibrated visual system parameters to take images due to different defect locations or required optical conditions. Therefore, the quality inspectors cannot provide the engineers with multiple gear images with new types of defect characteristics at the first time to enable the engineers to make the aforementioned pre-classified gear images. Assuming that the defects can be successfully captured by the camera with pre-configured parameters, the engineers can then use machine learning technology to construct a defect identification model included in the defect identification software. The defect identification model is generated using the following steps:

[0005] Step a: inputting multiple sample images into a machine learning model to obtain a predicted class information of each of the sample images; wherein the multiple sample images include multiple gear images and multiple gear images with defect characteristics, and each of the sample images is labeled with a class label;

[0006] Step b: adjusting model parameters of the machine learning model according to a difference between the predicted class information of the sample images and the class label, thereby obtaining an adjusted machine learning model;

[0007] Step c: classifying multiple test images using the adjusted machine learning model to obtain a classification accuracy; wherein the multiple test images include multiple gear images and multiple gear images with defect characteristics;

[0008] Step d: repeating all the foregoing steps in the case where the classification accuracy does not reach an accuracy threshold; and

[0009] Step e: in the case where the classification accuracy reaches the accuracy threshold, using the adjusted machine learning model as a defect identification model integrated in the defect identification software.

[0010] In other words, if the gear manufacturer requires the engineer to re-calibrate the automatic vision inspection system to enable it to be used for correct defect detection and identification of a model B gear, the engineer has to calibrate not only the setting height, setting angle, camera distance, aperture size and other camera parameters of each camera, but also the light range and intensity of the light source. More importantly, the quality inspector of the gear manufacturer, without the knowledge of the optical equipment, cannot provide multiple images of the gear with the defect features in the continuous production process by adjusting the optical equipment, so that the engineer cannot make the corresponding standardized gear images and defect identification model. Therefore, to support multiple models of gear detection, the early optical setting work is not easy to complete, and it takes considerable professional manpower and time to complete.

[0011] Therefore, a new automatic vision inspection system should be developed, which allows the quality inspector to set the relevant parameters of the light source and camera, and can learn the quality inspection standards of different models of gears under the flexible and intuitive operation of the quality inspector, to train the defect identification model to correctly detect and identify the appearance defects of different models of gears. Therefore, the present inventor has made great efforts to research and develop a flexible and intuitive configuration system for an automatic vision inspection system. SUMMARY

[0012] The main purpose of the present application is to provide a flexible and intuitive configuration system, which is realized by an electronic device coupled to a master control electronic device of an automatic vision inspection system, so as to be linked with N cameras and N mechanical arms of the automatic vision inspection system. When a setting program is performed on the automatic vision inspection system, a hand arm setting operation is performed on each mechanical arm, and a camera setting operation is performed on the camera held by the mechanical arm, and then the camera is controlled to shoot a specified object. During the process, the flexible and intuitive configuration system records the configuration parameters of the camera and the mechanical arm, and uploads multiple images of the object to a remote electronic device, so that the remote electronic device generates an object defect identification model using the multiple images of the object. Finally, the object defect identification model is installed in the master control electronic device, so that the automatic vision inspection system is suitable for automatic defect detection of the object.

[0013] For example, the flexible and intuitive configuration system of the present application can be applied to an application program installed in a tablet computer (i.e., the aforementioned electronic device), so that a quality inspector of a manufacturer can operate the tablet computer to perform the aforementioned setting procedure on any automatic visual inspection system in his factory, so that the automatic visual inspection system can be reconfigured and then used to automatically detect defects of a designated product. In particular, during the entire process of the setting procedure, the quality inspector can intuitively adjust the angle of the robotic arm according to his experience in manual quality inspection, and can complete the image capturing of the defect sample through the photographing mode of the smart handheld device. The engineer only needs to operate the remote electronic device to generate a product defect recognition model according to the multiple images of the product provided by the quality inspector.

[0014] To achieve the above-mentioned purpose, an embodiment of the flexible and intuitive configuration system is proposed to perform a setting operation on an automatic visual inspection system so that the automatic visual inspection system is suitable for being applied to automatically detect defects of a product, wherein the automatic visual inspection system includes a conveying device, N robotic arms, N cameras respectively held by the N robotic arms, and a master electronic device, and N is at least a positive integer of 1; the flexible and intuitive configuration system includes:

[0015] An electronic device coupled to the master electronic device, so as to be information-linked with the N cameras and the N robotic arms through the master electronic device; wherein the electronic device includes a first processor and a first memory, and the first memory stores a first application program, so that the first processor executes the first application program by accessing the first memory, thereby enabling the following functions:

[0016] During the process that the robotic arm is operated to move the camera K times so that the camera has a camera height, a camera angle and a camera distance after each movement, one of the camera height, the camera angle and the camera distance is recorded as a set of external camera parameters after each movement of the camera, thereby obtaining K sets of the external camera parameters; wherein K is at least a positive integer of 1;

[0017] In a case that the camera receives a camera adjustment operation after each movement of the camera to adjust an aperture, a depth of field, a shutter speed, an ISO, and a focus, a recording of one of the aperture, one of the depth of field, one of the shutter speed, one of the ISO, and one of the focus as a set of internal camera parameters, so that K sets of the internal camera parameters are obtained after K movements of the camera;

[0018] In a case that the camera receives a camera adjustment operation after each movement of the camera to adjust an aperture, a depth of field, a shutter speed, an ISO, and a focus, a recording of one of the aperture, one of the depth of field, one of the shutter speed, one of the ISO, and one of the focus as a set of internal camera parameters, so that K sets of the internal camera parameters are obtained after K movements of the camera;

[0019] In a case that the camera receives a camera adjustment operation after each movement of the camera to adjust an aperture, a depth of field, a shutter speed, an ISO, and a focus, a recording of one of the aperture, one of the depth of field, one of the shutter speed, one of the ISO, and one of the focus as a set of internal camera parameters, so that K sets of the internal camera parameters are obtained after K movements of the camera;

[0020] In an embodiment, the electronic device is any one selected from a group consisting of a smartphone, a tablet computer, a desktop computer, an all-in-one computer, and a notebook computer.

[0021] In an embodiment, the camera is included in a mobile electronic device, and the mobile electronic device is any one selected from a group consisting of a smartphone and a tablet computer. Also, the mobile electronic device includes an inertial measurement unit (IMU) and a laser imaging detection and ranging (LiDAR) unit to obtain a three-dimensional movement data and a three-dimensional measurement data after each movement of the camera, and to transmit the three-dimensional movement data and the three-dimensional measurement data to the electronic device through the mobile electronic device, so that the electronic device integrates the three-dimensional movement data and the three-dimensional measurement data into the external camera parameters.

[0022] In another possible embodiment, the camera is included in a mobile electronic device, wherein the mobile electronic device includes an inertial measurement unit (IMU) and is arranged on a clamp connected to the robot arm, and a laser detection and ranging (LiDAR) unit is arranged on the clamp and is electrically connected to the mobile electronic device. After each movement of the camera, the inertial measurement unit and the laser detection and ranging unit obtain a three-dimensional movement data and a three-dimensional measurement data, respectively, and the mobile electronic device transmits the three-dimensional movement data and the three-dimensional measurement data to the electronic device, so that the electronic device integrates the three-dimensional movement data and the three-dimensional measurement data into the external camera parameters.

[0023] In an embodiment, the first processor records and stores N sets of the external camera parameters corresponding to N robot arms in the memory, and records and stores N sets of the internal camera parameters corresponding to N cameras in the memory.

[0024] In an embodiment, after the first processor executes the first application program, a display of the electronic device displays an operation interface, so that a user can operate the operation interface to perform a roll marking process on the multiple images of the first articles with regular features and the multiple images of the second articles with defect features.

[0025] In an embodiment, the conveying device includes a conveyor belt mechanism, a motor for driving the conveyor belt mechanism, and a distance sensor, and the first processor executes the first application program to further enable the following functions:

[0026] In the case where the motor is set to operate at a rotating speed to drive the conveyor belt mechanism, the rotating speed is recorded as a motor control parameter;

[0027] In the case where the conveyor belt mechanism conveys the article to fall within a camera range of the jth camera, the jth movement distance is recorded as an article positioning parameter; wherein j ∈ N; and

[0028] According to the motor control parameter and N article positioning parameters, a parameter update operation is performed on a device control software installed in the master electronic device.

[0029] In an embodiment, the automatic visual inspection system further includes N light sources, and the first processor executes the first application program to enable the following functions:

[0030] In the case where the light source has a light range, a light intensity, a color temperature, and a light color through a light source adjustment operation, the light range, the light intensity, the color temperature, and the light color are recorded as a set of lighting parameters; and

[0031] performing said parameter updating operation on the device control software installed in the host electronic device according to the illumination parameter.

[0032] In one embodiment, the host electronic device comprises a second processor and a second memory storing a second application program, such that the second processor executes the second application program by accessing the second memory, thereby enabling the following functions:

[0033] controlling the mechanical arm to move the camera according to its corresponding external camera parameter;

[0034] controlling the light source to provide a detection light to the article according to its corresponding illumination parameter; and

[0035] controlling the camera to capture an article image from the article according to its corresponding internal camera parameter;

[0036] performing a defect recognition operation on the article image by using the article defect recognition model, thereby determining whether the article has at least one defect.

[0037] In one embodiment, the automatic visual inspection system further comprises at least one ejection device arranged adjacent to an ejection port of the conveying device, and the first processor executes the first application program thereby further enabling the following functions:

[0038] after one of the articles moves N movement distances on the conveying mechanism and then moves an (N+1)th movement distance, recording the (N+1)th movement distance as an ejection parameter;

[0039] in a case where the ejection device is operated to eject one of the articles to a collection box, the article has an initial height, a placement angle, and a final height, recording the initial height, the placement angle, and the final height as a set of ejection device parameters; and

[0040] performing said parameter updating operation on the device control software installed in the host electronic device according to the ejection parameter and N of the ejection device parameters.

[0041] In one embodiment, the automatic visual inspection system further comprises at least one article placement device arranged adjacent to an inlet port of the conveying device, and the first processor executes the first application program thereby further enabling the following functions:

[0042] In the case that the article placing device is operated to place one of the articles onto the conveying device, the article has an initial height, a placement angle and a final height, the initial height, the placement angle and the final height are recorded as an article placing device parameter; and

[0043] The device control software installed in the master electronic device is subjected to the parameter updating operation according to the article placing device parameter. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A block diagram of an automatic vision inspection system including a flexible and intuitive configuration system according to the present application;

[0045] Figure 2A A first perspective view of the automatic vision inspection system;

[0046] Figure 2B A second perspective view of the automatic vision inspection system;

[0047] Figure 3A A first perspective view of the jth robot and the jth camera;

[0048] Figure 3B A second perspective view of the jth robot and the jth camera;

[0049] Figure 4A A first perspective view of the jth robot, the jth camera and the jth light source; and

[0050] Figure 4B A second perspective view of the jth robot, the jth camera and the jth light source.

[0051] BRIEF DESCRIPTION OF DRAWINGS

[0052] 1: flexible and intuitive configuration system

[0053] 11: electronic device

[0054] 11P: first processor

[0055] 11M: first memory

[0056] 2: automatic vision inspection system

[0057] 20: conveying device

[0058] 200: conveying belt mechanism

[0059] 201: motor

[0060] 202: distance sensor

[0061] 21: robot

[0062] 22: camera

[0063] 23: master electronic device

[0064] 23P: second processor

[0065] 23M: second memory

[0066] 24: light source

[0067] 25: discharging device

[0068] 26: article placing device

[0069] 3: remote electronic device

[0070] 4: normal product collection box

[0071] 5: NG product collection box DETAILED DESCRIPTION

[0072] In order to make the flexible and intuitive configuration system of the automatic visual inspection system proposed by the present application more clearly described, the preferred embodiment of the present application will be described in detail below with reference to the drawings.

[0073] Figure 1 is a block diagram of an automatic visual inspection system including the flexible and intuitive configuration system of the present application. Moreover, Figure 2A and Figure 2B are the first and second perspective views of the automatic visual inspection system. As shown in Figure 1 , Figure 2A and Figure 2B , the present application proposes a flexible and intuitive configuration system 1 for performing a setting operation on an automatic visual inspection system 2 so as to make the automatic visual inspection system 2 (hereinafter referred to as "AVI system 2") suitable for being applied to automatically inspecting a product. For example, the AVI system 2 includes a conveying device 20, an article placing device 26, N mechanical arms 21, N cameras 22, a master electronic device 23, N light sources 24, and a discharging device 25, wherein N is at least a positive integer of 1, and the master electronic device 23 is coupled to the conveying device 20, the article placing device 26, the N mechanical arms 21, the N cameras 22, the master electronic device 23, the N light sources 24, and the discharging device 25. In more detail, the master electronic device 23 is installed with a device control software and an article defect recognition software, and the device control software and the article defect recognition software are integrated in an automatic visual inspection application program.

[0074] Therefore, when the AVI system 2 is used to inspect a batch of articles (e.g. gears), the main control electronic device 23 executes the device control software to control the conveyer 20, the article placing device 26, the N number of mechanical arms 21, the N number of cameras 22, the N number of light sources 24, and the discharging device 25 to act at appropriate time points. Further, the article placing device 26 (i.e. a mechanical arm with a gripper) is controlled to pick up an article located on the conveyer 20 and then place the article on the conveyer 20 so that the article is located on the conveyer 20 at a placing angle. Further, the first mechanical arm 21 holding the first camera 22 is controlled to move the first camera 22 so that the camera 22 has a set of external camera parameters including a camera height, a camera angle, and a camera distance. Then, the first camera 22 is controlled to take a picture of the article located within a camera range of the camera 22 to obtain an article image. It is noted that the first mechanical arm 21 and the first camera 22 form a first inspection station. In the first inspection station, the first camera 22 is configured to have a first set of external camera parameters and then take a picture of the article to obtain a first article image. In practice, in the first inspection station, the first camera 22 can also be configured to have a second set of external camera parameters (i.e. different camera height, camera angle, and camera distance) and then take a picture of the article to obtain a second article image. Similarly, the article can also be taken a third, fourth, and so on article images in the first inspection station.

[0075] Similarly, when the article is continuously conveyed by the conveyer 20 to the first inspection station formed by the second mechanical arm 21 and the second camera 22, the second mechanical arm 21 is controlled to move the second camera 22 so that the camera 22 has a set of external camera parameters including a camera height, a camera angle, and a camera distance. Then, the second camera 22 is controlled to take a picture of the article located within a camera range of the camera 22 to obtain an article image. It is noted that in the second inspection station, the second camera 22 is configured to have a first set of external camera parameters and then take a picture of the article to obtain a first article image. In practice, in the second inspection station, the second camera 22 can also be configured to have a second set of external camera parameters (i.e. different camera height, camera angle, and camera distance) and then take a picture of the article to obtain a second article image. Similarly, the article can also be taken a third, fourth, and so on article images in the second inspection station.

[0076] It is to be noted that, in the jth inspection station (j e N), the jth camera 22 is configured with a set of internal camera parameters including Aperture, Depth of field, Shutter speed, ISO, and Focus, such that the camera 22 is capable of taking images of the article under the specified parameters. Thus, after the article has passed through the N inspection stations, the master electronic device 23 has successfully obtained a plurality of images of the article, and then the master electronic device 23 executes the article defect identification software to perform a defect identification operation on each of the images of the article, so as to determine whether the article has at least one defect. Specifically, the article defect identification software includes an image processing unit (e.g. spatial filtering), a feature extraction unit, and an article defect identification unit, wherein the feature extraction unit is configured to perform a feature extraction process on the images of the article to obtain an article feature image. On the other hand, the article defect identification unit has a pre-trained article defect identification model, and utilizes the article defect identification model to perform a feature comparison between the article feature image and at least one reference defect feature image stored in the database, so as to determine whether the article under inspection has a defect defined by the relevant inspection regulation.

[0077] After the article has completed the defect detection and identification, the master electronic device 23 controls the discharging device 25 (i.e. a robotic arm with a gripper) to clamp the article that is a NG (not good) product away from the conveying device 20, and into a NG product collection bin 5. On the contrary, the article that has passed the defect detection (i.e. a normal product) is sent into a normal product collection bin 4.

[0078] As can be seen from the foregoing description, after the master electronic device 23 is installed with the device control software and the article defect identification software, the AVI system 2 including the master electronic device 23, the conveying device 20, the article placement device 26, the N robotic arms 21, the N cameras 22, the master electronic device 23, the N light sources 24, and the discharging device 25 can be used to perform defect detection on a batch of A articles (e.g. model A gears). However, practical experience shows that the AVI system 2 is not necessarily suitable for performing defect detection on a batch of B articles (e.g. model B gears). In this case, as shown in Figure 1 、 Figure 2A and Figure 2B , the product inspector of the manufacturing factory can operate the flexible and intuitive configuration system 1 of the present application to perform a setting procedure on the same AVI system 2, such that the AVI system 2 is capable of being adapted to perform defect detection on a batch of B articles after a series of setting operations.

[0079] AsFigure 1 、 Figure 2A With Figure 2B referring to FIG. 1, the inventive flexible and intuitive configuration system 1 mainly comprises an electronic device 11 and an AVI system setting software (referred to as a first application) installed in the electronic device 11, so that the product inspector of the manufacturer can operate the tablet computer to perform a series of setting operations on any AVI system 2 in the factory. According to the design of the present application, the electronic device 11 is coupled to the master electronic device 23, thereby being information-linked to the N cameras 22 and the N mechanical arms 21 through the master electronic device 23. The electronic device 11 comprises a first processor 11P and a first memory 11M, and the first memory 11M stores a first application (i.e., the AVI system setting software) therein, so that the first processor 11P executes the first application by accessing the first memory 11M, thereby enabling a plurality of functions to complete a series of setting operations of the AVI system 2. In an embodiment, after the first processor 11P executes the first application, a display of the electronic device 11 displays a user interface (UI), so that the product inspector of the manufacturer can operate the user interface to perform the setting program of the AVI system 2.

[0080] First, the product inspector can manually operate the jth (e.g., the 1st) mechanical arm 21, so that the jth camera 22 has a camera height, a camera angle, and a camera distance (i.e., a set of external camera parameters). Then, the product inspector can set the aperture, the depth of field, the shutter speed, the light sensitivity, and the focal length of the jth camera 22 (i.e., a set of internal camera parameters), and then control the jth camera 22 to take a picture of the article falling within its camera range, so as to obtain an article image. It should be understood that the jth mechanical arm 21 and the jth camera 22 constitute the jth inspection station. Figure 3A 、 Figure 3B FIGS. 1A and 1B are first and second perspective views of the jth mechanical arm 21 and the jth camera 22. As shown in FIG. 1A, in the jth inspection station, the jth camera 22 is configured with the 1st set of external camera parameters, and then takes a picture of the article to obtain the 1st article image. In practical applications, as shown in FIG. 1B, in the jth inspection station, the 1st camera 22 is also configured with the 2nd set of external camera parameters (i.e., different camera height, camera angle, and camera distance), and then takes a picture of the article to obtain the 2nd article image. Similarly, the article in the jth inspection station can also be taken the 3rd and 4th article images. Figure 3A Figure 3B

[0081] ​​Therefore, during the process that the mechanical arm 21 is operated to move the camera 22 K times so that the camera 22 has a shooting height, a shooting angle and a shooting distance after each movement, the first processor 11P records one of the shooting height, one of the shooting angle and one of the shooting distance after each movement of the camera 22 as a set of external shooting parameters, so that K sets of the external shooting parameters are obtained and stored in the first memory 11M, where K is at least a positive integer of 1. Similarly, in the case that the camera 22 has an aperture, a depth of field, a shutter speed, a light sensitivity and a focal length after each movement and then receives a camera adjustment operation, one of the aperture, one of the depth of field, one of the shutter speed, one of the light sensitivity and one of the focal length are recorded as a set of internal shooting parameters, so that K sets of the internal shooting parameters are obtained and stored in the first memory 11M after the camera 22 moves K times.

[0082] When the inspector completes the setting of the internal shooting parameters and the external shooting parameters at each inspection station, the first processor 11P records and stores multiple sets of the external shooting parameters corresponding to the N mechanical arms 21 in the memory 11M, and records and stores multiple sets of the internal shooting parameters corresponding to the N cameras 22 in the memory 11M. It should be understood that in the jth inspection station, the jth camera 22 shoots at least one product image of the product. Therefore, after N inspection stations, the electronic device 11 obtains multiple product images, wherein the multiple product images include multiple first product images with regular features and multiple second product images with defect features, and the inspector can label the multiple first product images with regular features and the multiple second product images with defect features by operating the operation interface. Then, the electronic device 11 can upload the multiple first product images with regular features and the multiple second product images with defect features to a remote electronic device 3, so that the remote electronic device 3 generates at least one update module using the multiple first product images with regular features and the multiple second product images with defect features. Finally, the operator can perform a model update operation on the product defect recognition model installed in the master electronic device 23 using the update module, or install another set of product defect recognition model in the master electronic device 23 using the update module.

[0083] Further, Figure 4A 、 Figure 4B The first and second perspective views of the jth mechanical arm 21, the jth camera 22 and the jth light source 24. As shown in FIGS. 2A and 2B, the jth mechanical arm 21 is provided with a first camera 22A and a second camera 22B, and the jth light source 24 is provided with a first light source 24A and a second light source 24B. The first camera 22A and the first light source 24A are arranged on the same side of the jth mechanical arm 21, and the second camera 22B and the second light source 24B are arranged on the other side of the jth mechanical arm 21. The first camera 22A and the second camera 22B are arranged on the same side of the jth mechanical arm 21, and the first light source 24A and the second light source 24B are arranged on the other side of the jth mechanical arm 21. Figure 4A And Figure 4BAs shown, the camera 22 is included in a mobile electronic device, such as a smartphone and a tablet computer, and the mobile electronic device further includes an inertial measurement unit (IMU) and a laser imaging detection and ranging (LiDAR) unit. For example, iPad Pro and iPhone 12 both have a LiDAR unit and an IMU unit. With this design, after each movement of the camera 22, the IMU unit and the LiDAR unit acquire a three-dimensional movement data and a three-dimensional measurement data, and then the mobile electronic device transmits the three-dimensional movement data and the three-dimensional measurement data to the electronic device 11, so that the electronic device 11 integrates the three-dimensional movement data and the three-dimensional measurement data into the external camera parameters.

[0084] However, not all mobile electronic devices contain a LiDAR unit. Therefore, in another possible embodiment, the camera 22 is included in a mobile electronic device, and the mobile electronic device includes an IMU unit and is arranged on a clamp connected to the robot arm 21, and a LiDAR unit is arranged on the clamp and is electrically connected to the mobile electronic device. With this arrangement, after the robot arm 21 drives the clamp and the camera 22 to move, the IMU unit and the LiDAR unit respectively obtain a three-dimensional movement data and a three-dimensional measurement data, and the mobile electronic device transmits the three-dimensional movement data and the three-dimensional measurement data to the electronic device 11, so that the electronic device 11 integrates the three-dimensional movement data and the three-dimensional measurement data into the external camera parameters.

[0085] In summary, the flexible and intuitive configuration system 1 of the present application can be an application installed on a tablet computer (i.e., the aforementioned electronic device 11), so that a product inspector of a manufacturer can operate the tablet computer to perform the aforementioned setting procedure on any AVI system 2 in his factory, so that the AVI system 2 can be reconfigured to be used for automated defect detection on a specified product (e.g., a model B gear) after the reconfiguration. In particular, during the entire process of the AVI system 2 receiving the setting procedure, the product inspector can complete the adjustment of the configuration parameters of the N cameras 22 and the N robot arms 21 by himself according to his experience (i.e., manually operating the robot arms 21 and manually adjusting the camera parameters). On the contrary, the engineer only needs to operate the remote electronic device 3 to generate a product defect recognition model according to the multiple images of the product provided by the product inspector.

[0086] It is to be noted that the electronic device 11 can also be a smart phone, a desktop computer, an all-in-one computer, or a notebook computer. In one embodiment, the camera 22 is included in a mobile electronic device, such as a smart phone or a tablet computer. In another embodiment, the camera 22 is network-enabled. In other words, the camera 22 has a first communication interface for data transmission with a second communication interface of the electronic device 11 or a main communication interface of the host electronic device 23.

[0087] In addition to the model updating operation of the flaw identification model installed in the host electronic device 23, the flexible and intuitive configuration system 1 can also perform a software parameter updating operation of the device control software installed in the host electronic device 23. As shown in Figure 1 、 Figure 2A and Figure 2B The conveying device 20 includes a conveyor mechanism 200, a motor 201 for driving the conveyor mechanism 200, and a distance sensor 202. Thus, the software parameter updating operation can also be performed after the first processor 11P executes the first application. Specifically, in order for the conveyor mechanism 200 to transport N items to the first to N inspection stations, respectively, the inspector can operate the host electronic device 23 to set the rotation speed of the motor 201, so that the motor 201 is operated at a rotation speed to drive the conveyor mechanism 200. Thus, in the case where the motor 201 is set to be operated at a rotation speed to drive the conveyor mechanism 200, the first processor 11P records the rotation speed as a motor control parameter and stores it in the first memory 11M. Meanwhile, in the case where the conveyor mechanism 200 transports the item so that it falls within the camera range of the jth camera 22, the first processor 11P records the jth movement distance as an item positioning parameter and stores it in the first memory 11M.

[0088] As shown in Figure 4A and Figure 4B In actual operation, the inspector can also set the light source 24 of each inspection station for adjustment, so that the light source 24 of each inspection station has a specified illumination range, light intensity, color temperature, and light color after adjustment. Thus, in the case where the jth light source 24 has an illumination range, a light intensity, a color temperature, and a light color after a light source adjustment operation, the first processor 11P records the illumination range, the light intensity, the color temperature, and the light color as a set of lighting parameters and stores them in the first memory 11M.

[0089] After obtaining the motor control parameters, the N number of article positioning parameters, and the N sets of lighting parameters, the electronic device 11 can perform a parameter update operation on the device control software installed in the master electronic device 23 according to the motor control parameters, the N number of article positioning parameters, and the N sets of lighting parameters. Thus, after the model update operation and the parameter update operation, the AVI system 2 can be used to automatically detect defects of a specified article. Specifically, as shown in Figure 1 、 Figure 2A and Figure 2B , the master electronic device 23 includes a second processor 23P and a second memory 23M, and the second memory 23M stores a second application program (including a device control software and an article defect identification software) therein, so that the second processor 23P accesses the second memory 23M to execute the second application program, thereby automatically detecting defects, including the following steps:

[0090] controlling the N number of mechanical arms 21 to move according to the corresponding external camera parameters, thereby moving the N number of cameras 22 correspondingly;

[0091] controlling the N number of light sources 24 to provide N number of detection lights to the N number of articles falling within the camera ranges of the N number of cameras 22 according to the corresponding lighting parameters; and

[0092] controlling the N number of cameras 22 to capture the articles falling within the camera ranges thereof according to the corresponding internal camera parameters, thereby obtaining a plurality of article images; and

[0093] performing a defect identification operation on each of the article images by using the article defect identification model, thereby determining whether the article has at least one defect.

[0094] After the flaw detection and identification of the article is completed, the main control electronic device 23 controls the discharging device 25 (i.e. a mechanical arm with a clamp) to clamp the article belonging to the NG (not good) product away from the conveying device 20 and put it into an NG product collecting box 5. Therefore, the flexible and intuitive configuration system 1 of the present application can also perform the following procedures. Specifically, after an article moves N moving distances on the conveying belt mechanism 200 and then moves an N+1th moving distance, at this time, the article falls within the clamping range of the discharging device 25, therefore the first processor 11P records the N+1th moving distance as a discharging parameter and stores it in the first memory 11M. Then, the discharging device 25 is operated to discharge an article to an NG product collecting box 5. It should be understood that during the discharging of the article, it must have an initial height, a placement angle and a final height, therefore the first processor 11P records the initial height, the placement angle and the final height as a set of discharging device parameters and stores them in the first memory 11M. In this way, the parameter updating operation of the device control software installed in the main control electronic device 23 can be performed by the electronic device 11 according to the discharging parameter and N discharging device parameters.

[0095] On the contrary, when a batch of articles (e.g. gears) are placed on the conveying device 20 to be subjected to flaw detection, the article placement device 26 (i.e. a mechanical arm with a clamp) must be operated to clamp the article on the conveying device 20 and then place the article on the conveying device 20 so that the article is located on the conveying device 20 at a placement angle. It should be understood that in the case where the article placement device 26 is operated to place an article on the conveying device 20, the article has an initial height, a placement angle and a final height, at this time the first processor 11P records the initial height, the placement angle and the final height as an article placement device parameter and stores it in the first memory 11M. Finally, the parameter updating operation of the device control software installed in the main control electronic device 23 can be performed by the electronic device 11 according to the article placement device parameter.

[0096] After the updating of the control parameters of the article placement device 26 and the discharging device 25 is completed, the AVI system 2 can perform the full-automatic operation of article placement, article flaw detection and identification, and NG product discharging on a batch of articles.

[0097] Thus, the flexible and intuitive configuration system of the automatic visual inspection system of the present application has been completely and clearly described above. However, it must be emphasized that the above detailed description is a specific description of the feasible embodiments of the present application, and is not intended to limit the patent scope of the present application. Any equivalent implementation or modification made without departing from the spirit of the present application shall be included in the patent scope of the present application.

Claims

1. A flexible and intuitive configuration system for setting up an automatic visual inspection system to be suitable for use in automated defect inspection of an article, wherein the automatic visual inspection system comprises a conveyor, N robotic arms, N cameras held by the N robotic arms respectively, and a master electronic device, and N is at least a positive integer of 1; the flexible and intuitive configuration system comprises: an electronic device coupled to the master electronic device to be informationally linked with the N cameras and the N robotic arms by the master electronic device; wherein the electronic device comprises a first processor and a first memory storing a first application program, so that the first processor executes the first application program by accessing the first memory to enable the following functions: in the process that each robotic arm is operated to move the camera K times so that the camera has a camera height, a camera angle and a camera distance after each movement, recording one camera height, one camera angle and one camera distance after each movement of the camera as a set of external camera parameters to obtain K sets of external camera parameters; wherein K is at least a positive integer of 1; in the case that the camera has an aperture, a depth of field, a shutter speed, a light sensitivity and a focal length after each movement of the camera followed by a camera adjustment operation, recording one aperture, one depth of field, one shutter speed, one light sensitivity and one focal length as a set of internal camera parameters to obtain K sets of internal camera parameters after the camera moves K times; after the camera takes pictures of an article transported by the conveyor to obtain a plurality of first article images with regular features and a plurality of second article images with defect features, uploading the plurality of first article images with regular features and the plurality of second article images with defect features to a remote electronic device; and after the remote electronic device generates at least one update module using the plurality of first article images with regular features and the plurality of second article images with defect features, performing a model update operation on a first article defect recognition model installed in the master electronic device using the update module, or installing a second article defect recognition model in the master electronic device using the update module.

2. The flexible intuitive configuration system of claim 1, wherein, The electronic device is any one selected from the group consisting of a smartphone, a tablet computer, a desktop computer, an all-in-one computer and a notebook computer.

3. The flexible intuitive configuration system of claim 1, wherein, The camera is included in a mobile electronic device, and the mobile electronic device is any one selected from the group consisting of a smartphone and a tablet computer.

4. The flexible intuitive configuration system of claim 3, wherein, The action electronic device comprises an inertial measurement unit and a laser detection and ranging unit, which are used to obtain a three-dimensional movement data and a three-dimensional measurement data after each movement of the camera, and the action electronic device transmits the three-dimensional movement data and the three-dimensional measurement data to the electronic device, so that the electronic device integrates the three-dimensional movement data and the three-dimensional measurement data into the external camera parameters.

5. The flexible intuitive configuration system of claim 3, wherein, The action electronic device comprises an inertial measurement unit and a laser detection and ranging unit, which are used to obtain a three-dimensional movement data and a three-dimensional measurement data after each movement of the camera, and the action electronic device transmits the three-dimensional movement data and the three-dimensional measurement data to the electronic device, so that the electronic device integrates the three-dimensional movement data and the three-dimensional measurement data into the external camera parameters.

6. The flexible intuitive configuration system of claim 5, wherein, The action electronic device comprises an inertial measurement unit and a laser detection and ranging unit, which are used to obtain a three-dimensional movement data and a three-dimensional measurement data after each movement of the camera, and the action electronic device transmits the three-dimensional movement data and the three-dimensional measurement data to the electronic device, so that the electronic device integrates the three-dimensional movement data and the three-dimensional measurement data into the external camera parameters.

7. The flexible intuitive configuration system of claim 1, wherein, The first processor records and stores a plurality of sets of the external camera parameters corresponding to N mechanical arms in the first memory, and records and stores a plurality of sets of the internal camera parameters corresponding to N cameras in the first memory.

8. The flexible intuitive configuration system of claim 1, wherein, After the first processor executes the first application program, a display of the electronic device displays an operation interface, so that a user can operate the operation interface to perform a roll marking process on the plurality of images of the first articles with regular features and the plurality of images of the second articles with defect features.

9. The flexible intuitive configuration system of claim 1, wherein, The conveying device comprises a conveying belt mechanism, a motor for driving the conveying belt mechanism, and a distance sensor, and the first processor executes the first application program to further enable the following functions: In the case that the motor is set to operate at a rotating speed to drive the conveying belt mechanism, record the rotating speed as a motor control parameter; In the case that the conveying belt mechanism conveys the articles to make them fall within a camera range of the jth camera, record the jth movement distance as an article positioning parameter; where j ∈ N; and Perform a parameter update operation on a device control software installed in the master electronic device according to the motor control parameter and N article positioning parameters.

10. The flexible intuitive configuration system of claim 9, wherein, The automatic visual inspection system further comprises N light sources, and the first processor executes the first application program to enable the following functions: In the case that the light sources have a light range, a light intensity, a color temperature, and a light color through a light source adjustment operation, record the light range, the light intensity, the color temperature, and the light color as a set of lighting parameters; And Perform the parameter update operation on the device control software installed in the master electronic device according to the lighting parameters.

11. The flexible intuitive configuration system of claim 10, wherein, The master electronic device comprises a second processor and a second memory, and the second memory stores a second application program, so that the second processor executes the second application program by accessing the second memory to enable the following functions: Control the mechanical arms to move the cameras according to their corresponding external camera parameters; controlling the light source to provide a detection light to the article according to the corresponding illumination parameter thereof; and controlling the camera to capture an article image from the article according to the corresponding internal camera parameter thereof; and performing a defect recognition operation on the article image by using the article defect recognition model to determine whether the article has at least one defect.

12. The flexible intuitive configuration system of claim 11, wherein, The automatic visual inspection system further comprises at least one ejection device disposed adjacent to an ejection port of the conveying device, and the first processor executes the first application program to further enable the following functions: after one of the articles moves N movement distances on the conveying mechanism and then moves an (N+1)th movement distance, recording the (N+1)th movement distance as an ejection parameter; in a case where the ejection device is operated to eject one of the articles to a collection box, the article has an initial height, a placement angle, and a final height, and the initial height, the placement angle, and the final height are recorded as a set of ejection device parameters; and performing the parameter updating operation on the device control software installed in the main control electronic device according to the ejection parameter and N ejection device parameters.

13. The flexible intuitive configuration system of claim 12, wherein, The automatic visual inspection system further comprises at least one article placement device disposed adjacent to an article inlet of the conveying device, and the first processor executes the first application program to further enable the following functions: in a case where the article placement device is operated to place one of the articles on the conveying device, the article has an initial height, a placement angle, and a final height, and the initial height, the placement angle, and the final height are recorded as an article placement device parameter; and performing the parameter updating operation on the device control software installed in the main control electronic device according to the article placement device parameter.

Images