Methods, apparatuses, computer programs, and media including computer instructions for performing item inspection

Abstract

A method (and corresponding apparatus, computer program and medium) for determining whether an item being processed is defective or non-defective is provided, the method comprising the steps of: determining (S10), by a local neural network (11) and based on sensing measurements performed on the item while the item is being processed, a local classification result indicative of whether the item is defective or non-defective. At step S20, a confidence index indicative of a confidence level that the local classification result is correct is determined. Then, at step S30 and in response to the confidence index being lower than a given threshold, a central classification result indicative of whether the item is defective or non-defective is determined, by a central neural network (21) and based on said sensing measurements, wherein the local neural network (11) has less computational resources than the central neural network (21).

Description

Technical Field

[0001] The present invention relates to methods, apparatus, computer programs, and media for executing computer instructions for inspecting articles, particularly for determining whether an article being processed is defective or not. Background Technology

[0002] Visual inspection equipment is used on the production line to check for defects in products being manufactured. This equipment typically includes image sensors such as cameras, memory, and a CPU to perform image processing and general processing. These devices are small enough to be mounted on parts of the production line (such as robots) and operate locally for reasons such as fast response times, avoiding communication problems with other equipment, and keeping their construction and installation as simple as possible. Different image processing techniques can be used to determine whether an image refers to a defective or non-defective product; each of these techniques is suited to its respective application or use case, and each is characterized by certain errors and corresponding computational complexity.

[0003] In addition, there are other types of inspection systems that are not necessarily based on visual analysis, but rather on, for example, measuring parameters of the actual product to determine whether it has defects.

[0004] However, known technologies may be susceptible to errors and / or require significant processing power. Summary of the Invention

[0005] Therefore, one object of the present invention is to improve existing systems for performing inspections and / or overcome at least some of the problems present in existing technical solutions.

[0006] This objective can be achieved in the following ways:

[0007] According to aspect A1, an inspection method is provided for determining whether an article being processed is defective or not, the method comprising the following steps:

[0008] The local classification result indicating whether the item is defective or not is determined by a local neural network (11) and based on sensing measurements performed on the item while it is being processed (S10).

[0009] Determine (S20) the confidence index indicating the level of confidence in the local classification result as correct;

[0010] In response to a confidence index falling below a given threshold, a central classification result indicating whether the item is defective or not is determined (S30) by the central neural network (21) and based on the sensing measurements, wherein the local neural network (11) has fewer computational resources than the central neural network (21).

[0011] A2. In the method according to aspect A1, the confidence index is determined when the local neural network (11) is trained, and wherein preferably a given threshold is determined empirically when the local neural network (11) is trained.

[0012] A3. In the method according to aspects A1 and / or A2, the confidence index is determined by relating the actual activation patterns exhibited by multiple nodes of the local neural network (11) when determining the local classification result to the reference activation patterns exhibited by the multiple nodes of the local neural network (11) during training of the local neural network (11).

[0013] And the given thresholds include a relevant threshold that indicates a predetermined level of relevance.

[0014] A4. In any of the methods according to the foregoing aspects, the confidence index is determined by correlating at least one feature vector obtained by the local neural network (11) while determining the classification result with at least one corresponding reference feature vector obtained by the local neural network (11) while training the local neural network (11).

[0015] According to aspect A5, an inspection device (10) is provided for determining whether an article being processed is defective or not defect-free, the local inspection device (10) comprising:

[0016] A local neural network (11) is configured to determine a local classification result indicating whether the item is defective or not based on sensing measurements performed on the item while it is being processed.

[0017] The processor (12) is configured to determine a confidence index that indicates the confidence level of matters that are correct in the local classification results;

[0018] The output section (13) is configured to output a central classification indication notification in response to a confidence index falling below a given threshold. This central classification indication notification is executed by a central neural network to indicate whether the item being processed is defective or not, wherein the local neural network has fewer computational resources than the central neural network (21).

[0019] A6. In the local inspection device (10) according to aspect A5, the central classification instruction notification includes a request for the central classification result to be determined by the central neural network, and wherein the output part (13) is configured to send the request to the central inspection device (20) including the central neural network (21).

[0020] A7. In the inspection device (10) according to aspect A5 or A6, a confidence index is determined when the local neural network (11) is trained, and wherein preferably a given threshold is determined empirically when the local neural network (11) is trained.

[0021] A8. In the examination according to any of aspects A5 to A7, the confidence index is determined by correlating the actual activation patterns exhibited by multiple nodes of the local neural network (11) when determining the local classification result with the reference activation patterns exhibited by the multiple nodes of the local neural network (11) during the training of the local neural network (11).

[0022] And the given thresholds include a relevant threshold that indicates a predetermined level of relevance.

[0023] A9. In any of the methods according to aspects A5 to A8, the confidence index is determined by correlating at least one feature vector obtained by the local neural network (11) while determining the classification result with at least one corresponding reference feature vector obtained by the local neural network (11) while training the local neural network (11).

[0024] According to aspect A10, a central inspection device (20) is provided for determining whether an article being processed is defective or not, the central inspection device (20) comprising:

[0025] A central neural network (21) is configured to determine a classification result indicating whether the item is defective or non-defective based on sensing measurements performed on the item while it is being processed.

[0026] The receiver (22) is configured to receive instructions to perform central classification, which instruct the local neural network (11) to determine the classification result based on sensing measurements with a confidence level below a given threshold, wherein the local neural network (11) has fewer computational resources than the central neural network (21).

[0027] A11. In the central inspection device (20) according to aspect A10, the central inspection device (20) is configured to operate the central neural network (21) by using more computing resources than are available at the local neural network.

[0028] A12. In the central inspection device (20) according to aspect A10 or A11, a confidence index is determined when the local neural network (11) is trained, and wherein preferably a given threshold is determined empirically when the local neural network (11) is trained.

[0029] A13. In the central inspection device (20) according to any of aspects A10 to A12, the confidence index is determined by relating the activation patterns exhibited by multiple nodes of the local neural network (11) when determining the local classification result to a reference activation pattern exhibited by the multiple nodes of the local neural network (11) during training of the local neural network (11).

[0030] And the given thresholds include a relevant threshold that indicates a predetermined level of relevance.

[0031] A14. In the central inspection device (20) according to any of the aspects A10 to A13, the confidence index is determined by correlating at least one feature vector obtained by the local neural network (11) while determining the classification result with at least one corresponding reference feature vector obtained by the local neural network (11) while training the local neural network (11).

[0032] According to aspect A15, an inspection system is provided for determining whether an article is defective or non-defective based on sensing measurements performed on the article while it is being processed, using at least one of a local neural network (11) and a central neural network (21), wherein the local neural network has fewer computational resources than the central neural network.

[0033] In response to determining that the local confidence index is not higher than a predetermined confidence level, the local confidence index indicates the confidence level of the local classification result as correct. The central neural network (21) is configured to determine whether the obtained sensing measurement represents a defective item or a non-defective item in the central classification result.

[0034] Furthermore, the central classification result is used as the system's classification result.

[0035] A16. In the inspection system according to aspect A15, the confidence index is determined when the local neural network (11) is trained, and wherein preferably a given threshold is determined empirically when the local neural network (11) is trained.

[0036] A17. In the inspection system according to aspect A15 or A16, the confidence index is determined by correlating the activation patterns exhibited by multiple nodes of the local neural network (11) when determining the local classification result with the reference activation patterns exhibited by the multiple nodes of the local neural network (11) during training of the local neural network (11).

[0037] And the given thresholds include a relevant threshold that indicates a predetermined level of relevance.

[0038] A18. In an inspection system according to any of aspects A15 to A17, a confidence index is determined by correlating at least one feature vector obtained by the local neural network (11) while determining the classification result with at least one corresponding reference feature vector obtained by the local neural network (11) while training the local neural network (11).

[0039] According to aspect A19, a computer program is provided, the computer program including instructions configured to perform the steps of any one of aspects A1 to A4 when the instructions are executed on a computer.

[0040] According to aspect A20, an inspection method for determining whether an article being processed is defective or not is provided, the method comprising the steps of: determining (S10) a classification result indicating whether the article is defective or not by means of a neural network (11) and based on sensing measurements performed on the article while it is being processed; determining (S20) a confidence index indicating a confidence level of a correct matter; and outputting a notification message in response to the confidence index being lower than a given threshold, the notification message indicating that the classification result indicating whether the article is defective or not has a confidence level lower than the given level.

[0041] A21. The method according to aspect A20, wherein the notification message is output to a device for notification and / or output to a device for further processing.

[0042] A22. The method according to aspect A20 or A21, wherein the confidence index is determined when training the neural network (11), and wherein preferably a given threshold is determined empirically when training the neural network (11).

[0043] A23. The method according to any one of aspects A20 to A22, wherein a confidence index is determined by correlating the actual activation patterns exhibited by a plurality of nodes of the neural network (11) when determining a classification result with a reference activation pattern exhibited by the plurality of nodes of the neural network (11) while training the neural network (11), and wherein a given threshold includes a correlation threshold indicating a predetermined correlation level.

[0044] A24. The method according to any one of aspects A20 to A23, wherein a confidence index is determined by correlating at least one feature vector obtained by the neural network (11) while determining the classification result with at least one corresponding reference feature vector obtained by the neural network (11) while training the neural network (11), wherein a feature vector obtained by the neural network preferably includes a vector containing feature parameters obtained by at least one node of one or more layers of the neural network (11).

[0045] A25. An inspection device (10) for determining whether an article being processed is defective or not defective, the inspection device (10) comprising: a neural network (11) configured to determine a classification result indicating whether the article is defective or not defective based on sensing measurements performed on the article while it is being processed; a processor (12) configured to determine a confidence index indicating a confidence level of a correct classification result; and an output portion (13) configured to output a notification message in response to a confidence index falling below a given threshold, the notification message indicating that the result indicating whether the article being processed is defective or not defective has a confidence level below the given level.

[0046] A26. The inspection device (10) according to aspect A25, wherein the output part (13) is further configured to output notification messages to a device for notification and / or to a device for further processing.

[0047] A27. The inspection device (10) according to aspect A25 or A26, wherein the confidence index is determined when training the neural network (11), and wherein preferably a given threshold is determined empirically when training the neural network (11).

[0048] A28. An inspection device (10) according to any of aspects A25 to A27, wherein a confidence index is determined by correlating the actual activation patterns exhibited by a plurality of nodes of a neural network (11) when determining a classification result with a reference activation pattern exhibited by the plurality of nodes of the neural network (11) while training the neural network (11), and wherein a given threshold includes a correlation threshold indicating a predetermined correlation level.

[0049] A29. An inspection device (10) according to any of aspects A25 to A28, wherein a confidence index is determined by correlating at least one feature vector obtained by the neural network (11) while determining the classification result with at least one corresponding reference feature vector obtained by the neural network (11) while training the neural network (11).

[0050] A30. A computer program comprising instructions configured to perform, when executed on a computer, steps according to any one of aspects A1 to A5 or any one of aspects A20 to A24.

[0051] A31. A medium comprising instructions configured to perform, when executed on a computer, steps according to any one of aspects A1 to A5 or any one of aspects A20 to A24. Attached Figure Description

[0052] Figure 1This is a flowchart illustrating a method according to an embodiment of the present invention.

[0053] Figure 2 A block diagram of a device according to one embodiment of the present invention is shown.

[0054] Figure 3 A block diagram of a device according to one embodiment of the present invention is shown.

[0055] Figure 4 A block diagram of a system according to one embodiment of the present invention is shown.

[0056] Figure 5 A block diagram of a computer adapted to execute instructions according to an embodiment of the present invention is shown.

[0057] Figure 6 An example of a neural network activation monitoring scheme is shown.

[0058] Figure 7 An example of a neural network output monitoring scheme is shown. Detailed Implementation

[0059] Visual inspection equipment or systems can use different techniques to determine if a product has defects, such as one or a combination of "image matching" or "parameter matching" processes:

[0060] - Image matching processing: A picture of the product being manufactured is taken; the picture is compared to a reference picture of the product; if the picture matches the reference picture, the product is considered (as a result of the match) to be without defects (or "good"), otherwise it is defective (or "bad"). The reference picture is, for example, a picture of a product that is without defects or contains an acceptable amount and / or type of defect; a match is determined between the picture and the reference picture when a given (e.g., predetermined) similarity exists between the two pictures. Different techniques can be used to establish the match, such as pixel-by-pixel comparison.

[0061] - Parameter Matching Process: Image processing is performed on the captured image to extract certain parameters. These extracted parameters are then compared to reference parameters. If a match is found between the extracted and reference parameters, the output is "good" (no defects); otherwise, the output is "bad" (no defects). Examples of parameters include any one or a combination of color, brightness / contrast, shape, function, etc. Reference parameters can be determined based on one or more images taken on products considered to be defect-free and / or defective; therefore, parameter matching exists when the corresponding extracted parameters and reference parameters are the same or different within a given range.

[0062] AI technology can be applied to inspection equipment installed on production lines, for example, based on one or a combination of the basic technologies described above. For instance, an AI machine can be trained on a set of photographs known to correspond to defective and non-defective products; once trained, it can be used to inspect products being manufactured by having the trained AI machine classify photographs of such products online. Training can be achieved using one or a combination of the methods described above (although other methods are also possible, independently or in combination). For example, regarding parameter matching processing, the AI ​​machine can be trained based on parameters (or features (within a feature space determined for a specific processing or task) extracted from a set of available images, where for each of these images it is known whether it refers to a defective or non-defective product. Once learning is complete, the AI ​​machine can classify newly photographed images as defective or non-defective by processing the corresponding extracted parameters using the trained model.

[0063] However, such AI machines are prone to error and may therefore incorrectly classify products as, for example, falsely defective (when the actual product is good) or falsely non-defective (when the actual product is not actually good). Furthermore, and optionally, such AI inspection systems can provide a third type of output (an output other than defective and non-defective), for example, when the AI ​​machine cannot determine with a certain level of confidence whether a product should be classified as defective or non-defective. In other words, the third type of output can indicate that the AI ​​machine cannot classify the image and the corresponding product; in this case, the product may need to be manually inspected to determine if it is defective. While this third type of output can reduce the number of falsely defective / falsely non-defective classifications (and advantageously help in determining the handling of defective and non-defective items in any way), there are situations where classification cannot be completed accurately and / or cannot be completed in a timely manner. For AI machines, this generally refers to entities comprising neural networks that can be trained on known items to perform a given task, in this case, inspecting items; in operation, i.e., when training is complete and the AI ​​machine is deployed for practical use, the AI ​​machine is typically able to perform the given task on unknown items. "Completed training" means that sufficient training has been conducted before an operation can begin; training can also continue after the operation has begun, such as when new data becomes available, and / or periodically.

[0064] One way to improve accuracy is, for example, by increasing the computational power of the AI ​​machine by increasing the number of layers in the neural network included in the AI ​​machine, and / or by choosing a more complex configuration of the neural network. Furthermore, it is conceivable that, for example, training the trained model on a larger dataset, could potentially combine it with a more efficient neural network (i.e., with a neural network that has higher computational performance) to improve the accuracy of the trained model.

[0065] The aforementioned conceivable AI solutions will still be based on local implementations, thus aligning with the usual practice of performing inspections on-site on the products being processed, in order to ensure minimal latency and waiting time, which is considered a critical factor during inspection. For example, the probability of finding defective items is relatively lower than the probability of finding non-defective items; therefore, ensuring minimal latency and waiting time in the determination process is particularly important, as otherwise the entire process could be significantly delayed; at the same time, it is also important that the determination is correct, for example, that defective items are not missed.

[0066] Based on the above considerations and understanding, the inventors conceived of distributing inspection analysis between a local AI machine and a remote AI machine: the local machine may not be as computationally efficient as the remote machine, allowing it to be small and suitable for on-site installation near where the product is being processed. Although generally considered an unsuitable solution, the involvement of remote AI has proven advantageous when used in conjunction with a local AI machine, as the potential latency and waiting time introduced by remote analysis is overcompensated (i.e., entirely less) than the latency and waiting time caused by the local AI machine's inaccuracy or failure to make appropriate determinations. Importantly, this solution allows these effects to be achieved while keeping the local inspection equipment structurally and / or installation small and simple enough for on-site use. Furthermore, regarding the described local and remote AI machines, it should be noted that, according to variations of the inventors' concept, the remote machine (hereinafter also referred to as the central neural network) can be omitted. In practice, the local AI machine can output an indication that a determination of defective or non-defective has been reached within a given level of accuracy / confidence, and may require, for example, additional evaluation and / or verification to more accurately determine whether an item is defective or non-defective.

[0067] refer to Figure 1This describes a first implementation of a method for determining whether an article being processed is defective or not. Defective and not defective refer respectively to whether the article conforms to the technical specifications and / or quality standards of the article's design. For example, a product is defective when its (mechanical, electrical, chemical, and / or optical, etc.) values ​​are outside their given tolerances and / or when it has scratches caused by processing. The method includes step S10, where a local neural network (11) determines a local classification result based on sensing measurements performed on the article while it is being processed. The local classification result indicates whether the article is defective or not. The term "local" indicates that the classification result is obtained by the local neural network 11, or in other words, by the operation of the local neural network. Local refers to a neural network placed near the article being processed. Sensing measurements indicate any measurement results obtained by any sensor coupled to the article (i.e., that can interact with or engage with the article for measurement purposes, the engagement being not limited to mechanical engagement), and include, for example, image data obtained by means of a camera sensor that takes pictures of the article; mechanical measurements, such as length data obtained by means of a laser-based measuring device; electrical values ​​(e.g., voltage, current, etc.) measured by the corresponding sensor; density measurements of the article's composition; measurements indicating the optical properties of the article obtained by the corresponding appropriate sensor, etc. These measurements are performed on the article being processed, indicating that the article is being processed in a certain process, and include, for example, moving and / or handling the article on a production line, such as handling the article along a transport route at a checkpoint to verify whether the transport was damaged, etc. For completeness, note that local classification results may also indicate any defective and non-defective state of the article that cannot be determined. Therefore, in one example, the classification result may output one of an indication that the item is (i) defective and (ii) non-defective; in another example, the classification result may output (i) possibly or (ii) impossible to classify as defective or non-defective; in yet another example, the classification result may output one of (i) an indication that the item is defective, (ii) an indication that the item is non-defective, and (iii) a corresponding determination that (defective / non-defective) is impossible. These examples can be combined, and the device can be configured to change dynamically between these configurations. In the discussion above and below, the determination of whether the item being processed is defective or non-defective is used to illustrate this determination; however, the same consideration applies in determining whether the item being processed exists or does not exist in a given area; therefore, "whether the item is defective or non-defective" can also be understood as "whether the item exists".The given area can be: predefined (e.g., by means of coordinates); determined based on, for example, a sensor that performs sensing measurements (e.g., an area of ​​a scene that can be captured by a camera); defined in relation to a component being manufactured (e.g., a robot); or a range within which the sensor can perform the aforementioned sensing measurements, or a combination thereof.

[0068] In step S20, a confidence index is determined, where the confidence index indicates the level of confidence that the local classification result is correct. As is well known in the field of machine learning, a confidence level represents the probability that the result produced by the AI ​​machine is true or correct, and it can be expressed, for example, as a percentage of the probability that the AI ​​estimates correspond to the actual value (or a value within a range, such as the confidence index). In an illustrative example, a 90% confidence level could indicate that one of the results of the local AI machine classifying the product as defective / no defect corresponds to a 90% probability that the product is actually defective or, respectively, no defective. The same or different threshold levels can be assigned to each defective / no defect determination. Confidence levels can be obtained in a variety of well-known ways, see, for example, “Confidence Score of Distance-Based Neural Network Classifiers,” Amit Mandelbaum and Daphna Weinshall, arXiv:1709.09844v1 [cs.AI] 28 Sep 2017 (https: / / arxiv.org / pdf / 1709.09844.pdf).

[0069] Referring to the above determination, "cannot be classified as defective / non-defective" indicates that obtaining such a classification is infeasible, or that such a determination has not been reached within a given or predetermined confidence level. Therefore, for example, when neither a defective nor a non-defective classification can be output within the corresponding first and second confidence level thresholds (in a non-limiting example, the two thresholds can be the same), the AI ​​machine can output a determination that "cannot be classified as defective / non-defective."

[0070] In response to a confidence index falling below a given threshold, in step S30, the central neural network 21 determines a central classification result based on the sensed measurements, preferably based on the same information used by the local neural network 11. Similar to the local classification result, the central classification result indicates whether the item is defective or not. However, unlike the local neural network, the central neural network 21 has more computational resources than the local neural network 11. Therefore, the central indication processing power is greater than that of the local network, which allows for, for example, more complex configurations for the central neural network; preferably, the central neural network is located away from the item being processed, i.e., not near the item being processed and / or the local neural network. Therefore, the central neural network 21 is more likely to be able to determine a defective / not defective classification result. In other words, the central classification result is used as the final or actual classification result; however, if the local classification result is higher than (or optionally equal to) a given threshold, the local classification result becomes the actual classification result without activating the central neural network. Thus, the additional latency and waiting time due to the intervention of the local neural network are limited to those cases where the local central network's computing resources are insufficient to provide accurate results; therefore, since most detections are performed locally, the entire system remains efficient, and detections are only deferred to the central neural network, which, in any case, can obtain results faster and / or more accurately than other systems, when needed. This achieves high overall performance because the latency / waiting time associated with the activation of the central network overcompensates for the possible waiting time / latency caused by the local network's failure to determine accurate results. In one example, the central neural network represents or includes computing resources with an AI model and is connected to the local inspection machine via a communication network (such as the Internet and / or an intranet), and has greater computing power than the local machine; furthermore, the central neural network does not necessarily need to be located in the same location as the local machine.

[0071] For example, larger computational resources include situations where the central neural network 21 is provided with a higher number of layers than the local neural network 11 (e.g., a higher number of intermediate layers in the neural network; this includes cases where the local neural network has no intermediate layers and the central network has one or more; more generally, for example, the central neural network has at least one more layer than the local neural network; in other examples, the central neural network has more nodes than the local neural network, which can be combined with a higher number of layers), and / or a more precise neural network structure at the cost of its larger / more complex structure, and / or larger memory that enables the neural network to process a larger amount of information, etc.; additionally or alternatively, larger computational resources may include situations where the central neural network 21 is trained on a larger dataset and / or is able to manage more complex training models, etc.

[0072] Preferably, the above-described inspection method includes the following steps: when a confidence index indicates a level of confidence that the local classification result is correct, outputting a classification result indicating at least one of whether the article is defective or undefective, based on the local classification result or based on the central classification result. In other words, the results of the local neural network and / or the results of the central neural network are used to determine the actual classification result based on the confidence level of the classification result determined by the local neural network. In particular, it is conceivable to combine the local and central results to further improve accuracy. As will be apparent from this disclosure, the classification result (local, central, or final) may indicate only one of defective or undefective. For example, the method may be configured to output only the result corresponding to undefective articles (or only for defective articles); when no result is provided, it may be implicitly determined that the article is defective (or correspondingly undefective).

[0073] In a variation of the first embodiment, a method including steps S10 and S20 as described above is foreseeable, wherein the local neural network is preferably located near the article being processed, near the device processing the article, or near the sensor providing sensing measurements performed on the article; the neural network used in this variation can also be called a native-end neural network, and can preferably (but not necessarily) have the same characteristics as the local neural network. In particular, the native-end neural network can be a neural network with limited processing capabilities; for example, it can be a neural network suitable for execution in a device with limited processing resources (e.g., a client computer or client controller to be installed on a device such as a device placed on a production line), rather than a neural network that requires, for example, a large server or cloud-based execution to operate. However, this variation is not limited to the specific limitations of computing resources. Both the native end and the local neural network are provided with sensing performed on the article while it is being processed, which means that the corresponding sensor is coupled to the article being processed, as explained above with reference to the local neural network. For simplicity, this variation is also referred to as native-end, and the method of the first embodiment described above is also referred to as local / central. In this variation, after step S20, a notification message is output. The notification message indicates that the classification result (indicating whether the item is defective or not) has a confidence level below a given level. This notification may optionally include an indication of whether the item is defective or not; in this case, the indication regarding accuracy preferably refers to the defective or not-defective indication included in the notification. The given confidence level may be a predetermined confidence level, which may be set statically or dynamically. Furthermore, the given confidence level may include a threshold, and preferably may be the same as or different from the threshold used in determination step S20. In other words, at this step, an indication is provided that a determination of whether an item is defective or not, even if achieved, may not be highly or sufficiently accurate; therefore, it may be preferably determined that further evaluation and / or confirmation is needed to determine whether the item is defective or not.

[0074] Preferably, according to this variation, the notification message is output to a device for notification (or in other words, for initiating a notification by means of the device) and / or output to a device for further processing. Preferably, the notification message may include alarm messages and / or warning messages indicating, for example, that a determination could not be performed precisely. Preferably, alternatively or additionally, the notification message may include a request to confirm that the determination made by the (local) neural network is correct; such confirmation may be performed by another inspection automation analysis (e.g., image recognition, measurement, and / or testing of an item, etc.), by a central neural network (e.g., in a local / centralized case), by an operator, etc. Preferably, the output (e.g., after step S20 of the variation) includes sending a notification message to the device if the device is different from the local device or is not near the inspection device that produced the output. More preferably, the output of the notification refers to a notification initiated on the device and may include, for example, notifying the operator that the determination of defective / non-defective is inaccurate, and / or notifying the application (e.g., a monitoring application) that the determination is inaccurate. Notifications initiated by the device, particularly when addressed to an operator, may be delivered via a display and / or acoustic signals, and may include other types of notifications; examples of such notifications include alarm messages, warning messages, specific GUI configurations, and any combination thereof. As anticipated above, the message may be output to the device for further processing; this includes devices for collecting data related to potentially defective / non-defective items (e.g., yield determination devices for determining the yield of processed or processed items), devices for monitoring the processing of items, and devices for controlling the processing of items (e.g., to determine whether to stop processing and / or change the speed of items being processed). In one example, the notification may be a notification that performs more precise AI determination, for example, by means of a remote neural network (in which case the notification message may include a message sent to the remote neural network).

[0075] It is clear from the above that this article references... Figure 1 This describes methods for local and central networks, as well as native neural networks. Similarly, the content explained herein with reference to the first and / or other embodiments and / or other examples is also applicable to this variation and other variations that are obvious to those skilled in the art, thereby avoiding repetition.

[0076] Optionally, a confidence threshold can be set for comparison with the confidence index, such that the expected intervention rate of the central neural network is within a given intervention threshold. The intervention threshold can be set empirically and / or based on the characteristics of one or more components of the system (such as the accuracy of measurements from sensors, the accuracy level of local and / or central neural networks, the latency and / or waiting time of communication between the central and local neural networks, etc.) and / or the model or function of one or more components of the system; furthermore, the intervention threshold can also be set dynamically based on empirical values ​​and / or the characteristics of one or more components of the system and / or its model / function, i.e., dynamically changed.

[0077] Optionally, a confidence index is determined when training the local neural network 11, i.e., the probability that the classification result provided by the trained neural network is correct, either during or at the end of the training process. Optionally, a given threshold is preferably determined empirically during training the local neural network (i.e., during or at the end of the training process). Empirically means that experiments or tests can be performed on the training dataset to determine the confidence level of the local neural network. However, as mentioned above, the given threshold can also be determined based on characteristics of one or more components of the system, etc.

[0078] We will now describe two alternative schemes of this method, which we will call the neural network activation monitoring scheme (or simply activation monitoring) and the neural network output monitoring scheme (or simply output monitoring). These methods can also be combined.

[0079] According to the activation monitoring options, the confidence index can be determined by the correlation between the actual activation pattern and the reference activation pattern, where the actual activation pattern is the activation pattern exhibited by the local neural network 11 when determining the local classification result (e.g., when the network is operating on a production line), and the reference activation pattern is the activation pattern exhibited by the local neural network while training the local neural network. Note that "while training the network" means that the exhibited pattern is the pattern exhibited during network training, and this can be analyzed during training or after training is completed, in which case the exhibited patterns are stored or at least realized until they are analyzed. The actual and reference activation patterns preferably involve one or more nodes of the local neural network. For example, "activation nodes exhibited when training on defective data" can represent the reference activation pattern in the presence of defects, which we can also call a "defective" pattern or a defective reference pattern; during operation, the activated nodes generate actual activation patterns that are correlated (e.g., compared) with the defective reference patterns. Each actual and reference activation pattern can be represented by a data structure (such as a vector, array, linked list, matrix, etc.), where each node is represented by a value (e.g., a bit, flag, etc.) indicating whether the corresponding node has been activated; one or a combination of the following is possible: all nodes of one layer can be represented in the data structure; all nodes of two or more layers can be represented in the data structure; at least one or more nodes of one or more layers can be represented in the data structure (e.g., in a table-like or matrix structure or other similar representation, which would therefore have values ​​like 0 or 1 corresponding to unactivated or activated nodes, the convention between 0 and 1 can be reversed). In one example, such a data structure contains binary values ​​indicating the activation state of the corresponding node. In the activation monitoring scheme, a given threshold includes a correlation threshold indicating a predetermined correlation level, i.e., the pattern exhibited during use can be exactly the same as the pattern exhibited during training, or different according to a predetermined level / rule (e.g., a given number of nodes can be different in the pattern, or they belong to a specific layer, etc.; see later). Figure 6(Example provided). For example, suppose that during training, a set of activated patterns are detected corresponding to defective results, and let us name each pattern in this set Pd1, Pd2, ..., Pdi, ..., Pdn, where Pdi includes only the nodes activated when a defective result is output (e.g., for Pdi: N1Pdi, N2Pdi, ..., Nnpdi; thus, Njpdi indicates that node Nj of pattern Pdi was activated when a defective result was produced). During operation, i.e., when the network runs to produce a result classification of measured items, suppose the defect is output by the local neural network; before determining that the defect is a classification result, compare (here is an example of correlation) whether the nodes activated during the actual classification process are included in one of the patterns Pd1, ..., Pdn. If yes, "defective" is determined as the actual classification result; if no, "no defect" is output, or an indication that it is impossible to determine whether it is defective or no defect. Thus, by means of activation pattern correlation, the accuracy of determining defective or no defect prediction can be improved. The above is based on the inventor's understanding that, when determining a defective state, there is a bias in the nodes that tend to be activated, resulting in a relatively sparse number of activation patterns when considering the space represented by all hypothetical node combinations. Similar considerations apply to determining a defect-free state.

[0080] In the output monitoring scheme, a confidence index is determined by correlating at least one actual feature vector with at least one corresponding reference feature vector. The actual reference vector is obtained by the local neural network 11 while determining the classification result of the item being processed, i.e., when the neural network is in operation after training. The reference feature vector is obtained by the local neural network 11 while training the local neural network (refer to the activation monitoring case above for "simultaneous determination"). The (actual and / or reference) feature vectors obtained by the local neural network preferably include vectors containing feature parameters obtained (e.g., as outputs) by at least one node of one or more layers of the local neural network; in other words, the vector contains values ​​corresponding to the outputs of at least one node of one or more layers of the local neural network. Therefore, such vectors typically contain values ​​representing at least some features predicted by the network; typically, such vectors contain non-binary values ​​because each value is a representation of a feature element. For example, in one example, the feature vector may include values ​​corresponding to the outputs of all nodes in the final layer; in this case, the feature vector would represent (actual or reference) feature estimates generated by the entire network. In another example, the vector contains values ​​corresponding to the outputs (or the outputs of all nodes) from one (or more) intermediate layers, making the vector a possible intermediate (actual or reference) estimate of the network. In yet another example, the vector contains values ​​corresponding to the outputs of a subset of nodes from one or more layers of the local neural network (i.e., one or more nodes, but fewer than all nodes in a given layer); again, in this case, the (actual or reference) vector would represent an intermediate estimate produced by the network. In the preceding examples, when determining the reference or actual vector, the coefficients can be associated with nodes and / or layers (these coefficients can be determined during the training phase based on the type of neural network, etc.). The preceding examples can be combined with each other in any way. One advantage of intermediate estimation is that decisions can be made earlier without waiting for all network outputs to be processed, thus reducing the latency required to decide whether to invoke the central neural network. Therefore, by using intermediate estimation (i.e., taking one or more nodes that are not the final outputs of the local neural network), the time required to invoke the central neural network is reduced, thus improving the overall latency. The selection of which layers and / or which nodes and / or how many nodes in a given layer to obtain (actual or reference) vectors can be empirically determined to achieve a trade-off between the accuracy of determining the confidence index and the reduction in system latency. This can be determined during training or based on specific rules depending on the type of network being implemented. Therefore, feature vectors, especially when they relate to the non-final outputs of a neural network, can also be described as a class of compressed node information representing features such as the number of activated nodes in a certain layer of a local neural network (and the features generated by each node individually) or the set of local outputs of nodes in a certain layer of the network.Preferably, the nodes determined (or selected) for the reference feature vector are the same as those determined (or selected) for the actual vector; however, the nodes can be different, particularly the number of nodes for the actual vector can be less than the number of nodes in the reference network. Optionally, the architecture of the central neural network can be such that it includes some or all of the layers of the local neural network (e.g., the complexity of the central network is accompanied by "downstream-to-output" layers): in this case, intermediate results can be sent to the central neural network, so that the central neural network can start processing from this intermediate estimate instead of from the beginning, thereby further reducing the overall latency. Similar to the above, the inventors have recognized that defective outputs (and similarly, defect-free outputs) result in a non-dense (e.g., sparse) number of feature vectors in the feature space. Optionally, the correlation between the determined actual feature vector and the reference feature vector is within a given range, tolerance, or rule. Therefore, by verifying that the actual feature pattern correlates with the reference feature pattern of the defective output, it can be determined that the actual classification result is defective; similar considerations apply to the defect-free case.

[0081] In other words, a first-level confidence index and a second-level confidence index can be obtained by referring to optional activation and output monitoring schemes. The first-level confidence index can be one of the classification results of the neural network output (e.g., based on known techniques or empirical rules or models). The second-level confidence index can be obtained by using activation and / or output monitoring schemes to verify that the first-level confidence index is correct. The second-level confidence index (i.e., one or both of the above schemes) can always be effective, or only in certain situations, such as when the first-level confidence index is not higher than a certain threshold. When the second-level confidence condition is always used, the first-level index can also be omitted (i.e., existing techniques or empirical methods / models can be omitted, and the confidence index can be calculated based on the activation and / or output monitoring schemes). However, as mentioned above, applying one or both schemes is optional.

[0082] Optionally, in the method described herein, the central neural network includes the same neural network as the local neural network; the local neural network sends the feature vectors obtained by the local neural network to the central neural network, and the central neural network begins processing based on the feature vectors received by the local neural network.

[0083] For example, all nodes in all layers of the local neural network are also found in the central local network. The output (feature vector) produced by the local neural network is sent to the central neural network, which then begins processing from the layer immediately following the layer corresponding to the output layer of the local neural network. Similarly, if the feature vector is obtained from an intermediate layer (or node) and sent to the central neural network, the central neural network begins processing from the corresponding layer (or node). Also refer to the example above regarding intermediate estimation, which represents an intermediate feature vector produced by the local neural network.

[0084] We note that, unless otherwise stated, the above content is equally applicable to other implementation methods and examples below, and vice versa, thereby avoiding repetition.

[0085] refer to Figure 2 The second embodiment will now be described with respect to an inspection device 10 for determining whether an article being processed is defective or not. The inspection device 10 includes a local neural network 11, a processor 12, and an output section 13. The local neural network 11 is configured to determine a local classification result indicating whether the article is defective or not based on sensing measurements performed on the article while it is being processed. The processor 12 is configured to determine a confidence index indicating the level of confidence that the local classification result is correct, i.e., the determined defective or not-defective state may correspond to an actual defective or not-defective article, respectively. The output section 13 is configured to output a central classification notification in response to a confidence index falling below a given (predetermined) threshold. This central classification notification indicates whether the article being processed is defective or not, and this will be executed by the central neural network. The local neural network 11 has fewer computational resources than the central neural network 21. In other words, the central classification notification flag preferably obtains the central classification result rather than the local classification result, and therefore the output of the central neural network 21 can be considered as the actual (or final) classification result. In other words, the notification indicates that the local result may be inaccurate and that the central result may be more appropriate, and therefore can override the local result. The central classification notification can be represented by a flag, such as a single bit, indicating whether the central activation network must be activated to classify the result based on the sensed measurements; this notification can then be read or received by another device that can command the central neural network to perform the classification. Alternatively, the central classification notification may include a command or instruction sent directly to the central neural network or another device to obtain such a central classification result. Note that "local" and "central" in "local classification result" and "central classification result" refer to the classification results obtained by the local or central neural network, respectively.

[0086] Optionally, the central classification notification includes a request for the central neural network to determine the central classification result, i.e., a request for the central neural network to determine the classification result. In this case, the output section 13 can be configured to send the request to the central neural network 21 or to the central inspection device 20 that includes the central neural network 21; however, the notification can be sent to another network device, such as a management device, whose instructions are then directed to a more powerful suitable neural network than the local neural network (e.g., in a cloud environment, the central neural network is implemented in the cloud; optionally, the cloud device can direct the task to one of multiple neural networks deployed in the cloud).

[0087] In a variation of the second embodiment, an inspection device can be anticipated, which includes the components described above. Figure 2 The local neural network 11 and processor are described above. The output section 13 of this variant is configured to output a notification message in response to a confidence index falling below a given threshold. This notification message indicates whether the processed item is defective or not, and whether the result has a confidence level below a given level. Therefore, this variant of the second embodiment can be named native (similarly and corresponding to the variant of the first embodiment), while the second embodiment can be named local / central. Preferably, in this variant, the output section (13) is also configured to output the notification message to a device for notification and / or to a device for further processing. As can be clearly seen from the above, in this document... Figure 2 This refers to variations of the apparatus and the second embodiment used to describe the second embodiment. Furthermore, the content explained in this document with reference to the first and / or second and / or other embodiments, and / or other examples, and / or first variations of the first embodiment is also applicable to this variation of the second embodiment and other variations that are obvious to those skilled in the art, thereby avoiding repetition.

[0088] Optionally, the confidence index is determined when training the local neural network 11, and preferably a given threshold is determined empirically when training the local neural network 11.

[0089] Optionally, according to the neural network activation monitoring scheme, a confidence index is determined by correlating the (actual) activation patterns exhibited by multiple nodes of the local neural network 11 when determining the local classification result with reference activation patterns exhibited by corresponding multiple nodes of the local neural network 11 while training the same local neural network 11. A given threshold includes a correlation threshold indicating a predetermined correlation level between the actual and reference activation patterns. "Actual" in "actual activation pattern" refers to the neural network being operated when processing items to be classified.

[0090] Optionally, in the neural network output monitoring scheme, a confidence index is determined by correlating at least one feature vector obtained by the local neural network 11 while determining the classification result with at least one corresponding reference feature vector obtained by the local neural network 11 while training the same local neural network 11. In other words, as described above with reference to the first embodiment, it is determined whether one or more feature vectors generated during actual classification are the same as or within a given range, tolerance, or rule as the reference feature vectors obtained during training.

[0091] refer to Figure 3 The document describes a third embodiment for a central inspection device 20 used to determine whether an article being processed is defective or not. The central inspection device 20 includes a central neural network 21 and a receiver 22. The central neural network 21 is configured to determine a classification result indicating whether the article is defective or not based on sensing measurements performed on the article while it is being processed. The receiver 22 is configured to receive instructions to perform central classification. These instructions instruct a local neural network 11 that it has previously determined (or attempted to determine) a classification result based on the same sensing measurements, but reached a confidence level below a given threshold. The local neural network 11 has fewer computational resources than the central neural network 21. Therefore, the instructions instruct the central neural network 21 to perform a more accurate classification than the classification process attempted by the local neural network.

[0092] Optionally, the central inspection device 20 is configured to operate the central neural network 21 by using more computing resources than are available to the local neural network.

[0093] Optionally, the confidence index is determined when training the local neural network 11, and preferably a given threshold is determined empirically when training the local neural network 11.

[0094] Optionally, according to the neural network activation monitoring scheme, a confidence index is determined by correlating the actual activation pattern with a reference activation pattern. The actual activation pattern is the pattern represented by multiple nodes of the local neural network 11 when determining the local classification result, while the reference activation pattern is the pattern represented by multiple nodes of the local neural network 11 while training the same local neural network 11. A given threshold includes a correlation threshold indicating a predetermined level of correlation between the two patterns.

[0095] Optionally, according to the neural network output monitoring scheme, the confidence index is determined by correlating at least one feature vector obtained by the local neural network 11 at the same time as determining the classification result with at least one corresponding reference feature vector obtained by the local neural network 11 when training the classification result.

[0096] refer to Figure 4A fourth embodiment of the inspection system is now described, which determines whether an article is defective or non-defective based on sensing measurements performed on the article while it is being processed, using at least one of a local neural network 11 and a central neural network 21. The local neural network has fewer computational resources than the central neural network. The local and central neural networks are capable of wireless and / or wired communication via wireless and / or wired interconnection networks. The local neural network is configured to determine whether the article is defective or non-defective; furthermore, the system is capable of determining a local confidence index that indicates a level of confidence in the local classification result as correct. Then, in response to determining that the local confidence index is not higher than a predetermined confidence level (i.e., the result may be incorrect), the central neural network determines a central classification result indicating whether the obtained sensing measurements represent a defective or non-defective article. In other words, if the classification result achieved by the local neural network 11 is deemed insufficiently accurate, a classification result is obtained through the central neural network. Therefore, the central classification result becomes the system's classification result.

[0097] Optionally, the local neural network 11 may be part of or represent the local inspection device itself; similarly, the central neural network 21 may be part of or represent the central inspection device itself. The local neural network is typically located near the item being processed, and therefore near the sensors that perform measurements on the item. The central local network may be located away from the local neural network and connected via a network.

[0098] Optionally, in the above system, the confidence index is determined when training the local neural network 11, and preferably a given threshold is determined empirically when training the local neural network.

[0099] refer to Figure 5 The fifth embodiment will be described with respect to a computer program including instructions configured to, when executed on a computer, perform, for example, the steps of the method described above and any variations thereof, or combinations thereof, when referred to in the first embodiment. Figure 5 A block diagram illustrating a computer (500) capable of running the above-described program is shown. In particular, the computer (500) includes a memory (530) for storing program instructions and / or data required for its execution, a processor (520) for executing the instructions themselves, and an input / output interface (510).

[0100] In another embodiment, not shown, a medium is provided for supporting a computer program configured to perform one or a combination of steps according to the methods described above (e.g., referring to the first embodiment) when the program is run on a computer. Examples of the medium are static and / or dynamic memory; a fixed disk or any other medium such as a CD, DVD, or Blu-ray. The medium also includes means capable of supporting signals representing instructions, including means for cable transmission (Ethernet, fiber optic, etc.) or wireless transmission (cellular, satellite, digital terrestrial, etc.).

[0101] Based on the examples already mentioned, the sensor can be represented by a camera, and the sensing measurements can correspond to image data obtained by means of the camera. For example, when an item (e.g., a product) is on a production line, the camera can take pictures of the item (e.g., the product). In another example, the camera can take pictures of the item while it is being moved from one location or transitioning to another.

[0102] According to another example, the sensor can be represented by a voltage and / or current sensor adapted to perform corresponding measurements on the electronic product while it is being manufactured or transferred from one point to another. Other examples are represented by sensors that measure the length and / or width and / or height of an article, the optical properties of an article, the mechanical and / or chemical properties of an article, etc.

[0103] Now for reference Figure 6 Explain an example of a neural network activation monitoring scheme. Figure 6 A neural network with L layers is described, where each layer i has Ni nodes. The nodes in layer i are numbered N. i，1 N i，2 ...N i，Ri Let's assume that during training, the discovery of defective products (e.g., known images corresponding to defective products) leads to node activation, such that only the first and / or second nodes are activated in each layer, while the remaining nodes in each such layer are not activated; this is in Figure 6The lower part is graphically represented, where an array is given for each layer, and only the first two positions differ from zero (1 indicates activation, 0 indicates inactivation). This can represent the reference pattern of node activation when a defective item is identified, and thus can be associated with an indication of a defective item (see an example of the second level confidence index above). During operation, i.e., after training and when the neural network is deployed for classification results, we observe how nodes are activated, and further observe the network's output; the patterns of nodes detected during operation are also referred to in this paper as actual patterns. If during operation the neural network (a) produces a classification result indicating that the product is defect-free, and simultaneously (b) detects an actual pattern in which only one of the first two nodes is activated, the system can determine that the classification result is incorrect because the activation pattern does not correspond to the reference activation pattern. In this example, patterns are considered given all layers and all nodes in each layer (e.g., 601, 602 to 60...). L All form a pattern); however, it can be applied only to one or more layers and only to one or more nodes in each such layer (e.g., 601, 602 to 60). L The pattern can be defined as a single or any combination of patterns. Furthermore, more patterns can be defined (not just one), all corresponding to defective items; in fact, it has been found that only a finite subset of all possible combinations of node activations corresponds to a given classification result. Moreover, a particular pattern can be associated not only with the general classification result but also with one of its subcategories; for example, the patterns discussed above and... Figure 6 The patterns shown can be associated with defects represented by scratches on an item. The above discussion refers to a classification result that is "defective," and the same consideration applies to the case of "no defects." Clearly, the level of correlation between the reference pattern and the actual pattern can be defined depending on the situation, for example, by comparing each node of the reference pattern with each node of the actual pattern, and determining a correlation when an exact match exists or when at least some activated nodes are found in both the reference pattern and the action pattern. When there is no correlation, the system determines that the output may be incorrect, i.e., the local network cannot make a (precise) determination. Therefore, a central neural network intervenes to determine the classification result. The correlation can be defined mathematically or by means of rules (e.g., based on if-then, etc.).

[0104] Now for reference Figure 7 An example of a neural network output monitoring scheme is explained. Figure 7A neural network 700 is depicted providing data representing features as output 720; the output features are the result of network 700 being stimulated by certain sensed measurements given as input 710. Network 700 can be an example of the local neural network described above. For simplicity, it is assumed that the feature is a vector with only two components (A, B), such that it can be graphically represented in a two-axis coordinate system. It is assumed that during training, defect-free items are always or primarily composed of... Figure 7 The reference feature vector RF1 shown represents the features extracted by the neural network 700 when stimulated by data representing known defective products during training. Now assume that during operation, i.e., once training is complete and the network for classifying items is deployed, the network 700 analyzing the first item I1 outputs (a) "defective" as the classification result and (b) AF1 as the actual feature vector for the first item I1. When classifying the second item I2, the network 700 still outputs (a) "defective", but then outputs (b') AF2 as the actual vector extracted for the second item I2. If the vector product (RF1xAF1) of the reference feature and the actual feature is within a given threshold, the classification result is determined to be correct; otherwise, the result is determined to be incorrect. Visually, in this simplified example, the threshold can be represented by the angle and / or length of the vector. For simplicity, assume that the output classification result is confirmed if AFi is within a 45° rotation of RF1. Figure 7 The values ​​show that AF1 has a 30° rotation relative to RF1, i.e., within the 45° threshold, causing the first item I1 to be identified as "defective" because feature monitoring confirms the classification result given at output 720. However, AF2 rotates 90° relative to RF1, making the system's determination of "defective" potentially incorrect, i.e., the local network cannot classify the item. Therefore, the central neural network intervenes to correct the classification. This example is illustrative, and in reality, multiple RFs can exist, and each vector can have only one value or more than two values ​​(i.e., the two values ​​discussed herein are for illustrative purposes only). The invention still applies because it has been found that the number of reference vectors is relatively small compared to all possible vector representations in a given space during classification.

[0105] The neural networks described herein can be implemented using hardware and / or software, such as local or central neural networks. Specifically, the central neural network can be implemented on distributed hardware and / or software resources (e.g., in the cloud), which are also remotely connected to each other, with each remotely connected to the local neural network. Distributed implementation of the local neural network is also conceivable; however, if done, its level of distributed implementation is lower than that of the central neural network because the processing latency and / or waiting time of the local neural network is less than that of the corresponding ID of the central neural network.

[0106] In this specification, reference is made to neural networks or units (such as sensors, memories, processors, etc.). The invention is not limited to the specific networks and / or units described herein, and is equally applicable to corresponding devices; therefore, neural networks, memories, processors, sensors, etc., can be replaced by neural network devices, memory devices, processing devices, sensing devices, etc., respectively. These networks and / or units (or corresponding devices) can be implemented as distinct / independent units / entities or distributed units / entities (i.e., implemented through multiple components connected to each other, regardless of whether they are physically close or remote); these can be centralized or distributed, and can be further implemented by hardware, software, or a combination thereof.

[0107] Numerous implementations and examples have been explained with reference to the steps of the methods or processes. However, the described content can also be implemented in a program that is to run on an entity (which is also distributed) or its apparatus is properly configured. As mentioned above, the entity can be implemented in a single device via HW / SW or a combination thereof, or in multiple interconnected units or devices (again, HW, SW or a combination thereof). Naturally, the descriptions set forth above with respect to implementations and examples applying the principles recognized by the inventors are provided only by way of example of these principles and should not be construed as limiting the scope of the invention claimed herein.

Claims

1. An inspection method for determining whether an article being processed is defective or not defect-free. The item in question is a product manufactured on a production line. The process includes manufacturing the article on the production line and / or moving the article on the production line. The inspection method includes the following steps: A local classification result indicating whether the item is defective or not is determined (S10) by means of a local neural network (11) and based on sensing measurements performed on the item while it is being processed. (S20) Determine (S20) the confidence index indicating the confidence level of the matter that the local classification result is correct; In response to the confidence index falling below a given threshold, a central classification result indicating whether the item is defective or not is determined (S30) by a central neural network (21) and based on the sensing measurements, wherein the local neural network (11) has fewer computational resources than the central neural network (21).

2. The inspection method according to claim 1, wherein, The confidence index is determined when the local neural network (11) is trained, and the given threshold is determined empirically when the local neural network (11) is trained.

3. The inspection method according to claim 1 or 2, wherein, The confidence index is determined by relating the actual activation patterns exhibited by multiple nodes of the local neural network (11) when determining the local classification result to the reference activation patterns exhibited by the multiple nodes of the local neural network (11) during training of the local neural network (11). And the given threshold includes a correlation threshold that indicates a predetermined correlation level.

4. The inspection method according to claim 1 or 2, wherein, A confidence index is determined by correlating at least one feature vector obtained by the local neural network (11) while determining the classification result with at least one corresponding reference feature vector obtained by the local neural network (11) while training the local neural network (11), wherein the feature vector obtained by the local neural network includes a vector containing feature parameters obtained by at least one node of one or more layers of the local neural network.

5. The inspection method according to claim 1 or 2, wherein, The central neural network includes the same neural network as the local neural network. The local neural network sends the feature vector obtained by the local neural network to the central neural network, and The central neural network begins processing based on the feature vector received by the local neural network.

6. An inspection device (10) for determining whether an article being processed is defective or not defective. The item in question is a product manufactured on a production line. The process includes manufacturing the article on the production line and / or moving the article on the production line. The inspection device (10) includes: A local neural network (11) is configured to determine a local classification result indicating whether the item is defective or not based on sensing measurements performed on the item while it is being processed. Processor (12), configured to determine a confidence index indicating the confidence level of the matter that the local classification result is correct; The output section (13) is configured to output a central classification notification in response to the confidence index falling below a given threshold. The central classification notification is executed by a central neural network indicating whether the item being processed is defective or not, wherein the local neural network has fewer computational resources than the central neural network (21).

7. The inspection device (10) according to claim 6, wherein, The central classification notification includes a request for a central classification result to be determined by a central neural network, wherein the output portion (13) is configured to send the request to a central inspection device (20) including the central neural network (21).

8. The inspection device (10) according to claim 6 or 7, wherein, The confidence index is determined when the local neural network (11) is trained, and the given threshold is determined empirically when the local neural network (11) is trained.

9. The inspection device (10) according to claim 6 or 7, wherein, The confidence index is determined by relating the actual activation patterns exhibited by multiple nodes of the local neural network (11) when determining the local classification result to the reference activation patterns exhibited by the multiple nodes of the local neural network (11) during training of the local neural network (11). And the given threshold includes a correlation threshold that indicates a predetermined correlation level.

10. The inspection device (10) according to claim 6 or 7, wherein, The confidence index is determined by correlating at least one feature vector obtained by the local neural network (11) while determining the classification result with at least one corresponding reference feature vector obtained by the local neural network (11) while training the local neural network (11).

11. A central inspection device (20) for determining whether an article being processed is defective or non-defective. The item in question is a product manufactured on a production line. The process includes manufacturing the article on the production line and / or moving the article on the production line. The central inspection equipment (20) includes: A central neural network (21) is configured to determine a classification result indicating whether the article is defective or non-defective based on sensing measurements performed on the article while it is being processed. A receiver (22) is configured to receive instructions to perform central classification, the instructions indicating that a local neural network (11) has determined a classification result based on the sensing measurement with a confidence level below a given threshold, wherein the local neural network (11) has fewer computational resources than the central neural network (21).

12. The central inspection device (20) according to claim 11, wherein, The central inspection device (20) is configured to operate the central neural network (21) by using more computing resources than are available at the local neural network.

13. The central inspection device (20) according to claim 11 or 12, wherein, A confidence index indicating the confidence level is determined when the local neural network (11) is trained, and the given threshold is empirically determined when the local neural network (11) is trained.

14. The central inspection device (20) according to claim 11 or 12, wherein, A confidence index indicating the confidence level is determined by relating the activation patterns exhibited by multiple nodes of the local neural network (11) when determining the local classification result to a reference activation pattern exhibited by the multiple nodes of the local neural network (11) during training of the local neural network (11), and wherein The given threshold includes a correlation threshold that indicates a predetermined correlation level.

15. The central inspection device (20) according to claim 11 or 12, wherein, A confidence index indicating the confidence level is determined by correlating at least one feature vector obtained by the local neural network (11) while determining the classification result with at least one corresponding reference feature vector obtained by the local neural network (11) while training the local neural network (11).

16. An inspection system for determining whether an article is defective or non-defective by using at least one of a local neural network (11) and a central neural network (21) based on sensing measurements performed on the article while it is being processed. The item in question is a product manufactured on a production line. The process includes manufacturing the article on the production line and / or moving the article on the production line. The local neural network has fewer computational resources than the central neural network. in, In response to determining that the local confidence index is not higher than a predetermined confidence level, the local confidence index indicating the confidence level of a correct local classification result, the central neural network (21) is configured to determine whether the obtained sensing measurement represents a defective item or a non-defective item in the central classification result. Furthermore, the central classification result is used as the classification result of the inspection system.

17. The inspection system according to claim 16, wherein, The confidence index is determined when the local neural network (11) is trained, and a given threshold is empirically determined when the local neural network (11) is trained.

18. The inspection system according to claim 16 or 17, wherein, The confidence index is determined by relating the activation patterns exhibited by multiple nodes of the local neural network (11) when determining the local classification result to a reference activation pattern exhibited by the multiple nodes of the local neural network (11) during training of the local neural network (11). And the given thresholds include a relevant threshold that indicates a predetermined level of relevance.

19. The inspection system according to claim 16 or 17, wherein, The confidence index is determined by correlating at least one feature vector obtained by the local neural network (11) while determining the classification result with at least one corresponding reference feature vector obtained by the local neural network (11) while training the local neural network (11).

20. A computer program product comprising a processor and a memory storing a computer program including instructions configured to cause the processor to perform the method according to any one of claims 1 to 5 when executed on a computer.

21. A medium comprising instructions configured to perform the method according to any one of claims 1 to 5 when executed on a computer.

Images