Method and arrangement for optical monitoring of a process step in an industrial plant

Abstract

In one embodiment, a method for monitoring a process step in an industrial plant comprises optically detecting at least one movement of an object and generating corresponding image data, wherein the object is at least partially guided manually by a worker during the process step, generating at least one object trajectory as a function of the at least one detected movement based on the image data using an object recognition method, wherein the at least one object trajectory includes time information and associated location information, and storing several images of the object based on the at least one object trajectory and the image data as sample object images.

Description

[0001] The present invention relates to a method and an arrangement for optically monitoring a process step in an industrial plant.

[0002] In industrial production processes, products such as assembled semiconductor circuit boards are manufactured in several steps. The degree of automation is constantly increasing and now also extends to quality control of the finished product. The process steps considered in this application are part of a production or logistics process and are carried out, at least partially, manually with the involvement of a person, referred to below as a worker. The worker works at a static workstation that is part of an industrial plant.

[0003] It is known that to check the quality of a product or sub-product created in a process step—for example, to verify the completeness of an assembly created in that step—an image of this product is analyzed in a separate process. In this quality control process, the static image of the final product is compared with an image showing a target state of the product, such as a correctly assembled component. However, this approach places high demands on image quality. For example, the image of the product created in the process step must have a relatively high resolution and should ideally be free of reflections. Furthermore, training an algorithm used in this automated quality control requires a significant investment of time and extensive datasets. Despite the measures known so far, it is difficult, if not impossible, to analyze particularly small or very small components in this way.Reliable detection of transparent components is challenging. Different lighting conditions and reflections in the image of a processed assembly already lead to detection problems.

[0004] Therefore, one task is to further improve the quality control of process steps that are at least partially performed manually.

[0005] This problem is solved by the method according to claim 1 and by the arrangement according to claim 10. Advantageous embodiments of the invention can be found in the description, the drawings, and the dependent claims.

[0006] According to this method, a process step in an industrial plant is visually monitored. Specifically, at least the following steps are performed: - At least one movement of an object is optically detected and corresponding image data is generated, whereby the object is at least partially guided manually by a worker during the process step, - Depending on the at least one detected movement, at least one object trajectory is generated using the image data and an object recognition method, wherein the at least one object trajectory includes time information and associated location information, - Several images of the object are stored as sample object images based on at least one object trajectory and the image data.

[0007] Using the proposed method, the movement of an object—for example, a component, workpiece, or tool—which the worker moves by hand, manually with a tool, or by machine, is optically captured and recorded during the movement itself, i.e., during the process step. Image data, such as a video sequence comprising a series of individual frames, is generated. Based on this image data, the object trajectory is created. The object trajectory is a path, primarily two-dimensional, that abstractly reflects the movement experienced by the object. The object recognition method is used to identify the object within the image data. Based on the object trajectory and the image data, several images of the object are stored as template images. A template image serves as a model or model, which...which shows the object in such a way that it is recognizable. One or more sample object images can be referred to as a sample object data set.

[0008] By tracking the movement of the object according to the invention, pattern images are generated, each showing the object from a different viewing angle. The pattern images generated in this manner are preferably used to train the algorithm employed, i.e., the object recognition method. This leads to a significant reduction in the training effort required by prior art solutions. At the same time, the reliability of the object recognition is increased, resulting in improved quality control. Preferably, no optical markers, such as coding symbols in the form of barcodes or matrix codes, are required for this object recognition.

[0009] Saving primarily involves permanent saving.

[0010] According to further training, the object recognition method includes artificial intelligence (AI). This AI includes an artificial neural network, which can be implemented in particular as a convolutional neural network (CNN).

[0011] Such AI, especially a CNN, is suitable for detecting objects, such as screws, surface-mountable components or similar items, and enables reliable recognition of the respective object.

[0012] According to a training course, the AI ​​is trained using the stored sample object images.

[0013] In the training of the AI ​​used in the object recognition method, the sample object images each represent the so-called labeled data, i.e., target images of the object. The label attached to a sample object image enables the identification of good and bad images. The sample object images generated and stored during the process step are used to train the AI ​​of the object recognition method. Since, according to the invention, the sample images are generated based on the object's movement in space, a large number of sample images, each with a different orientation of the object, are automatically available.

[0014] Specifically, the underlying algorithm of the object recognition method, such as AI, is improved and refined over time through training and machine learning, starting from a coarse model. The starting point is, for example, sample object images generated when the object is moved correctly or as intended. The coarse model here refers, for instance, to an initial label for the object, where the label (e.g., a tag) is linked to a (small) number of sample images of the object. When using the coarse model, the detection rate is not yet optimal or is only acceptable under ideal conditions. In this initial training phase, a tagging method based on a small number of images can be used, or recognition based on a general model can be employed.An example of the latter is the recognition of a screw without considering further details such as its type (Phillips, slotted, Torx), length, color, or similar characteristics. The probability of recognition increases with the amount or scope of the training data. In other words, "coarse" refers to an initial labeling with a still low probability of recognition. Subsequently, the object recognition method, i.e., the underlying AI, trains itself by performing autonomous tagging based on knowledge of an object and its position.

[0015] According to another embodiment, the AI ​​training is carried out on a computing unit outside the industrial plant in advance and / or afterwards.

[0016] The computing unit includes, for example, a processor, which can be implemented as a CPU or GPU. The processor can also be specifically designed for the implementation of a neural network.

[0017] Due to the limited computing power typically available within an industrial plant, AI training is preferably conducted off-site on suitable hardware. The AI ​​is therefore trained before being run in the industrial plant. It can also be retrained at any time. When implementing the AI ​​using a neural network, the weights and / or activation functions used in the network are continuously adjusted during training. This training can be supervised or unsupervised. The adjustment of weights and the alternative or additional adjustment of the neural network's activation functions are carried out with the aim of minimizing a predefined loss function, thus ultimately reducing the deviation between the actual and target results.

[0018] In a further training course, the object recognition method includes zero-shot object recognition.

[0019] Zero-shot object recognition is a method from computer vision for recognizing objects and their classes in images without prior training or knowledge of the classes. For example, a model on which zero-shot object recognition is based receives an image showing the desired object as input, along with a list of candidate classes. The model then provides bounding boxes within which the object has been detected. In an example scenario, zero-shot object recognition is used for the initial detection of components or objects without using sample images or with a small number of sample images. When a worker presents a particular object for the first time, the object recognition method can independently identify the object, at least roughly, for example, that it is or could be a housing.The operator can then confirm or correct the detection.

[0020] The proposed method advantageously allows for detector training with reasonable effort, so that when monitoring the process step, a targeted object is reliably detected and tracked in space. This enables the quality of the process step and its result to be monitored and assessed during execution. The detector is essentially represented here by the object recognition method.

[0021] According to a further training, at least one object trajectory begins at a known starting position. Alternatively, at least one object trajectory begins at a known starting position and ends at a known end position. The starting position corresponds to the initial position of the object's movement. The end position corresponds to the target position of the object's movement.

[0022] Using a known starting position and optionally a known target position simplifies object detection and the creation of the object trajectory. This further optimizes the process, as object detection at the beginning of the movement, and optionally at the end, can be limited to the image area containing the starting or target position. Furthermore, even if the object is temporarily obscured during movement, for example by a tool, the object trajectory can be completed through interpolation or extrapolation. This allows the quality of the process step to be assessed despite this temporary obscuration.

[0023] In a further embodiment, storing multiple images of the object based on the at least one motion trajectory and the image data as template object images comprises: analyzing the image data, which consists of a sequence of individual images, and detecting the object in several individual images of the image data. The detection is based on the time and / or location information of the object trajectory.

[0024] To obtain sample object images that subsequently serve as labeled representations of the object, the image data is evaluated using the time and / or location information of the object's trajectory. The analysis can be limited to the image area in which the object is currently located according to its trajectory.

[0025] Knowing the object's start and end positions allows the object recognition method to be automatically improved over time with repeated execution of the process, based on the knowledge of the object's position within the image. Since multiple images are used to recognize an object, the requirements for the quality of a single image, such as its resolution or lighting, are lower. This further reduces the costs associated with implementing the process.

[0026] According to further training, the storage process also includes: storing several individual images, each showing the detected object, with each individual image being a sample object image, or alternatively, generating a section of an image from one or more of the individual images, each showing the detected object, with the respective image section comprising an image area of ​​the individual image that shows the object, and storing the respective image section as a sample object image.

[0027] Accordingly, a template object image is either a single frame of the image data recorded during the optical detection of at least one movement, or a section of such a single frame. In either case, the template object image shows the object during the execution of the process step. Using only a section of the image reduces the storage space required. Furthermore, the reduced amount of data allows for faster object recognition.

[0028] The sample object images can, for example, be stored permanently in a database.

[0029] According to another embodiment, the proposed method is repeated several times for at least one movement.

[0030] This generates a large number of additional sample object images, further improving the reliability of object recognition. The large number of sample object images generated in this way is used to continuously refine and improve the process, particularly the object recognition method employed. This can be achieved, for example, through reinforcement learning. This results in robustness to environmental changes at a workstation where the process step is performed.

[0031] A further aspect of the present invention is an arrangement for the optical monitoring of a process step in an industrial plant. The arrangement comprises a camera, a processing unit, and a storage unit, which are interconnected. The camera is configured to optically detect at least one movement of an object, wherein the object is at least partially guided manually by a worker during the process step. The processing unit is configured to generate at least one object trajectory based on the at least one detected movement using an object recognition method and to store several images of the object in the storage unit as sample object images based on the at least one object trajectory and the image data. The at least one object trajectory includes temporal information and associated spatial information.

[0032] The process step is performed at a static workstation within the industrial plant. The arrangement according to the invention is located at this workstation. The camera preferably comprises a video camera. For example, a so-called field-of-view camera is used, which is either permanently mounted at the production workstation or attached to the worker, for example, to their head or shoulder. In one embodiment of the invention, one or more additional cameras can be used for optical acquisition. In such an implementation, the video streams generated by the multiple cameras are combined in a suitable manner to produce the image data according to the claim. Advantageously, when using two cameras, 3D data, for example, three-dimensional trajectories or spatial curves, can be generated.

[0033] The processing unit is implemented, for example, on the basis of a microprocessor. The storage unit comprises permanent memory.

[0034] The arrangement according to the invention enables the generation of suitable training data, namely the sample object images, for optical monitoring of a process step in an industrial plant, thereby reducing the training effort. The optical requirements underlying object recognition are also reduced. Despite this, it is possible to reliably recognize even small or transparent objects, so that the quality of the activity performed in the process step can be assessed.

[0035] Furthermore, the explanations regarding the method according to the invention apply accordingly to the arrangement, this applies in particular with regard to advantages and embodiments.

[0036] The embodiments described herein can be combined with one another unless explicitly stated otherwise.

[0037] In particular, the arrangement according to the invention is designed to carry out the method according to the invention.

[0038] In an exemplary process step, the inventive method or arrangement is used to monitor the assembly of small screws in mobile phone assembly. The aim is to verify whether all screws have been installed. In the inventive monitoring of this assembly step, a screw consisting of a head and screw body is captured multidimensionally from multiple spatial directions using a multitude of images during its fixation on a screwdriver and during its movement towards the drilled hole. This allows the screw to be detected more easily and reliably and tracked based on its position in space. Since the starting and ending positions of the components are known for repetitive process steps, the screw's position can be determined by extrapolation even if it is temporarily obscured in the image.If, however, as is known from the prior art, quality is checked solely on the basis of a static image after completion of the manual process step, then in this example only the small screw heads, which make up a small proportion of the overall image, are recognizable, which makes quality control more difficult and impairs its reliability.

[0039] In a further embodiment, by detecting a tool used in the process step, provided it is easily detectable, the object recognition method can be used to infer the component's position, thus significantly narrowing the image area to be examined. This is particularly advantageous for very small components, such as mounting screws, or components that are difficult to detect optically, such as lenses, filters, or similar items. These are thus detected by their contact with a screwdriver, tweezers, suction cup, or similar tool.

[0040] The invention is described below by way of example only, with reference to the drawings. Functionally or effectively equivalent drawing elements bear the same reference numerals. The drawings show: Fig. 1 a schematic representation of an exemplary manufacturing workstation, Fig. 2A to 2C schematic diagrams of an exemplary process step, and Fig. 3. An example of zero-shot object detection.

[0041] Fig. Figure 1 shows a schematic representation of an exemplary production workstation at which the method or arrangement according to the invention is used. The production workstation is part of an industrial plant. A person, here a worker 10, works at the exemplary production workstation 30 using their right hand 11 and left hand 12. For example, a soldering iron 13 is guided with the right hand 11. A solder 14 is held with the left hand 12. A component 15 to be soldered is placed on a circuit board 16. A stationary camera 20 is mounted in relation to the worker 10 and the production workstation 30 such that it can optically record the movements of the worker 10. The recorded movements are evaluated in an evaluation unit 21 connected to the camera 20. The sample object images are stored in the storage unit 22.

[0042] In the exemplary presentation of Fig. Worker 10 has already taken the soldering iron 13 from its rest position 31 at the production workstation 30 and positioned it on the component 15 to be soldered. Furthermore, worker 10 has taken the solder 14 from its rest position 32 at the production workstation 30 and also moved it to the component 15 to be soldered.

[0043] The Fig. Figures 2A to 2C show exemplary schematic representations relating to an exemplary process step in which the method according to the invention is used. The process step is carried out, for example, at a manufacturing workstation as in Fig. As shown in Figure 1, a pad 17 is placed on a circuit board 16.

[0044] Fig. Figure 2A shows the initial situation for this process step, in which one of the worker's hands, here his left hand 12, is in a starting position 35. The component 17 is in a starting position 36.

[0045] Fig. Figure 2B shows a schematic representation of a movement performed in this process step. The movement of component 17 is depicted as movement B6. Movement B6 begins with hand 12 grasping component 17 at the starting position 36 and ends with placing pad 17 at the target position 37 on the circuit board 16.

[0046] Fig. 2C shows the optically detected movement B6 from Fig. 2B generated object trajectory G6. Object trajectory G6 is represented relative to a spatial distance (shown on the z-axis) to the target position 37 of component 17 at Z=0, in relation to time t. At time t0, trajectory G6 begins at the starting position 36. At time t1, trajectory G6 ends at the target position 37.

[0047] The execution of movement B6 is done with a camera, for example camera 20. Fig. 1. Optically captured, i.e., a large number of images are generated, each showing at least component 17. The generated images may also show, in addition to component 17, hand 12 and / or circuit board 16, as well as the production workstation in the background. Due to the movement, component 17 is shown in different orientations in the images. Using the object recognition method, component 17 is recognized as the moving object in each of the generated images. Accordingly, the object trajectory G6 is generated. Several of the generated images, which show an image of component 17, are stored as sample object images, either as a single image or as a section of a single image showing component 17, for example in storage unit 22. Fig. 1.

[0048] These sample object images are used, for example, to train an AI encompassed by the object recognition method. This AI can also be trained on an external, not shown, computing unit. The trained AI can then be transferred back to the evaluation unit 21 in Fig. 1 will be transferred.

[0049] By tracking the movement B6 in space according to the invention, a large number of image data points with different orientations of the object 17 are generated. The object recognition method therefore "knows" which object it is and where it is located in the image. Starting from a coarse model, the object recognition method improves automatically with each iteration of the proposed method.

[0050] Fig.Figure 3 shows an example of zero-shot object recognition that can be used in the described process. The image shows a snapshot during a process step in which a thermally conductive mat, here referred to as HeatMat, is placed on a circuit board, referred to as PCBA, using tweezers. The tweezers are guided by the worker's hand. The object recognition method has identified the three components and placed a bounding box R10 on the circuit board, a bounding box R20 on the tweezers, and a bounding box R30 on the thermally conductive mat. Through further training and / or self-learning, the object recognition method can be further developed to recognize, for example, a thermally conductive mat of a different color or size. Reference symbol list 10 workers 11, 12 Hand 13 soldering irons 14 solder 15 building elements 16 circuit boards 17 Component 20 cameras 21 evaluation unit 22 storage units 30 production workstations 31, 32, ..., 37 Position B1 Movement G1 Object trajectory t0, t1 Time R10, R20, R30 Boundary frames

Claims

[1] comprising a method for optical monitoring of a process step in an industrial plant: optical detection of at least one movement (B1) of an object (17) and generation of corresponding image data, wherein the object (17) is guided at least partially manually by a worker (10) during the process step, Generating at least one object trajectory (G1) depending on the at least one detected movement (B1) using the image data and an object recognition method, wherein the at least one object trajectory (G1) includes time information and associated location information, Storing multiple images of the object (17) based on the at least one object trajectory (G1) and the image data as sample object images. [2] Method according to claim 1, wherein the object recognition method comprises an artificial intelligence, AI, wherein the AI ​​comprises an artificial neural network, in particular a convolutional one. [3] Method according to claim 2, wherein the AI ​​is trained with the stored pattern object images. [4] Method according to claim 3, wherein the training of the AI ​​is carried out on a computing unit outside the industrial plant beforehand and / or afterwards. [5] Method according to any of the preceding claims, wherein the object recognition method comprises zero-shot object recognition. [6] Method according to any of the foregoing claims, wherein at least one object trajectory (G1) begins at a known starting position (36), or wherein the at least one object trajectory (G1) begins at a known starting position (36) and ends at a known final position (37) and where the starting position (36) corresponds to a starting position of the movement of the object (17) and where the end position (37) corresponds to a target position of the movement of the object (17). [7] Method according to one of the preceding claims, comprising storing multiple images of the object (17) based on the at least one object trajectory (G1) and the image data as template object images: Analyzing the image data, which comprises a sequence of individual images, and Detecting the object (17) in several individual frames of the image data, wherein the detection is based on the time and / or location information of the object trajectory (G1). [8] Method according to the preceding claim, wherein the storage further comprises: Storing multiple individual images showing the detected object (17), each individual image being a sample object image, or Generating a section of an image from one or more of the individual images showing the detected object (17), wherein the respective section of an image comprises an image area of ​​the individual image showing the detected object (17), and storing the respective section of an image as a sample object image. [9] Method according to any of the preceding claims, wherein the method is repeated several times for the at least one movement (B1). [10] Arrangement for optical monitoring of a process step in an industrial plant comprising a camera (20), a processing unit (21) and a storage unit (22) which are coupled to each other, wherein the camera (20) is set up to optically detect at least one movement (B1) of an object (17), wherein the object (17) is guided at least partially manually by a worker (10) during the process step, and wherein the processing unit (21) is configured to generate at least one object trajectory (G1) depending on the at least one detected movement (B1) using the image data and an object recognition method, and to store several images of the object (17) based on the at least one object trajectory (G1) and the image data as sample object images in the storage unit (22), wherein the at least one object trajectory (G1) includes time information and associated location information.

Images