Method for obtaining an identifier dataset for an industrial installation

Abstract

The invention relates to a method (100) for obtaining an identifier dataset for a plurality of objects to be inspected recurring along an industrial installation, on the basis of a sequence of successive images captured along the installation. In order to avoid redundant evaluation and to achieve an improved assignment of imaged objects to real objects at the location of the installation, it is proposed to assign an identifier to the recognized objects and to record said identifier in an identifier dataset.

Description

[0001] Description

[0002] Method for obtaining an identification data set for an industrial plant

[0003] The present invention relates to a method for obtaining an identification data set for a plurality of objects to be inspected that recur along an industrial facility based on a sequence of consecutive images recorded along the facility. Furthermore, the invention relates to a method for the visual inspection of such objects.

[0004] An example of such systems are conveyor belt systems for conveying bulk materials, for example. The bulk material, such as spoil, ore, fuels or building materials, is transported by means of a closed conveyor belt supported and guided by a plurality of support rollers. Conveyor belt systems can also be referred to as conveyor belt systems or belt conveyor systems. The support rollers are usually rotatably mounted in support structures, so-called support roller chairs, which are evenly spaced apart along the conveyor line. A support roller chair can have one or more support rollers. The support rollers can be rigid or movable, e.g. in the form of a support roller garland, in the support roller chair. In this sense, support rollers and support roller chairs are understood as objects that recur along the conveyor belt system.

[0005] Conveyor belt systems can have conveyor lines several kilometers long, which usually include several hundred support rollers per kilometer of conveyor line. For each individual support roller, operational and / or weather-related wear and tear, such as increased running resistance caused by defective bearings, can lead to increased energy consumption of the system or even damage to the conveyor belt. Due to the size of such systems and the associated high number of site-specific support rollers, an inspection of all support rollers in the system always involves a significant logistical effort. One option for carrying out an inspection of conveyor belt systems is to use a drone or other unmanned vehicle to take images along the system and then evaluate them to determine the functional status of the support rollers.Such a method is described, for example, in DE 102020206497 A1. Based on recorded thermal images, increased heat development in the area of ​​a support roller is detected, which indicates increased running resistance and thus a defect.

[0006] To enable maintenance of the support roller identified as defective in the thermal image, it is necessary to determine the exact location of this roller within the system so that, for example, maintenance personnel can be dispatched to the roller's location. The GPS data of a recorded image, usually recorded as metadata, cannot be used to determine the location of a support roller depicted in it with sufficient precision, partly because the stored GPS data indicates the location of the drone and not the roller itself.

[0007] As a solution to this problem, DE 10 2020 206 497 A1 proposes a prior survey of the entire system and the creation of a 3D model in which each roller is assigned a precise GPS position. The necessary preparatory work on the system to be inspected, as well as the process of creating the 3D model, are extremely time-consuming and must be carried out by specialized experts, which increases the overall cost of an inspection. Alternatively, DE 10 2020 206 497 A1 proposes attaching an RFID transponder to each idler and using appropriate readout electronics in the drone to enable the images to be clearly assigned to the rollers. This also requires time-consuming and costly preparatory work on the system. In addition, RFID transponders can be damaged by weather or during operation of the system and then have to be replaced, which is a complex process.

[0008] JP 2019 018999 A also discloses an inspection of

[0009] Conveyor belt systems based on the evaluation of images recorded along the system. The images are recorded using a rail-bound vehicle and compiled into a panoramic image, which is intended to enable better determination of the position of individual rollers. Due to the distance between the vehicle and the system, which is defined and, in particular, constant, the distance of the vehicle to the system can be better correlated (e.g. via GPS) with the position of individual rollers. However, rail-bound vehicles always require a rail system extending along the entire system, the installation of which is associated with a high level of labor and expense. Many systems are unsuitable for inspection using rail-bound vehicles due to their size or area of ​​application. Rail-bound vehicles are also generally much slower and consume more energy than free-moving or flying drones.According to JP 2019 018999 A, processing into a panoramic image also requires that consecutive images overlap by more than 50%, which further limits the maximum travel speed of the rail-bound vehicle along the system. Furthermore, such a high overlap results in the need to record and store a large amount of redundant data.

[0010] The object of the present invention is therefore to provide a method that facilitates the evaluation of images of objects to be inspected recorded along an industrial plant, enables precise assignment of the objects depicted and detected in the images to real objects in the plant, and places the lowest possible demands on the nature of the image acquisition process and the plant's infrastructure. In particular, image sequences recorded using freely moving or flying drones should be able to be evaluated efficiently and reliably.

[0011] This object is achieved by a method for obtaining an identification data set having the features of claim 1 and by a method for visual inspection having the features of claim 4. Preferred features are the subject of the dependent claims. Further advantages and features can be found in the general description and the exemplary embodiments.

[0012] The computer-implemented method according to the invention for obtaining an identification data set for a plurality of objects to be inspected, in particular support rollers and / or support roller chairs, which recur along an industrial plant, in particular a conveyor belt plant, on the basis of a sequence of consecutive images recorded along the plant, comprises the following steps:

[0013] Receiving an output image of the sequence of images depicting an object of the plurality of objects;

[0014] Detecting the object in the source image and assigning an identifier to the object;

[0015] Receiving a subsequent image of the sequence of images, which has been recorded from a different perspective than the initial image and on which the object and a further object of the plurality of objects not shown in the initial image are shown;

[0016] Detecting the object in the subsequent image and assigning the identifier to the object;

[0017] Recognizing the additional object and assigning a new identifier to the additional object;

[0018] Optional: Repeat the previous steps with the following image as the new starting image;

[0019] Capturing the identifiers of the detected objects as an image-specific identifier data set.

[0020] In this way, each object detected in one or more images can be assigned a unique identifier, e.g. in the form of an identification number or character string, and recorded in an identification data set. The identifier can also be referred to as an identifier or “ID”. The advantage of an identifier is that it can be assigned automatically or mechanically and initially without taking external information (such as position data, etc.) into account. The identifier can be ordered, for example, assigned as an ascending sequence of numbers. The identification data set is image-specific, i.e. the identification data set shows which identifiers were assigned on the basis of a specific image and which images are the basis for assigning a specific identifier. The identification data set can comprise one or more databases.

[0021] The object detected in the initial image and provided with an identifier is tracked or tracked in the subsequent image. In some embodiments of the invention, position and / or movement data from the recording device used to capture the sequence of images, e.g., a freely moving or flying drone, can also be used for tracking. The tracked object is preferably assigned the same identifier as in the initial image.

[0022] Alternatively, the object in the subsequent image can be assigned an identifier that is linked to the identifier of this object in the original image in a way that allows for unambiguous assignment of the identifiers. A "new" object detected in the subsequent image that is not depicted in the original image is assigned a new identifier.

[0023] The method can be applied iteratively to all images in the sequence. After all identifiers have been assigned, the subsequent image is selected as the new source image, and another subsequent image is selected. The object detection and identifier assignment are repeated. Preferably, the image in the sequence recorded immediately after the source image is used as the subsequent image.

[0024] The recording of the identifiers of the recognized objects as an image-specific identifier data set can, for example, be carried out immediately during the respective assignment or after each iteration step.

[0025] Instead of specifying a certain minimum overlap between consecutively recorded images, it is sufficient for the method according to the invention if one and the same object of the facility (or recognizable parts thereof) is depicted on two images, in particular those taken directly one after the other. Therefore, the method according to the invention is particularly suitable for image sequences recorded with freely moving or flying drones, where the recording perspectives between two images can vary significantly both translationally and rotationally. Furthermore, the method delivers easily analyzable results even with a relatively small number of images per longitudinal section of the facility to be inspected. This allows the duration of an inspection trip, particularly one carried out using a drone, to be significantly shortened and the amount of data produced and subsequently analyzed to be reduced.

[0026] The identifier assigned to the objects depicted and recognized in the images is a useful auxiliary variable that can be used for more efficient image analysis and more precise assignment to the corresponding real objects in the system. The identifier data set makes it possible to avoid redundant determinations about the functional status of an object through clever data management. Instead of evaluating all of the images in which an object is depicted, only a selection or just one of these images is evaluated and the remaining images or image areas are ignored. This makes image analysis more efficient. Furthermore, the mere assignment of an ordered identifier related to a specific reference or reference point in the system, e.g. in the form of a counting of the objects starting from one of the ends of the system, can provide a very simple way of assigning the recognized objects to the real objects in the system.In the present method, such a system-specific, ordered identifier is preferably assigned. For example, in a conveyor belt system, an assigned identifier and the route along which the sequence of images was recorded can be used to determine that a roller identified as defective by image analysis is the 42nd roller on the left side in the conveying direction of the system.

[0027] The method is based on a sequence of consecutive images recorded along the system. Images are understood to be the image data captured by electronic image sensors, such as CCD or CMOS sensors, which can be further processed using electronic data processing systems. For example, these images can have been captured by one or more cameras on a freely moving or flying drone. The images can be light images (RGB images) and / or thermal images (infrared images). Light images are somewhat better suited for object recognition due to their broader color spectrum. Thermal images are particularly suitable for detecting certain defects that are associated with increased heat generation, such as a bearing defect in a support roller.

[0028] The recognition and tracking of objects depicted in the images can be achieved using known suitable algorithms, in particular trained

[0029] Machine learning algorithms from the field of digital image processing are used. For example, "bounding boxes" can be used for object recognition. Image processing can include segmentation, particularly semantic segmentation. There are numerous suitable methods for object recognition based on image data, which is why a more detailed explanation will not be provided here.

[0030] In a preferred embodiment of the invention, an optically recognizable marker attached to the system is recognized in the initial image and / or the subsequent image, and the identifier is assigned based on the recognized marker. For example, in some conveyor belt systems, the idler rollers are provided with numbers or other markings using plates.

[0031] In a preferred embodiment of the invention, a distance between two neighboring objects depicted in the initial image and / or the subsequent image is determined, wherein a warning message about a potentially unrecognized object is recorded in the identification data set if the distance exceeds a predetermined threshold value or a threshold value determined on the basis of the sequence of images. In this way, the identification data set can be used to check whether the sequence of images also depicts all objects to be inspected and thus a complete inspection of the system can be ensured. For example, on the basis of a received warning message, an image recording can be repeated in an area of ​​the system where the potentially overlooked object is to be expected. Position data of the respective image recording position (e.g. in the form of GPS (meta) data) can advantageously be used for this purpose.Preferably, position data of the respective image recording position of the images are recorded in the identification data set.

[0032] The distance threshold can be determined based on previously known information about the system. For example, the specification of a conveyor belt system typically includes the nominal distance between the idler rollers and / or idler carriages, which can be taken into account when setting the threshold. The distance between imaged objects can also be statistically determined without prior information based on the sequence of images and taken into account when setting the threshold.

[0033] According to what has already been described above and what has been described below, the problem set out at the outset is also solved by a method having the features of claim 4.

[0034] The computer-implemented method according to the invention for the visual inspection of a plurality of objects to be inspected, in particular support rollers and / or support roller chairs, which recur along an industrial plant, in particular a conveyor belt system, comprises the following steps:

[0035] Obtaining an identification data set according to one of claims 1 to 3; determining a functional state of the objects;

[0036] Recording the respective functional status of the objects in the identification data record.

[0037] All the advantages described above and below are also realized in the inspection process.

[0038] In a preferred embodiment of the invention, the functional status of the objects is determined based on the sequence of images. The sequence of images can include light images and / or thermal images. In some embodiments, light images can be used for object recognition and creation of the identification data set, and the thermal image can be used to determine the

[0039] The functional status of the objects can be used. Even if photographs and thermal images are taken with different cameras, the relative position of the cameras is usually known, e.g., if both cameras are attached to a drone, so that the photographs can be reliably matched to the thermal images.

[0040] The identification data set can of course also be advantageously used in combination with inspection data other than visual, for example acoustic data recorded during the acquisition of the sequence of images.

[0041] In a further preferred embodiment of the invention, the functional status of the objects is determined based on a true subset of the sequence of images, with the subset being selected based on the identification data set. This allows for more efficient image evaluation. For example, redundant determinations regarding the functional status of an object can be avoided by evaluating only a selection or just one of these images instead of all images depicting an object, while ignoring the remaining images or image regions.

[0042] The invention is also directed to a computer program product for carrying out the method according to one of claims 1 to 3 and / or the method according to one of claims 4 to 6 on a computer.

[0043] In all embodiments of the inventions, the step of recording a sequence of successive images along the system, in particular by means of a freely moving or flying drone, can be part of the method according to the invention.

[0044] It is expressly pointed out that the above-mentioned

[0045] Embodiments of the invention can be combined individually or in any technically reasonable combination with each other with the subject matter of the independent claims.

[0046] Modifications and embodiments of the invention, as well as further advantages and details of the invention, can be found in the following description and the drawings. The schematic figures show:

[0047] Fig. 1 is a flow diagram of an embodiment of the inventive

[0048] Method for obtaining an identification data set;

[0049] Fig. 2 shows a flow chart of a control procedure for checking the

[0050] Completeness of the recognized objects, which can be part of the inventive method according to Fig. 1;

[0051] Fig. 3 shows an output image of a sequence of images taken along a conveyor belt system as can be used in a method according to the invention according to Fig. 1;

[0052] Fig. 4 shows a subsequent image of the sequence of images according to Fig. 3;

[0053] Fig. 5 a section of an image of a along a

[0054] Conveyor belt system, the support roller chairs of which are provided with optical markers, recorded sequence of images as can be used in a method according to the invention according to Fig. 1;

[0055] Fig. 6 is a flow diagram of an embodiment of the inventive

[0056] Method for the visual inspection of a number of objects to be inspected that are recurring throughout an industrial facility. Parts that are identical or similar in appearance are provided with identical reference numbers where appropriate.

[0057] Individual technical features of the embodiments described below can also be combined with previously described embodiments as well as the features of the independent claims and any further claims to form subject matter according to the invention.

[0058] Fig. 1 shows an embodiment of the method 100 according to the invention for obtaining an identification data set for a plurality of objects to be inspected that recur along an industrial plant, based on a sequence of consecutive images recorded along the plant, comprising the following steps:

[0059] Receiving an output image of the sequence of images depicting an object of the plurality of objects (step 110);

[0060] Detecting the object in the source image and assigning an identifier to the object (step 120);

[0061] Receiving a subsequent image of the sequence of images, recorded from a different perspective than the original image, on which the object and another object of the plurality of objects not depicted in the original image are depicted (step 130); Detecting the object in the subsequent image and assigning the identifier to the object (step 140);

[0062] Recognizing the further object and assigning a new identifier to the further object (step 150);

[0063] Optional: Repeat the previous steps with the following image as the new starting image (step 160);

[0064] Capturing the identifiers of the detected objects as an image-specific identifier data set (step 170).

[0065] The functioning of the method according to the invention is explained below with reference to Figures 3 to 5 using an embodiment of a conveyor belt system 2 in which the objects to be inspected are support rollers 10, 11, 12, 13, 14.

[0066] An exemplary initial image 300, as can be used in a method according to Fig. 1 or 2, is shown in Fig. 3. An associated subsequent image 400 is shown in Fig. 4. The initial image 300 shows a section of the conveyor belt system 2. The conveyor belt system 2 comprises a closed conveyor belt 4, supported and guided by a plurality of support rollers 10, 11, 12 for transporting, for example, bulk material. The support rollers 10, 11, 12 are rotatably mounted in support roller chairs 10', 11, 12' arranged along the conveying path (cf. conveying direction F). The support rollers 10, 11, 12 are evenly spaced from one another, with the upper support rollers 10, 12 having a distance D.

[0067] On the received output image 300 (Fig. 3), an object in the form of a support roller 10, 11, 12 can now be recognized, which is represented by dashed bounding boxes. Each support roller 10, 11, 12 can be assigned a respective identifier 30, 31, 32, which can also be referred to as an ID. In the present case, for example, the support roller 10 is assigned the identifier 30 ("15"). The other support rollers 11, 12 are numbered in ascending order from left to right by assigning the identifiers "16" and "17".

[0068] The received subsequent image 400 (Fig. 3) is recorded from a different perspective than the initial image 300, here from a position offset in the conveying direction F. The subsequent image 400 shows the support rollers 12, 13, 14. The support roller 12 is tracked in the subsequent image 400, i.e. it is determined that this support roller is the same support roller as is shown in the initial image 300. The support roller 12 is therefore assigned the same identifier 32 ("17") as in the initial image 300. Alternatively, a different identifier can also be assigned, which can be uniquely assigned to the identifier 32 in the initial image 300. The other support rollers 13, 14 recognized in the subsequent image 400, which were not already recognized in the initial image 300, are assigned new identifiers 33, 34, here further ascending numbers “18” and “19”.

[0069] Assigning an identifier in this way has many advantages, particularly in the field of visual inspection of support rollers. The identification data set, which shows, for example, which images in the sequence show a specific support roller, makes it possible to avoid redundant determinations about the functional status of a support roller through clever data management. Instead of evaluating all of the images in which the support roller is shown, only a selection or just one of these images is evaluated and the remaining images or image areas are ignored. Furthermore, the assignment of an ordered identifier related to a specific reference or reference point of the conveyor belt system, e.g. as in Figures 3 to 5 in the form of a count of the support rollers starting from one of the ends of the system, can provide a very simple way of assigning the support rollers identified in the images to real support rollers at the location of the conveyor belt system.A key advantage of an identifier is that it can be assigned automatically and initially without taking external information (such as location data, etc.) into account.

[0070] Fig. 5 shows an image section 500 in a further embodiment of the invention, in which each support roller frame 10' of the conveyor belt system 2 is provided with a visually recognizable marker 50 in the form of a number plate. In this case, the information of the marker 50 can be visually recognized and taken into account when assigning the identifier 30. In the present case, the number of the support roller frame is "42," so that the support roller 10 is also assigned the number "42."

[0071] Fig. 2 shows a control method 200 for checking the completeness of the detected objects, which can be part of the method 100 according to the invention. Here, a distance between two neighboring objects depicted in the initial image and / or in the subsequent image is determined (step 210). The distance is then compared with a predetermined threshold value (step 220). If the threshold value is exceeded, a warning message about a potentially unrecognized object is recorded in the identification data record. For example, with a view to Fig. 3, a distance D between the upper support rollers 10, 11 should remain constant over the entire conveyor section of the conveyor belt system 2. In the event of deviations or if the threshold value is exceeded, it can thus be determined that not all support rollers have been detected. The threshold value for the distance can be determined based on previously known information.Alternatively or additionally, the distance between imaged objects can be statistically determined based on the sequence of images and taken into account when setting the threshold value.

[0072] Fig. 6 shows an embodiment of the method 600 according to the invention for the visual inspection of a plurality of objects to be inspected that recur along an industrial plant, comprising the following steps:

[0073] Obtaining an identification data set according to one of claims 1 to 3 (step 610);

[0074] Determining a functional state of the objects (step 620); recording the respective functional state of the objects in the identification data record (step 630).

Claims

Patent claims 1. Computer-implemented method (100) for obtaining an identification data set for a plurality of objects to be inspected, in particular carrying rollers (10, 11, 12, 13, 14) and / or carrying roller chairs (10', 1T, 12', 13', 14'), recurring along an industrial plant, in particular a conveyor belt plant (2), based on a sequence of successive images recorded along the plant, comprising the following steps: Receiving an initial image of the sequence of images, on which one object of the plurality of objects is depicted (step 110); Identifying the object in the initial image and assigning an identifier to the object (step 120); Receiving a subsequent image from the sequence of images, recorded from a different perspective than the initial image, and showing the object as well as another object not shown in the initial image (step 130); recognizing the object in the subsequent image and assigning the identifier to the object (step 140); Identifying the additional object and assigning a new identifier to the additional object (step 150); Optional: Repeat the preceding steps with the next image as the new starting image (step 160); Capture the identifiers of the detected objects as an image-specific identifier data set (step 170).

2. Method according to claim 1, wherein the identifiers are plant-related ordered identifiers.

3. Method according to claim 1 or 2, wherein a distance between two adjacent objects depicted in the initial image and / or in the subsequent image is determined, wherein a warning message about a potentially non-existent object is included in the identifier data set. A detected object is captured when the distance exceeds a predetermined threshold or a threshold determined based on the sequence of images.

4. Computer-implemented method (600) for the visual inspection of a plurality of objects to be inspected, in particular carrying rollers and / or carrying roller chairs, recurring along an industrial plant, in particular a conveyor belt plant, comprising the following steps: Obtaining an identifier data set according to one of claims 1 to 3 (step 610); Determining the functional state of the objects (step 620); recording the respective functional state of the objects in the identifier data record (step 630).

5. Method according to claim 4, wherein the determination of the functional state of the objects is based on the sequence of images.

6. Method according to claim 5, wherein the determination of the functional state of the objects is based on a real subset of the sequence of images, wherein the selection of the subset is based on the identifier data set.

7. Computer program product comprising program code stored on a computer-readable medium for executing a method according to any one of claims 1 to 6.

Images