Method and device for interactive annotation of image data in service operation

Abstract

The invention relates to a method for training an image recognition algorithm for fault recognition in the maintenance of machines with a plurality of components, a computer program product for implementing the method and a device for implementing the method, wherein the method comprises the following method steps: - acquiring image data of the machine by means of at least one camera and storing the image data in a database; - creating a digital model of the machine, on the basis of the acquired image data; - creating a machine coordinate system for the digital model of the machine; - performing maintenance work on at least one faulty component of the machine; - acquiring a position of the maintained component and - generating position data for the maintained component in the machine coordinate system; - marking the acquired image data for the maintained component from the digital model of the machine on the basis of the assigned position data in the machine coordinate system.

Description

[0001] Description

[0002] Method and device for interactive annotation of

[0003] Image data in service mode

[0004] The invention relates to a method for training an image recognition algorithm for fault detection in the maintenance of machines with multiple components, a computer program product for carrying out the method and a device for carrying out the method.

[0005] Artificial intelligence algorithms, particularly machine learning algorithms, are trained in a learning phase for specific tasks in order to carry out these tasks in a work phase. During the learning phase, input data, particularly image data, is often annotated by people. If, for example, people or objects in photos are to be recognized by the algorithms during the work phase, photos with the objects to be recognized are annotated accordingly during the learning phase by manually marking the objects to be recognized in the photos. A reliable algorithm requires a large number of such annotated photos. In an industrial environment, annotation is difficult for several reasons. On the one hand, experts are required for the annotation. On the other hand, the amount of data required for training is often too small, for example because defects to be detected rarely occur.

[0006] The disclosure document DE 10 2018 214 210 A1 relates to a visualization device and a method for visualizing the interior or exterior of a vehicle, in particular for planning a retrofit of a vehicle.

[0007] The invention is based on the object of proposing a method for the simplified training of an image recognition algorithm for the maintenance of complex machines. This object is achieved by the subject matter of the independent patent claims. Further developments and refinements of the invention are found in the features of the dependent patent claims.

[0008] A method according to the invention for training an image recognition algorithm for fault detection in the maintenance of machines with multiple components comprises the following method steps: a. Capturing image data of the machine using at least one camera and storing the image data in a database; b. Creating a digital model of the machine based on the captured image data; c. Creating a machine coordinate system for the digital model of the machine; d. Carrying out maintenance work on at least one faulty component of the machine; e. Capturing a position of the serviced component and generating position data for the serviced component in the machine coordinate system; f. Marking the captured image data for the serviced component from the digital model of the machine based on the assigned position data in the machine coordinate system.

[0009] The machine is in particular a vehicle, in particular a rail vehicle.

[0010] The acquisition of image data of the machine in process step a is advantageously carried out using a large number of high-resolution cameras. This allows a detailed image of the machine to be generated.

[0011] This advantageously involves capturing all visible and at least partially accessible components of the machine. This means that image data of the entire machine is recorded from the outside.

[0012] According to a further development of the invention, a three-dimensional digital model (3D model) of the machine can be created in method step b based on the acquired image data. For this purpose, the acquired image data is first read from the database. Consequently, in method step c, a machine coordinate system is created for the digital 3D model of the machine.

[0013] The machine coordinate system is created from the image data using a suitable algorithm. The origin could be a reference point on the machine that is easily identifiable by an image recognition algorithm, such as a buffer or a wheel of a rail vehicle.

[0014] According to a further development, the machine is a rail vehicle, and the acquisition of image data of the machine using at least one camera is carried out when the rail vehicle enters a maintenance depot, for example, using a so-called camera tunnel. Upon entering the depot, the rail vehicle travels through this tunnel and is photographed from several sides, in particular from all sides.

[0015] Fewer cameras can then be arranged in the depot itself and aimed at the rail vehicle in its maintenance position, for example to record the position of the component being serviced according to process step e . The rail vehicle can be classified into the machine coordinate system, for example using a reference point on the machine that is easy to identify for an image recognition algorithm, for example a buffer or a wheel of the rail vehicle . Alternatively, process step a . takes place in the depot with the rail vehicle in its maintenance position . This can equally apply to process steps b . and c . as well as d . following process step a . The depot is then equipped accordingly with a large number of, in particular high-resolution, cameras in different positions and viewing angles on the rail vehicle .The monitoring of the maintenance work according to process step d . and the recording of the position of the serviced component according to process step e . can then be carried out by the same cameras .

[0016] The maintenance work, in particular the servicing, repair or replacement of components of the machine, is carried out by trained service personnel.

[0017] According to method step e, the position of the serviced component is detected. This can be done using at least one camera, in particular using at least two cameras, for example using the cameras in the depot when the rail vehicle is in its maintenance position.

[0018] In further development, the position of the serviced component is detected in method step e using a pattern recognition algorithm based on image data that is stereoscopically captured by at least two cameras. The position of the serviced component can then be detected based on a triangulation of the at least two camera positions. For this purpose, for example, movements of the service personnel servicing the component can be monitored and tracked. Such methods are known from the field of motion capture without markers.

[0019] According to a further development, process step e . precedes :

[0020] Marking the faulty component using a marker, in particular by the service personnel. This can also follow process step c. and precede process step d. or take place during process step d. For this purpose, a technical device, for example a pen, can be used as a passive marker, or a device that emits a signal, for example a laser pointer, can be used as an active marker, with which the service personnel can physically point to the faulty component. Such methods are known as position tracking of objects with markers. A tracking unit can be provided for this purpose.

[0021] In a further development, it is provided that process step d . includes :

[0022] - Marking of a fault on the faulty component using a marker, particularly by the service personnel.

[0023] This can also generally follow process step c and precede process step d. The service personnel can, for example, mark the fault themselves, for example by pointing to the fault or a defective part of the component, such as a crack in the component's housing or leaking oil.

[0024] Further developed in process step e. In addition to the position of the serviced component, the position of the fault is also recorded. In addition to generating position data for the serviced component in the machine coordinate system, position data for the fault is also generated in the machine coordinate system. Accordingly, in process step f. , the acquired image data for the fault from the digital model of the machine can also be marked based on the assigned position data in relation to the fault in the machine coordinate system.

[0025] In process step f, the recorded position data are transferred into the image data using the machine coordinate system and the recorded image data from the digital model of the machine are marked for the maintained component and / or the fault using the assigned position data in the machine coordinate system.

[0026] In this case, at least one image section containing an image of the faulty component and / or the fault is marked. Using the machine coordinate system, it is known which point or area in the image or image data represents which part of the machine, for example, the rail vehicle.

[0027] Further training follows the process step f . after :

[0028] - Determining the serviced component and / or the fault in the marked image data for the serviced component and / or the fault using an image recognition algorithm.

[0029] This means that the serviced component and / or the error is assigned to the corresponding image data.

[0030] This can be done by comparing reference image data for the component from a reference database.

[0031] Further training includes process step d . :

[0032] Identifying the faulty or maintained component or fault, particularly by maintenance personnel, for example by selecting from a list of machine components and / or faults, and generating information on the identified component and / or fault.

[0033] The information about the identified component can in turn be stored in one or the same database.

[0034] The process step f can then be followed by:

[0035] - Linking the captured image data for the serviced component from the digital model of the machine with the information generated for the serviced component of the machine, which information is read out from the database for this purpose. Service personnel usually create a maintenance report during or after the maintenance work has been carried out, containing information about the serviced component, the type of fault and / or the maintenance work carried out. This information can be read out and easily linked to the corresponding image data. In this way, the image data for faulty components and / or faults can be annotated with little effort.

[0036] Defective components or errors can then be easily identified using the annotated image data of the machine, in particular by means of an image recognition algorithm.

[0037] Service personnel simply carry out their normal maintenance work. The faulty component or error is marked passively, by tracking the maintenance work on the faulty component, or actively, with little effort, by the service personnel pointing out the faulty component. By evaluating the maintenance report, which also corresponds to the usual activity, the component and / or error determined from the marked image data as the serviced component and / or the error using the image recognition algorithm can then be annotated using detailed information about the component and / or the error.

[0038] By means of an image recognition algorithm trained according to the method according to the invention for the detection of faults in the maintenance of machines with multiple components, the detection of faults in the maintenance of machines with multiple components can be carried out with the following method steps:

[0039] Capturing image data of the machine using at least one camera; detecting a faulty component of the machine based on the captured image data of the machine using the image recognition algorithm.

[0040] Large parts of the method, in particular method steps a , b , c , e , and f , can be carried out using a computer program product. This includes instructions which, when the program is executed by at least one suitable terminal, cause the terminal to carry out the method according to the invention. The computer program product can be stored on a data carrier.

[0041] A device according to the invention is designed to carry out the method according to the invention. It comprises the means already mentioned, suitable for carrying out the respective method step:

[0042] - At least one camera for capturing image data of the machine;

[0043] - At least one memory for storing the image data in a database;

[0044] - At least one processor for creating a digital model of the machine from the captured image data;

[0045] - At least one processor for creating a machine coordinate system for the digital model of the machine;

[0046] - At least one tracking unit comprising at least one camera for detecting a position of the serviced component and at least one processor for generating position data for the serviced component in the machine coordinate system;

[0047] - At least one processor for marking the captured image data of the serviced component from the digital model of the machine based on the associated position data in the machine coordinate system. The processors can be one and the same processor of a computing unit. The cameras can also be the same cameras, or different cameras can be used to carry out process steps a. and e.

[0048] The invention permits numerous embodiments. It is explained in more detail with reference to the following figure.

[0049] The figure shows a schematic sequence of one embodiment of the method. In a first step a, image data of a machine are captured using at least one camera, and the image data are stored in a database. From these stored image data, a digital model of the machine is then created in method step b. Furthermore, a machine coordinate system is created for the digital model of the machine (method step c).

[0050] According to method step d, maintenance work is carried out on at least one faulty component of the machine. According to method step e, the position of the serviced component is recorded, and position data for the serviced component is generated in the machine coordinate system. For example, an external tracking unit, comprising at least one camera and a suitably designed processor, detects the position of a marker, for example a pen, with which a maintenance employee points to the faulty component, relative to the rail vehicle.

[0051] In process step f , the captured image data for the component to be serviced from the digital model of the machine are marked using the associated position data in the machine coordinate system .

Claims

Patent claims 1 . Method for training an image recognition algorithm for fault detection in the maintenance of machines with multiple components, comprising: a . capturing image data of the machine using at least one camera; b . creating a digital model of the machine based on the captured image data; c . creating a machine coordinate system for the digital model of the machine; d . carrying out maintenance work on at least one faulty component of the machine; e . capturing a position of the serviced component and generating position data for the serviced component in the machine coordinate system; f . marking the captured image data for the serviced component from the digital model of the machine based on the assigned position data in the machine coordinate system. 2 . Method according to claim 1, characterized in that in method step b . a three-dimensional digital model of the machine is created on the basis of the acquired image data. 3 . Method according to one of claims 1 or 2, characterized in that method step a . is carried out with a plurality of high-resolution cameras. 4 . Method according to one of claims 1 to 3 , characterized in that method step e . precedes : - Mark the faulty component using a marker.

5. A method according to any one of claims 1 to 4, characterized in that method step f follows: - Determining the serviced component in the marked image data for the serviced component using an image recognition algorithm.

6. A method according to any one of claims 1 to 5, characterized in that step d comprises: Identifying the maintained component and generating information about the identified component. 7 . Method according to claim 6 , characterized in that method step f . follows: - Linking the captured image data on the serviced component from the digital model of the machine with the information generated on the serviced component of the machine. 8 . Method for fault detection in the maintenance of machines with multiple components comprising an image recognition algorithm trained according to one of claims 1 to 7: - Capturing image data of the machine using at least one camera; - Detecting a faulty component of the machine based on the captured image data of the machine using the image recognition algorithm.

9. Computer program product comprising instructions which, when the program is executed by a suitable mobile terminal, cause the terminal to carry out the method according to one of claims 1 to 7.

10. A data carrier on which the computer program product according to claim 9 is stored. Device for carrying out the method according to the invention comprising: - At least one camera for capturing image data of the machine; - At least one memory for storing the image data in a database; - At least one processor for creating a digital model of the machine from the captured image data; - At least one processor for creating a machine coordinate system for the digital model of the machine; - At least one tracking unit comprising at least one camera for detecting a position of the serviced component and at least one processor for generating position data for the serviced component in the machine coordinate system; - At least one processor for marking the captured image data of the serviced component from the digital model of the machine based on the associated position data in the machine coordinate system.

Images