METHOD FOR THE COMPUTER-AIDED RECORDING AND EVALUATION OF A WORK PROCESS IN WHICH A HUMAN WORKER AND A ROBOTIC SYSTEM INTERACT

Description

[0001] The invention relates to a method and a device for the computer-aided recording and evaluation of a work process in which a human worker and a robotic system interact.

[0002] It is known from the prior art to record the movements of a human worker and a robotic system within a work process in which they interact. For example, document WO 2015 / 114089 A1 discloses a safety device for a worker in which a mobile detection device, worn by the worker, locates the worker within the working area of ​​a moving automatic machine, and a warning signal is transmitted to the worker in the event of a collision or risk of collision with the automatic machine. To date, there are no approaches in the prior art that subject the movements of a worker and a robotic system, recorded by such detection devices, to computer-aided analysis in order to determine the work activities performed by the worker and the robotic system, respectively.

[0003] Publication EP 2 772 811 A2 discloses a method according to the preamble of claim 1.

[0004] The object of the invention is to create a method and a device for the computer-aided recording and evaluation of a work process, in which the recorded movements of a worker and a robotic system are subjected to a comprehensive analysis for the extraction of work activities.

[0005] This problem is solved by the method according to claim 1 or the device according to claim 14. Further developments of the invention are defined in the dependent claims.

[0006] Within the framework of the method according to the invention, a work process in which a human worker and a robotic system interact is recorded and evaluated. The term "robotic system" is to be understood broadly and, depending on the configuration, can comprise one or more automatically moving machines (i.e., robots). Optionally, the robotic system can also consist of only a single robot.

[0007] According to the invention, digital motion data of the worker are determined based on an acquisition using a first sensor system. This data contains a multitude of position data sets for a multitude of successive time points, wherein each first position data set contains position values ​​describing the position and orientation of body parts of the worker in a fixed coordinate system at the respective time. In contrast to the first position data sets, the second position data sets mentioned below do not refer to the position and orientation of body parts of the worker, but rather to the position and orientation of moving components of the robotic system. The terms "first" and "second" position data sets are to be interpreted broadly. It is only necessary to ensure that the position values ​​of the position data sets allow the position and orientation of body parts of the worker or the robotic system to be determined.of moving components of the robotic system in a fixed coordinate system. Preferably, the position values ​​of the first and second position data sets describe the position and orientation in three-dimensional space. A fixed coordinate system is understood to be a global coordinate system that does not follow the movement of either the worker or the robotic system. Preferably, the fixed coordinate system and the local coordinate systems mentioned below are three-dimensional coordinate systems.

[0008] In the method according to the invention, digital motion data of the robotic system are further determined based on an acquisition by means of a second sensor and / or based on kinematic control data of the robotic system, which contain a plurality of second position data sets for a plurality of successive time points, wherein each second position data set contains position values ​​to describe the position and orientation of moving components of the robotic system in the stationary coordinate system at the respective time point.

[0009] The movement data of the worker or the robotic system described above represent preprocessed raw data originating from the first or second sensor or the robot's control system. Depending on the configuration, the preprocessing can include various steps besides the actual determination of the position data sets, such as normalization of sensor data, conversion into standardized units, transformations, and the like. Kinematic control data of the robotic system refers to control data for the robotic system that determines its movement. There are known methods that determine the aforementioned movement data of the robotic system from such control data using the CAD or kinematic model of the robotic system. One such known method is used within the framework of the method according to the invention.

[0010] In a further step of the method according to the invention, motion patterns are determined from the digital motion data of the worker and the robotic system. These motion patterns are contained in the digital motion data and comprise first and second motion patterns. The first motion patterns are each assigned to a movement of the worker from a plurality of predefined movements of the worker. In contrast, the second motion patterns are each assigned to a movement of the robotic system from a plurality of predefined movements of the robotic system. The extraction of the motion patterns is based on pattern recognition using previously known motion patterns. These previously known motion patterns are, for example, stored in a database. Any pattern recognition methods known from the prior art can be used to determine the first and second motion patterns.

[0011] After the movement patterns have been determined, they are fed as input data into a data-driven model that has been trained using machine learning based on training data. The data-driven model then determines activities as output data, comprising first and second activities. The first activities form classes, each specifying one or more first movement patterns as a worker activity from a plurality of predefined worker activities. In other words, a first activity corresponds to a worker activity. Similarly, the second activities form classes, each specifying one or more second movement patterns as a worker activity from a plurality of predefined worker activities. In other words, a second activity corresponds to a worker activity.According to the invention, the original movement patterns are thus assigned to corresponding types or classes of work activities by using a data-driven model. The work activities therefore represent a classification of the movements of the worker or the robotic system that is superior to the movement patterns.

[0012] In the method according to the invention, output is generated via a user interface based on the activities performed, in order to assist the worker. For example, the worker can receive qualitative feedback via the user interface as to whether a work activity has been carried out correctly.

[0013] Depending on the specific configuration of the method according to the invention, the movement patterns can be defined in various ways. A movement pattern can, for example, refer to the movement of a worker's finger, the raising of a worker's arm, the worker's bending forward, a specific robot movement, and the like. In contrast, activities abstract specific movement patterns into work tasks. Different movement patterns can potentially lead to the same activities. An activity can, for example, define a predetermined assembly step, a predetermined transport step, or any other work step.

[0014] In a preferred embodiment of the method according to the invention, segments comprising first and second segments are extracted from the digital motion data of the worker and the robotic system during the determination of the motion patterns. The first segments each comprise first position data records for several consecutive time points, and the second segments each comprise second position data records for several consecutive time points. After the segments have been extracted, the motion patterns are extracted from them, wherein the first motion patterns each comprise one or more temporally consecutive first segments, and the second motion patterns each comprise one or more temporally consecutive second segments.

[0015] Depending on the design, segments can be determined in various ways. In a simple version, the segments are consecutive first or second position data records within a fixed time window. If necessary, an association analysis of the movement data can be performed during segmentation. Implementing such an association analysis falls within the scope of professional expertise. Preprocessing via segmentation allows for simple and efficient implementation of subsequent pattern recognition.

[0016] In a further preferred implementation, the activities are subjected to process mining, which yields information about one or more work processes represented by the activities and contained within the workflow. This marks the first time that process mining methods are being used in the context of human-robot collaboration. Process mining methods are already established in the field, and their implementation in human-robot collaboration falls within the scope of expert practice. In addition to identifying work processes (process discovery), process mining can also be used to compare these processes and identify optimization opportunities.

[0017] In a preferred variant of the above embodiment, the output is generated via the user interface to assist the worker, based on the activities and information relating to the work process(s).

[0018] The data-driven model used in the inventive method for classifying movement patterns in activities can be designed in any way.

[0019] Preferably, the data-driven model is based on one or more neural networks, e.g., one or more multilayer and / or recurrent neural networks, or one or more Bayesian networks, or on support vector machines, or on decision trees. All these data-driven models are known per se and can be learned using suitable machine learning methods based on training data. Depending on the design, the learned data-driven model can be learned using supervised or unsupervised learning. In this case, the training data is designed such that it specifies the activity assigned to each group of one or more movement patterns.

[0020] In a preferred embodiment of the method according to the invention, the first sensor system comprises a plurality of body-worn sensors that are attached directly or indirectly (e.g., integrated into clothing) to the worker's body, preferably several fiber Bragg grating sensors and / or one or more position or orientation sensors and / or one or more acceleration sensors and / or one or more inertial sensors. These sensors are known from the prior art. For example, fiber Bragg grating sensors attached to a person's body are described in document US 2017 / 0354353 A1. Analogous to the first sensor system, the second sensor system (if present) can also be configured to comprise a plurality of robot-worn sensors attached to one or more movable components of the robotic device. These sensors can also be fiber Bragg grating sensors or position sensors or...They may be designed as acceleration sensors or inertial sensors.

[0021] In a preferred embodiment of the method according to the invention, local position values ​​in a local coordinate system assigned to a respective body part are determined by means of the first sensor system. The position values ​​of the first position data sets depend on the local position values ​​and a position value of a worker's reference point in the stationary coordinate system. The worker's reference point has a fixed positional relationship to the local coordinate systems.Alternatively or additionally, local position values ​​are determined for each moving component of the robotic system using the second sensor and / or from the kinematic control data. These values ​​are determined within a local coordinate system assigned to the respective moving component. The position values ​​of the second position data sets depend on the local position values ​​and a position value of a reference point of the robotic system in the stationary coordinate system. The reference point of the robotic system has a fixed positional relationship to the local coordinate systems.

[0022] Depending on the configuration of the variant just described, the worker's or the robotic system's reference point can be fixed or movable. In the case of a movable reference point, the first sensor system preferably includes a localization sensor that detects the (current) position of the worker's reference point in the fixed coordinate system using wirelessly transmitted signals. Alternatively or additionally, the second sensor system can also include a localization sensor that detects the (current) position of the robotic system's reference point in the fixed coordinate system using wirelessly transmitted signals.

[0023] For the wireless-based localization described above, established technologies can be used, such as those based on time-of-flight or field strength measurements. For example, the localization sensor of the first sensor and / or the localization sensor of the second sensor can be based on localization using ultrasonic signals and / or electromagnetic high-frequency signals, preferably via Bluetooth and / or UWB (Ultra-Wideband). Alternatively, any other short-range radio-based system can be used as the localization sensor. Localization can also be performed using one or more laser scanners.

[0024] In a further preferred embodiment of the method according to the invention, at least part and preferably the entire first sensor system is designed redundantly, so that one or more sets of redundant measured values ​​are obtained by the first sensor system, wherein the redundant measured values ​​of a respective set were recorded independently of each other and, in the event that the redundant measured values ​​of a respective set deviate from each other by a predetermined amount, one or more predetermined actions are executed automatically.Alternatively or additionally, at least part of the second set of sensors, and preferably the entire second set, can be designed redundantly, so that the second set of sensors provides one or more sets of redundant measurements, wherein the redundant measurements of each set are acquired independently of one another, and if the redundant measurements of a respective set deviate from each other by a predetermined amount, one or more predefined actions are automatically executed. Such a redundantly designed system enables functionally safe human-robot collaboration.

[0025] The measured values ​​from the first and second sensors described above can directly relate to the raw measurement data acquired or to measured values ​​pre-processed within the procedure. Depending on the configuration, the predefined action(s) can be defined differently. In one variant, the predefined actions include output via a user interface to inform the user that measurement errors have occurred. Similarly, a predefined action could consist of stopping the robotic system to avoid collisions.

[0026] In a further embodiment of the method according to the invention, it is monitored whether measured values ​​from the first sensor and / or the second sensor are transmitted at regular intervals via a data transmission link, which can be wired or wireless. If the regular intervals are not maintained, one or more predefined actions are executed automatically. The actions can be configured as desired and, in particular, can again include stopping the robotic system or issuing a warning message via a user interface. The case of non-compliance with the regular intervals can, for example, be defined by a predefined threshold value, whereby if a time interval deviates from a predefined regular interval by more than the predefined threshold value, the predefined action(s) are executed automatically.

[0027] In a further embodiment of the method according to the invention, the minimum distance between the worker and the robotic system is calculated from the digital motion data of the worker and the digital motion data of the robotic system, and the robotic system is controlled depending on this calculated minimum distance. Preferably, the speed of the robotic system decreases as the minimum distance decreases. It is also possible for the robotic system to be stopped if the minimum distance falls below a predetermined threshold. In this way, collisions between the worker and the robotic system can be prevented.

[0028] In a variant of the embodiment described above, it is further deduced, based on the detection by the second sensor and / or from the kinematic control data of the robotic system and / or based on detection by a further sensor, whether one or more workpieces with predetermined dimensions are being held by the robotic system at any given time. The predetermined dimensions of the held workpieces are taken into account when calculating the minimum distance by treating the held workpiece(s) as part of the robotic system. This further reduces the risk of collisions. To deduce whether one or more workpieces with predetermined dimensions are being held by the robotic system, the further sensor can, for example, be used.Force transducers / 6-axis force-torque sensors are used to identify which workpiece has been picked up based on its weight, center of gravity, and moment of inertia, and then to use the correct CAD model of the corresponding workpiece for distance calculation.

[0029] In addition to the method described above, the invention relates to a device for the computer-aided recording and evaluation of a work process in which a human worker and a robotic system interact. The device comprises a computer system for the computer-aided execution of the method according to the invention or one or more preferred variants thereof. The computer system may optionally be distributed across several different computer units. Furthermore, the first sensor system and also the second sensor system (if present) are components of the computer system. In addition, the computer system includes means for receiving kinematic control data from the robotic system's control system, provided that kinematic control data are processed in determining the digital motion data of the robotic system.

[0030] An embodiment of the invention is described in detail below with reference to the accompanying figures.

[0031] They show: Fig. 1 a schematic representation of a system with which an embodiment of the method according to the invention is carried out; Fig. 2 a detailed view of the body-near sensor system made of Fig. 1 ; Fig. 3 a detailed view of the localization system from Fig. 1 ; and Fig. 4 a flowchart showing the process of processing the movement data of the worker and the robot from Fig. 1 clarifies.

[0032] The system described below of Fig. 1 The system serves to capture the movements of a human worker 1 and a robotic device in the form of a robot 3, whereby the worker and the robot interact with each other. Based on the captured movements, the robot 3 is appropriately controlled by a robot controller 9. Furthermore, computer-aided activities are derived from the movements of the worker and the robot, and from these, work processes are derived using process mining. This information can support the worker 1 in carrying out their tasks.

[0033] Worker 1 is in Fig. 1 schematically represented by dashed lines. One arm of the worker is designated with reference symbol 101, and each of the worker's legs is designated with reference symbol 102. Furthermore, the worker's torso or back is designated with reference symbol 103. The task of locating the worker and his body parts 101 to 103 is achieved, firstly, by means of body-worn sensors in the form of fiber Bragg grating sensors 2, and secondly, by means of a localization system comprising a base station system 4 and a transponder device 5, which are described in more detail below. Fig. 3 This is evident. The transponder device 5 is worn by worker 1.

[0034] First, the body-worn sensor technology based on fiber Bragg grating sensors 2 is described. The operating principle of this sensor technology is known per se. Fig. 2 The figure shows the sensor setup in greater detail. In this figure, the worker is again designated by reference numeral 1, his arms by reference numeral 101, his legs by reference numeral 102, and his back / torso by reference numeral 103. A fiber Bragg grating sensor 2 extends along each of these body parts. This sensor comprises optical fibers 201 and an interrogator 202, which is located at one end of each fiber. The interrogator 202 transmits light with a predefined frequency spectrum into the corresponding fiber using a light source and simultaneously records the frequency spectrum of the light reflected by the fiber using a fiber optic spectrometer. The fibers can be fixed at specific points, for example with Velcro straps, or permanently integrated into the worker's work clothing (vest, jacket, trousers, overalls, gloves). If necessary, several fibers can be provided for each body part.

[0035] In the embodiment described here, the individual optical fibers 2 each comprise single-core fibers in which several fiber Bragg gratings (e.g., FGB triplets) are inserted slightly offset or parallel at the same height within a single fiber core to measure multidimensional bending. A fiber Bragg grating is characterized by a local periodic variation in the refractive index within the corresponding fiber core. The fiber Bragg gratings are positioned at corresponding bending or flexion points along body parts such as the shoulder joint, elbow, wrist, spine, hip, and knee. With corresponding bending or flexion movements, the reflection spectrum recorded by the fiber optic spectrometer of the interrogator 202 changes. By using multiple fiber Bragg gratings at the respective bending or flexion points, the reflection spectrum can be measured in greater detail.This allows for the real-time determination of three-dimensional bending, torsional loads, and movement of human body parts. Instead of using a single-core fiber, it may also be possible to use a multi-core fiber in which the fiber Bragg lattices are separately embedded in different fiber cores.

[0036] Each fiber optic interrogator 202 represents an evaluation unit that regularly (every few milliseconds) provides data on the 3D deformations of the fibers. This evaluation unit (also called a mobile fiber optic measuring device) is essentially a miniaturized spectrometer connected to the sensor fiber. The spectrometer acquires emission and absorption spectra, as well as information on frequency-dependent reflection. These are then converted into a 3D computer model using shape reconstruction software. For this purpose, the bending radii and directions for the individual measurement positions (i.e., the positions of the fiber Bragg gratings) are first calculated based on the known positions of the fiber Bragg gratings or FBG triplets within the fibers and the (relative) amplitudes of the spectral ranges addressed by each fiber Bragg grating.Using positions, bending radii, and directions, the Cartesian coordinates for each measurement point can then be calculated (and interpolated). This data enables a near real-time digital 3D representation of the sensor fibers. Since the sensor fibers are attached to the worker's body or work clothing, the 3D representation of the sensor fiber corresponds to the movement / posture of the respective limb. Thus, an abstract digital 3D model of the limb can be generated from the data. Consequently, the task of the fiber optic sensors is to determine the orientation and position of the sensor fiber in real time relative to its origin.

[0037] The 3D position data of each sensor fiber refers to a local coordinate system LK, which is located at the origin of the fiber at the location of the spectrometer. Fig. 2 Two such local coordinate systems for the fiber Bragg grating sensors of the legs 102 are shown as examples. The local coordinate systems LK also have a fixed positional relationship to a reference point RP of the worker, at which the worker's coordinate system WK is located. The data on the 3D deformations of the sensor fibers always refer to the corresponding local coordinate system. By relating the origins of the sensor fibers to each other, the local coordinate systems can be integrated into the (common) coordinate system WK of the worker. For this, the distances between different origins and endpoints of the sensor fibers must be defined. By integrating the local coordinate systems into the common coordinate system, the digital 3D models of the body parts can be combined to form a digital 3D model of the human.The digital 3D model of the human body allows for real-time analysis of the worker's movement and current posture. To increase reliability, the fibers of the fiber optic sensor system or the wearable sensors can be redundantly installed in all parts of the body.

[0038] The 3D information from the spectrometers of each fiber Bragg grating sensor is combined in a body-worn computing and communication unit located in Fig. 1 Designated with reference number 6, this device is worn by the operator. This computing and communication unit essentially consists of a safety microcontroller, as described in more detail below. The spectrometer information is transmitted to the computing and communication unit via cable or, if necessary, wirelessly. The unit has an interface (e.g., an add-on board for the safety microcontroller).

[0039] In addition to the fiber Bragg grating sensors, further sensors, such as orientation, acceleration, or inertial sensors, can be installed at selected points on the worker's body or clothing. The data from these sensors can also be combined in the interface unit mentioned above. Inertial sensors are a combination of accelerometer, gyroscope, and magnetometer. By using different sensor types and data, errors due to measurement tolerances can be corrected and compensated for, thus increasing accuracy, reliability, and operational safety. For example, the bending angle of an arm can be measured with a fiber Bragg grating sensor, and the arm's orientation can be measured using an orientation sensor attached to the wrist. Both sensors should provide consistent readings.By considering both measurements, the measured arm position can be confirmed or corrected if necessary.

[0040] The computing and communication unit essentially consists of a safe microcontroller with redundant processors that automatically compares the calculations performed by these processors. This functionally safe microcontroller is used for safe sensor data processing. Functionally safe microcontrollers can be based on either a discrete or multicore architecture. In a discrete architecture, two discrete microcontrollers are connected: one microcontroller executes a control algorithm, while the other executes a monitoring algorithm. The monitoring algorithm verifies the plausibility of the control algorithm. A functionally safe microcontroller is always used when robotics applications, according to the ISO 13849 standard, require a high performance level (at least Performance Level D Category 3 or SIL2) and fault tolerance.

[0041] The wearable sensor system, consisting of the fiber Bragg grating sensors described above and optionally additional orientation, acceleration, or inertial sensors, preferably also adheres to the functional safety requirements for robotics applications according to ISO 13849. This is achieved during motion detection (orientation) through redundant measurement, meaning that at least two measurement points are installed at each measurement position. Preferably, two fiber optic strands run parallel to each other and are evaluated by separate spectrometers. The measurement results are then compared by the safety-compliant microcontroller described above (discrete or multicore architecture, Category 3 according to ISO 13849). It must be ensured, however, that the redundant channels cannot interfere with each other. Both channels are monitored continuously and simultaneously.If the measurement results of the two channels differ, the microcontroller detects an error, which is then transmitted to the computer 8 described below, which in turn causes the robot 3 to be brought to a safe stop by means of the robot control 9.

[0042] The wearable computing and communication unit 6 is powered by batteries. In addition to a wired interface, such as USB and LAN, the unit has a wireless interface for communicating the sensor data recorded by the wearable sensors to an external receiver unit 7. Fig. 1 The data transmitted to receiver unit 7 are designated POS. This data represents the position data of the sensor fibers of the fiber Bragg grating sensors in their respective local coordinate systems (LC). The POS position data is transmitted to receiver unit 7 at regular intervals (e.g., every 25 ms) to update body postures. The receiver unit also performs a watchdog function. According to this function, the receiver unit receives incoming data packets and expects a set or group of data records at a predetermined frequency from the body-worn sensors. If no set of data records arrives within the expected time, the data records were either not sent or they were lost on the wireless transmission path between the processing and communication unit 6 and receiver unit 7.Based on the watchdog function of the receiver unit 7, an error is triggered in such a case, which causes the computer 8 described below to stop the robot 3.

[0043] The receiver unit 7 and the wearable computing and communication unit 6 can, for example, be designed as PROFINET / EtherCAT-compatible components and may optionally use the PROFIsafe / Safety-over-EtherCAT protocol. The receiver unit 7 can be connected to the computer 8 via an Ethernet cable. In this case, the computer is an industrial PC or a software or hardware PLC (PLC = Programmable Logic Controller).

[0044] In order to relate the local position data POS to the fixed coordinate system KS, the movement of worker 1, i.e., from their reference position RP, must also be recorded. For this purpose, in the embodiment of the Fig. 1 a localization system is used, comprising a base station system 4 and a transponder device 5, and is described in detail in Fig. 3 As shown, in the embodiment depicted there, two different technologies are used for localization in order to redundantly determine the reference position RP of worker 1.

[0045] The localization system of Fig. 3 The system comprises a first and a second localization system. The first localization system wirelessly transmits signals via four base stations 401, with the signals being wirelessly received by a pair of transponders 501 of the transponder unit 5. As a minimum requirement, the first localization system comprises three base stations and one transponder, preferably with a larger number of base stations and transponders. Upon receiving corresponding signals, each transponder 501 transmits a response signal, which is received again by the base stations 401. By measuring the travel time of the respective signals at the individual base stations 401, the distance to the respective transponders 501 can then be determined, and the position of the transponders relative to the fixed coordinate system KS can be determined by triangulation (see [reference]). Fig. 1 The two 501 transponders are in a fixed positional relationship to the reference position RP, allowing the reference position to be determined separately for each transponder. The final reference position can be obtained, for example, by averaging the reference positions of all 501 transponders, provided the two reference positions do not deviate too much from each other. If a significant deviation is detected, a localization error may be identified, and the corresponding reference position value must be discarded. By using two or, if necessary, more transponders for the localization system, the reliability of the localization results can be improved, and the corresponding reference position of the person can be confirmed or corrected.

[0046] The first localization system based on the 401 base stations and 501 transponders is a so-called ultrasonic beacon-based localization system, in which ultrasonic pulses are transmitted and received wirelessly by the 401 base stations and 501 transponders. However, another technology can also be used for this localization system, for example, based on Bluetooth. In this case, the wireless signals are high-frequency signals in the Bluetooth frequency range.

[0047] In addition to the first localization system based on 401 base stations and 501 transponders, the localization system of Fig. 3 A second localization system, analogous to the first localization system, comprises four base stations 402 and a corresponding pair of transponders 502. The transponders are again attached to the human worker 1 and are part of the transponder device 5. As a minimum requirement, the second localization system contains three base stations and one transponder, with a larger number of base stations and transponders being preferred.

[0048] The second localization system uses a different localization technology. In the embodiment described here, a UWB-based localization system is used, in which signals are transmitted and received by the base stations or UWB anchors 402 and the transponders 502 using ultra-wideband transmission technology. The localization process of this second localization system corresponds to that of the first localization system described above. That is, localization is achieved by measuring the time of flight of the signals from the respective base stations to the transponder and from the transponder back to the base stations, with triangulation based on this measurement. Alternatively, localization can also be achieved by measuring the input angles of the signals at the base stations.

[0049] By using an ultrasonic beacon-based localization system as the primary method, a worker can be located within buildings to within centimeters (approximately 2 cm). For precise localization in three-dimensional space, at least three base stations, also known as short-range radio beacons, are required. These short-range radio beacons are distributed around the worker's work area. A UWB-based localization system also enables the determination of a worker's position within buildings to within centimeters. The advantages of using UWB transmission technology lie in its exceptionally high precision.

[0050] In the embodiment described here, the reference point RP of worker 1, and thus the location of the coordinate system WK, is calculated by both the first and the second localization systems. This calculation is performed, for example, in a predefined base station of the base stations of the first and second localization systems, whereby the travel times from other base stations to the predefined base station are transmitted, which then determines the reference position via triangulation.

[0051] The localization data LD in the form of the reference positions of the first and second localization systems are transmitted to the computer 8 via a wired connection or, if necessary, wirelessly, as described in the embodiment shown here. Fig. 1 As indicated, computer 8 continuously compares the localization results of the first and second localization systems. This allows measurement errors to be identified and the determined reference position of the person to be confirmed or corrected. Furthermore, if deviations are too large, computer 8 can issue a command to robot controller 9 to stop robot 3. In addition, the measurement data from the respective localization systems, which can sometimes lead to erratic positions of the worker, can be cleaned using filter algorithms, such as a Kalman filter, thereby improving the quality of the results.

[0052] In a modified embodiment of the in Fig. 3 In the localization system shown, the localization can also be configured such that the reference position for the first and second localization systems is determined by one of the transponders 501 and 502, respectively, which the worker 1 carries. In this case, the reference position is transmitted from the corresponding transponder to the processing and communication unit 6. The time-of-flight measurement is performed by the respective transponder, measuring the travel time of wireless signals from the respective transponders to the respective base stations and then back to the transponders. The reference positions determined by the transponders are wirelessly transmitted by the processing and communication unit 6 to the receiving unit 7.

[0053] As mentioned above, the computing and communication unit 6 is designed as a functionally safe microcontroller. This microcontroller can therefore not only compare the sensor data from the fiber Bragg grating sensors but also compare the separately determined reference positions of the two localization systems. Wireless transmission of the corresponding reference positions to the receiver unit 7 only occurs if the results of both localization systems show sufficient agreement, whereby sufficient agreement can be defined based on a threshold criterion. If this threshold criterion is not met, an error code is transmitted to the receiver unit 7.

[0054] Using the local position values ​​POS of the fiber Bragg grating sensors and the localization data LD relating to the reference position RP of worker 1, the computer determines 8 motion data BD, which specify the corresponding position values ​​(i.e., the position and orientation) of worker 1's body parts in the global coordinate system KS at successive times. The motion data BD correspond to a digital human model of worker 1 and describe his movements in space.

[0055] Furthermore, in the embodiment of the Fig. 1 The robot controller 9 transmits kinematic control data KD to the computer 8. This kinematic control data is known per se and serves to control the movements of the individual movable components 301 to 304 of the robot 3 relative to the fixed reference position RP' assigned to the robot 3. The movable components of the robot 3 are, in the embodiment of the Fig. 1 The three interconnected robot arms 301, 302, and 303, as well as the gripper 304 attached to robot arm 303, are used. From the kinematic control data KD, the computer 8 determines, in a known manner and by accessing the kinematic model of the robot 3, corresponding motion data BD' (i.e., temporally successive position values ​​in the form of position and orientation) of the robot's movable components with respect to the global coordinate system KS. The motion data BD' correspond to a digital robot model of the robot 3 and describe its movements in space.

[0056] In the embodiment described here, the minimum distance d min between worker 1 and robot 3 is determined using the motion data BD and BD'. Depending on this minimum distance, computer 8 then sends control data CD to the robot controller 9 to influence the robot's movement. The smaller the minimum distance d min between worker 1 and robot 3, the lower the robot's speed. This reduces the risk of collisions between the robot and the worker. Furthermore, if the minimum distance d min falls below a certain threshold, computer 8 can send a control command to the robot controller 9, causing the robot to stop to prevent collisions.

[0057] It is also possible that further assistance functions can be provided using the digital human model based on the motion data BD, or using the digital robot model based on the motion data BD'. For example, the digital human model can be used to identify unergonomic working postures of the worker. The worker can then be informed via a suitable information system about how to perform their work more ergonomically in the future. It is also conceivable that, after an unergonomic posture of the worker is detected, the robot is controlled in such a way that the working height at which the worker works, or the height of a workpiece presented to them, is adjusted to the worker's height, thus enabling an ergonomic posture.

[0058] In another embodiment, the calculation of the minimum distance between worker and robot also takes into account whether the robot is currently holding a workpiece on its gripper 303. This information can also be obtained from the robot's kinematic control data. In this case, the kinematic control data contains the CAD data of the workpiece. If it turns out that the robot is holding a workpiece on its gripper, this is factored into the calculation of the minimum distance between human and robot in such a way that a collision between the worker and the held workpiece is also avoided. In other words, in this case, the workpiece is treated as if it were part of the robotic system.

[0059] The preceding section described an embodiment in which the robot's motion data is derived from its kinematic control data. Alternatively or additionally, however, it is also possible to transfer the concept of the worker's close-coupled sensors to the robot. In this case, analogous to the worker, corresponding fiber Bragg grating sensors are attached to the robot's moving components across its joints, for example, using straps, Velcro, and the like. In other words, the robot is equipped with close-coupled sensors that... Fig. 1 The sensor fibers of the fiber-Bragg grating sensors then output the relative deformation with respect to their local coordinate origin. Consequently, the sensor fibers, each covering the robot's joints, must be related to each other and to a common coordinate system. This is determined by fixed static distances between the start and end points or a defined starting point for the sensor fiber on the robot.

[0060] Just as with the body-worn sensors, the data from the fiber Bragg gratings are again aggregated in an interface unit and then forwarded to a computing and communication unit. This unit, analogous to the body-worn sensors, can in turn contain a secure microcontroller with redundant processes. The microcontroller forwards the data from the robot-worn sensors to computer 8, which then determines the digital robot model in the form of motion data BD'.

[0061] In the embodiment of the Fig. 1 The reference position RP' of robot 3 is fixed and known in advance, so a localization system for determining this reference position is unnecessary. However, if the robot as a whole is movable, the (current) reference position RP' can also be determined using a localization system, analogous to the reference position RP of the worker, and then combined with the local positions of the fiber Bragg grating sensors to obtain position data of the robot's moving components relative to the fixed coordinate system KS. In a further embodiment, the local positions can also be determined by transponders attached to the robot, as described above.

[0062] Depending on the design, the sensor fibers of the fiber Bragg grating sensors can be attached to the robot in various ways, e.g., straight, spiral, and the like. Furthermore, the reliability of the determined position data can be further improved by using additional sensors such as orientation sensors, accelerometers, inertial sensors, and the like, as can also be achieved with the body-worn sensor technology described above.

[0063] The based on Fig. 1 The components described above, i.e., all components except those described below (10 and 11), constitute a so-called safety cycle for the entire system consisting of the worker and the robot. This safety cycle includes all safety-relevant components of the system. These include the wearer-worn sensors (2) in combination with the localization system (4, 5) for detecting the worker's movement, the kinematic control data (KD) from the robot controller (9) or, optionally, from sensors located near the robot, as well as the computing and communication unit (6), the receiver (7), the computer (8), and the robot controller (9). These components ensure that movement data is reliably captured for both the worker and the robot and transmitted to the computer (8).

[0064] This is where the safety-relevant part of the data fusion takes place; that is, a combined human-machine model is created using the data, and safety-relevant distances between the human and the robotic system are calculated. Based on this calculation, the robot's distance-dependent speed is then controlled by transmitting control data or control commands CD from computer 8 to the robot controller 9.

[0065] In addition to this security cycle, the system also includes... Fig. 1 A further non-secure value-added cycle, which is explained below and represents an essential aspect of the invention, is implemented. To realize this value-added cycle, the motion data BD and BD' are transmitted via a suitable interface to a further computer system 10. Preferably, this data is transmitted via the internet, and the computer system 10 is an internet server. Communication between the computer 8 or a corresponding interface with the internet server can, for example, be achieved using Representational State Transfer Services (REST). The internet server 10 can optionally also be a cloud-based computing and storage system. The motion data BD and BD' are then further processed on the server 10.

[0066] The further processing of the movement data BD and BD' by the internet server 10 is in Fig. 4 The motion data BD is a temporal sequence of position data of the respective body parts of worker 1 with respect to the coordinate system KS, whereas the motion data BD' represents a temporal sequence of position data of the respective moving components of robot 3 with respect to the coordinate system KS. As can be seen from Fig. 4 This results in the following: from the movement data BD and BD', (temporal) segments SE, comprising first segments SE1 and second segments SE2, are first extracted in a single step S1. The first segments SE1 originate from the temporal stream of movement data BD, whereas the second segments SE2 are derived from the temporal stream of movement data BD'.

[0067] The segmentation of the movement data BD and BD' into corresponding segments can be carried out using established methods. In the simplest case, a predetermined time window is used, and all position data within this window are successively assigned to a segment. Alternatively, segmentation can also be performed using an established association analysis, which identifies correlations in the position data. Implementing an association analysis to determine segments SE1 and SE2 falls within the scope of professional practice.

[0068] After the first segments SE1 and second segments SE2 have been extracted from the motion data BD and BD', pattern recognition is performed in the next step S2 for both the first segments SE1 and the second segments SE2. This yields motion patterns BM comprising the first motion patterns BM1 and the second motion patterns BM2. The first motion patterns BM1 are extracted from the first segments SE1 relating to the worker's movement, while the second motion patterns BM2 are derived from the second segments SE2 relating to the robot's movement. The motion patterns are determined using a database DB containing a large number of predefined motion patterns BM'. Each predefined motion pattern comprises one or more temporally sequential segments and a specification of the movement assigned to the motion pattern, i.e., a specific type of movement, such as...Movement of the worker's finger, raising of the arm, bending forward, movement of the robot's gripper, and the like. The database DB contains movement patterns for both worker and robot movements.

[0069] In step S2, a known pattern recognition method is used to obtain the first and second movement patterns BM1 and BM2 contained in the consecutive segments SE1 and SE2, respectively, by comparing them with the predefined movement patterns BM' in the database DB. Implementing a suitable pattern recognition method falls within the scope of expert practice and can, for example, be based on cluster analysis. Preferably, the recognized patterns BM1 and BM2 are stored as newly added patterns in the database DB, thus increasing the number of patterns BM' in the database and continuously improving pattern recognition.

[0070] In a subsequent step S3, the respective movement patterns BM1 and BM2 are fed into a data-driven model MO, which has been previously trained using suitable training data TD. In the embodiment described here, the data-driven model MO consists of a first model, which receives a predefined number of temporally sequential first movement patterns BM1 as input data, and a second model, which receives a predefined number of temporally sequential second movement patterns BM2 as input data. In other words, a time window of a predefined size (so-called sliding window) runs through the temporal sequence of the first and second movement patterns, with the first and second movement patterns contained within the respective position of the time window forming the input data of the corresponding data-driven model.

[0071] The output of the respective data-driven models is activity AK, where first activities AK1 correspond to movement patterns BM1 and second activities AK2 correspond to movement patterns BM2. These activities thus classify the movement patterns. The training data for each model consisted of known combinations of consecutive first and second movement patterns within a specific time window, each with a known associated activity. The activities represent the respective tasks of worker 1 (activity AK1) and robot 3 (activity AK2) within the context of the workflow under consideration (e.g., gripping a screw, moving a workpiece, etc.).

[0072] In the variant described here, the data-driven model MO is based on a neural network structure that has been learned in a known manner using machine learning based on the training data TD. The neural network structure contains one or more neural networks of artificial neurons with a large number of hidden layers, which are learned via deep learning. Alternatively, other data-driven models in the form of appropriate classification or regression procedures can be used in step S3. These models are also learned in a suitable manner using machine learning based on training data.

[0073] After identifying activities AK1 and AK2, these can be saved and, if required, output via a user interface 11. The user interface is located in Fig. 1 This is exemplified by a visual user interface in the form of a screen. The user interface is visible to the worker, allowing them to obtain information about their own activities and those of the robot.

[0074] In the embodiment of the Fig. 4The activities AK are further processed in step S4. In this step, process mining (PM) is performed. Process mining itself is a well-known technique in the field of process management that supports the analysis of business processes based on event logs. This process mining is now applied to the activities AK, and its implementation for such activities falls within the scope of professional practice. For example, the open-source framework "ProM" can be used to implement the process mining. Process mining uses techniques such as trend detection, pattern analysis, and performance analysis to reconstruct and improve business processes by analyzing the generated data. This approach makes it possible to improve and automate processes from the bottom up, even if the underlying process model is unknown or highly variable.The result is information on PR work processes, which are represented by the activities AK. In this sense, one or more sequential activities are classified as work processes.

[0075] In a concrete implementation of process mining (PM), the activities (AK) and one or more predefined reference work processes are fed into the mining process as input data. The reference work processes are stored as models in a database. A reference work process could, for example, be an assembly process within the context of human-robot collaboration. Information about the work processes is then extracted using established data mining techniques. Process identification, process comparison, and process optimization can be performed in a familiar manner. For process identification, sequence and association pattern analysis can be used, whereas for process comparison, pattern matching is the appropriate method. In pattern matching, the activities of the current process are compared with the activities of one or more reference work processes.The process mining process yields one or more identified work processes, the results of comparing these processes with reference work processes, and optimization potential for the identified process. This information is stored and can be output to the operator via user interface 11 as needed or in real time.

[0076] The embodiments of the invention described above offer a number of advantages. In particular, robust fiber-optic sensor methods based on fiber Bragg grating sensors are combined with high-frequency communication technologies for localization. This minimizes interference, such as electromagnetic fields, daylight, frequency bands of existing IT networks, and the like. The combination of these two methods fulfills the technical industrial requirements for real-time capability, localization accuracy and measurement resolution, robustness and insensitivity to interference, system neutrality, and seamless integration into the work environment. Furthermore, the wearable sensor technology is designed to be considered a functionally safe system.This is achieved through the redundant design of individual components (at least two sensors at one measuring point) and through the use of functionally safe microcontrollers.

[0077] In one configuration, the robot is also equipped with a sensor system analogous to the body-worn sensors of the worker. This allows the robot's position to be determined at any given time. In this way, the robot's position data can be obtained regardless of the manufacturer, without access to a proprietary interface. Nevertheless, it may also be possible to read the position data directly from the robot's control system.

[0078] Furthermore, the invention enables the derivation of activities from the captured motion data of the worker and the robot through the use of data-driven models learned with appropriate machine learning methods. In addition, these activities can optionally be further processed using a known process mining method to obtain more detailed information about the worker's work processes in interaction with the robotic system.

Claims

1. A method for computer-assisted detection and evaluation of a workflow, in which a human worker (1) and a robotic system (3) interact, wherein: - based on a detection by means of a first sensor system (2, 4, 5), digital movement data (BD) of the worker (1) are determined, which contain a plurality of first position data sets for a plurality of successive time points, wherein a respective first position data set contains position values for describing the position and orientation of body parts (101, 102, 103) of the worker (1) in a stationary coordinate system (KS) at the respective time point; - based on a detection by means of a second sensor system (12) and / or based on kinematic control data (KD) of the robotic system (3), digital movement data (BD') of the robotic system (3) are determined, which contain a plurality of second position data sets for a plurality of successive time points, wherein a respective second position data set contains position values for describing the position and orientation of movable components (301, 302, 303, 304) of the robotic system (3) in the stationary coordinate system at the respective time point; - from the digital movement data (BD, BD') of the worker (1) and of the robotic system (3), movement patterns (BM) are determined, which are contained in the digital movement data (BD, BD') and which comprise first movement patterns (BM1) and second movement patterns (BM2), wherein the first movement patterns (BM1) are each assigned to a movement of the worker (1) from a plurality of predefined movements of the worker (1) and the second movement patterns (BM2) are each assigned to a movement of the robotic system (3) from a plurality of predefined movements of the robotic system (3), wherein the extraction of the movement patterns (BM) is based on a pattern recognition with access to pre-known movement patterns (BM'); characterized in that - the determined movement patterns (BM) are supplied as input data to a data-driven model (MO), which is trained via machine learning based on training data (TD), wherein the data-driven model (MO) determines as output data activities (AK), which contain first and second activities (AK1, AK2), wherein the first activities (AK1) form classes, which each specify one or more first movement patterns (BM) as a work activity of the worker (1) from a plurality of predefined work activities of the worker (1), and wherein the second activities (AK2) form classes, which each specify one or more second movement patterns (BM) as a work activity of the robotic system (3) from a plurality of predefined work activities of the robotic system (3); - based on the activities (AK), an output is generated via a user interface (11) for assisting the worker (1).

2. The method according to claim 1, characterized in that during the determination of the movement patterns (BM), from the digital movement data (BD, BD') of the worker (1) and of the robotic system (3), segments (SE) are initially extracted, which comprise first segments and second segments (SE1, SE2), wherein the first segments (SE1) each comprise first position data sets for a several successive time points and the second segments (SE2) each comprise second position data sets for several successive time points, wherein from the segments (SE) the movement patterns (BM) are subsequently extracted, wherein the first movement patterns (BM1) each comprise one or more first temporally successive segments (SE1) and the second movement patterns (BM2) each comprise one or more second temporally successive segments (SE2).

3. The method according to claim 1 or 2, characterized in that the activities (AK) are subjected to a process mining (PM), whereby information is obtained about one or more work processes (PR), which are represented by the activities (AK) and are contained in the workflow.

4. The method according to claim 3, characterized in that based on the activities (AK) and the information about the one or more work processes (PR), the output is generated via the user interface (11) for assisting the worker (1).

5. The method according to one of the preceding claims, characterized in that the data-driven model (MO) is based on one or more neural networks and / or on one or more Bayesian networks and / or on support vector machines and / or on decision trees.

6. The method according to one of the preceding claims, characterized in that the first sensor system (2, 4, 5) comprises a plurality of near-body sensors (2), which are attached to the body of the worker (1), preferably one or more fiber Bragg grating sensors and / or one or more position sensors and / or one or more acceleration sensors and / or one or more inertial sensors, and / or in that the second sensor system (12) comprises a plurality of near-robot sensors, which are attached to one or more movable components (301, 302, 303, 304) of the robotic device (3), preferably one or more fiber Bragg grating sensors and / or one or more position sensors and / or one or more acceleration sensors and / or one or more inertial sensors.

7. The method according to one of the preceding claims, characterized in that via the detection by means of the first sensor system (2, 4, 5) for a respective body part (101, 102, 103), local position values are determined in a local coordinate system (LK), which is assigned to the respective body part (101, 102, 103), wherein the position values of the first position data sets depend on the local position values and a position value of a reference point (RP) of the worker (1) in the stationary coordinate system (KS), wherein the reference point (RP) of the worker (1) has a fixed positional relationship to the local coordinate systems (LK), and / or in that via the detection by means of the second sensor system (12) and / or from the kinematic control data (KD) for a respective movable component (301, 302, 303, 304) of the robotic system (3), local position values are determined in a local coordinate system, which is assigned to the respective movable component (301, 302, 303, 304), wherein the position values of the second position data sets depend on the local position values and a position value of a reference point (RP') of the robotic system (3) in the stationary coordinate system (KS), wherein the reference point (RP') of the robotic system (3) has a fixed positional relationship to the local coordinate systems.

8. The method according to claim 7, characterized in that the first sensor system (2, 4, 5) comprises a localization sensor system (4, 5), with which the position value of the reference point (RP) of the worker (1) in the stationary coordinate system (KS) is detected with the aid of a localization based on wirelessly transmitted signals, and / or in that the second sensor system (12) comprises a localization sensor system, with which the position value of the reference point (RP') of the robotic system (3) in the stationary coordinate system (KS) is detected with the aid of a localization based on wirelessly transmitted signals.

9. The method according to claim 8, characterized in that the localization sensor system (4, 5) of the first sensor system (2, 4, 5) and / or the localization sensor system of the second sensor system (12) is based on a localization by means of ultrasound signals and / or by means of electromagnetic high-frequency signals, preferably via Bluetooth and / or UWB.

10. The method according to one of the preceding claims, characterized in that at least a part of the first sensor system (2, 4, 5) is designed redundantly, so that one or more sets of redundant measured values are obtained by the first sensor system (2, 4, 5), wherein the redundant measured values of a respective set were detected independently of one another and in the case that the redundant measured values of a respective set deviate from one another by a predetermined amount, one or more predefined actions are carried out automatically, and / or in that at least a part of the second sensor system (12) is designed redundantly, so that one or more sets of redundant measured values are obtained by the second sensor system (12), wherein the redundant measured values of a respective set were detected independently of one another and in the case that the redundant measured values of a respective set deviate from one another by a predetermined amount, one or more predefined actions are carried out automatically.

11. The method according to one of the preceding claims, characterized in that it is monitored whether measured values of the first sensor system (2, 4, 5) and / or of the second sensor system (12) are transmitted at regular time intervals via a data transmission link, wherein in the case that the regular time intervals are not complied with, one or more predefined actions are carried out automatically.

12. The method according to one of the preceding claims, characterized in that the minimum distance (dmin) between the worker (1) and the robotic system (3) is calculated from the digital movement data (BD) of the worker (1) and the digital movement data (BD') of the robotic system (3), and the robotic system (3) is controlled as a function of the calculated minimum distance (dmin).

13. The method according to claim 12, characterized in that based on the detection by means of the second sensor system (12) and / or from the kinematic control data (KD) of the robotic system (3) and / or based on a detection by means of a further sensor system, it is further derived whether one or more workpieces with predetermined dimensions are held by the robotic system (3) at the respective time point, wherein the predetermined dimensions of held workpieces are taken into account in the calculation of the minimum distance (dmin) by treating the held workpiece or workpieces as part of the robotic system (3).

14. An apparatus for the computer-assisted detection and evaluation of a workflow, in which a human worker (1) and a robotic system (3) interact, wherein the apparatus comprises a computer means, wherein the computer means is configured to carry out a method in which: - based on a detection by means of a first sensor system (2, 4, 5), digital movement data (BD) of the worker (1) are determined, which contain a plurality of first position data sets for a plurality of successive time points, wherein a respective first position data set contains position values for describing the position and orientation of body parts (101, 102, 103) of the worker (1) in a stationary coordinate system (KS) at the respective time point; - based on a detection by means of a second sensor system (12) and / or based on kinematic control data (KD) of the robotic system (3), digital movement data (BD') of the robotic system (3) are determined, which contain a plurality of second position data sets for a plurality of successive time points, wherein a respective second position data set contains position values for describing the position and orientation of movable components (301, 302, 303, 304) of the robotic system (3) in the stationary coordinate system (KS) at the respective time point; - from the digital movement data (BD, BD') of the worker (1) and of the robotic system (3), movement patterns (BM) are determined, which are contained in the digital movement data (BD, BD') and which comprise first movement patterns (BM1) and second movement patterns (BM2), wherein the first movement patterns (BM1) are each assigned to a movement of the worker (1) from a plurality of predefined movements of the worker (1) and the second movement patterns (BM2) are each assigned to a movement of the robotic system (3) from a plurality of predefined movements of the robotic system (3), wherein the extraction of the movement patterns (BM) is based on a pattern recognition with access to pre-known movement patterns (BM'); characterized in that - the determined movement patterns (BM) are supplied as input data to a data-driven model (MO), which is trained via machine learning based on training data (TD), wherein the data-driven model (MO) determines as output data activities (AK), which contain first and second activities (AK1, AK2), wherein the first activities (AK1) each classify one or more first movement patterns (BM) as a work activity of the worker (1) from a plurality of predefined work activities of the worker (1) and wherein the second activities (AK2) each classify one or more second movement patterns (BM) as a work activity of the robotic system (3) from a plurality of predefined work activities of the robotic system (3); - based on the activities (AK), an output is generated via a user interface (11) for assisting the worker (1).

15. The apparatus according to claim 14, characterized in that the apparatus is configured to carry out a method according to one of claims 2 to 13.