Computer-implemented processing monitoring method, apparatus, system, and recording medium

Abstract

Computer-implemented method for monitoring a process to be performed by a person, comprising: obtaining at least one image of the person performing the process; detecting a body posture of the person in the at least one image; detecting at least one object in the at least one image; returning monitoring information about the process based on at least one geometric relationship between the detected body posture and the detected at least one object.

Description

Technical Field

[0001] This disclosure relates to the field of motion detection and processing monitoring. Specifically, this disclosure relates to a computer-implemented method for monitoring processes to be performed by a human. The process may be an industrial process, such as manufacturing or repair. Background Technology

[0002] Recent research has shown that despite automation, most quality defects in industrial processes are related to human error. Human workers are easier to train and more flexible than robots, but they introduce variability into processes because their performance depends on factors that are not easily controlled, such as fatigue, age, physical or mental health.

[0003] In attempts to monitor actions performed by humans, end-to-end AI systems relying on action detection have been developed. However, because these systems need to implicitly understand the complex actions being monitored, they require a lot of training data. Furthermore, they are often considered black boxes due to the difficulty in understanding how they work, and are sometimes not well-received by humans. Therefore, there is room for improvement. Summary of the Invention

[0004] In this regard, this disclosure relates to a method for monitoring computer-implemented processes to be performed by humans, comprising:

[0005] Obtain at least one image of the person performing the processing;

[0006] Detect human pose in at least one image;

[0007] Detect at least one object in at least one image;

[0008] Based on at least one geometric relationship between the detected human pose and at least one detected object, monitoring information about the processing is returned.

[0009] Obtaining at least one image may include, for example, acquiring an image via an image acquisition module such as a camera, or retrieving an already acquired image from a database (e.g., a local or remote server). In the following text, unless otherwise stated, “image” refers to at least one image. More generally, in the following text, the article “the” may refer to “the at least one”.

[0010] Humans are human beings. Human pose detection (also known as human pose estimation) is known in the field of machine learning, for example. Human pose detection can use specialized artificial neural networks and can be configured to output at least one indicator of the position, size, and / or pose of at least one human being (preferably each human being) in an image.

[0011] Object detection itself is known, for example, in the field of machine learning. Detection of at least one object can be performed using a specialized artificial neural network (i.e., different from an artificial neural network that performs human pose detection), and can be configured to output at least one metric of the location, size, and / or type of at least one object in an image. The at least one object to be detected can be predetermined, for example, due to its importance in the process to be monitored.

[0012] Considering the above, it should be understood that human pose detection and object detection are performed separately, explicitly, and potentially independently of each other. "Explicitly" means that the detected human pose and the detected object are provided as explicit outputs of the respective detection steps. In contrast to end-to-end trained action detection systems that learn to detect actions without knowing precisely what is a person or an object in an image, or even whether a person exists in the image, the above method leverages the fact that, in the process to be monitored, the interactions between the person performing the process and important objects (potentially those they might interact with) are adequately archived. Therefore, the problem of monitoring the process can be simplified to identifying people, identifying objects important to the process, and determining at least one geometric relationship between the detected human pose and at least one detected object. Based on this, monitoring information is returned.

[0013] In particular, the use of explicit object detection and human pose detection improves the understandability of the monitoring methods compared to end-to-end trained artificial neural networks for action detection, which are typically used as black boxes. Furthermore, object detection and human pose detection are easier tasks than end-to-end action detection, making the monitoring methods faster to train, even if at least one of them uses an artificial neural network. In summary, the above monitoring methods demonstrate improved efficiency and reliability.

[0014] In some embodiments, at least one image comprises multiple consecutive frames of a video segment. Alternatively, at least one image may comprise multiple non-consecutive frames of a video segment selected, for example, at a given sampling frequency (e.g., every three frames of a video segment). Also alternatively, the at least one image may comprise one or more still images, such as photographs. Using frames from a video segment allows for the incorporation of temporal information, thereby enabling the acquisition of broader and more detailed surveillance information.

[0015] In some embodiments, the process includes a repetitive loop, and the method includes identifying at least one occurrence of the loop in a video clip and returning monitoring information for each of those occurrences. For example, on an assembly line, or more generally on a production line, industrial processes typically include the repetition of subprocesses or loops. In these cases, it is desirable to be able to identify the time boundary of one of these subprocesses, i.e., one occurrence of the loop, in a video clip and return monitoring information based on the content of that occurrence (optionally, independently of the content occurring in other occurrences of the loop). Monitoring information can be returned for each of the detected occurrences, thereby providing, for example, information about the product for each process. The monitoring information can have the same nature for each occurrence.

[0016] In some embodiments, the monitoring information is determined based on at least one geometric relationship in at least two consecutive frames. The two consecutive frames may belong to the same cycle. This allows for redundancy to limit false detections. The geometric relationship may be the same, for example, used to measure the time of an action performed during the cycle, or it may differ from one frame to another, for example, when a second given step is assumed to follow a first given step.

[0017] In some embodiments, at least one object includes an object that a person interacts with while performing the process. The at least one object may include an object (such as an object to be manufactured or repaired), a component (optionally, a component on which or using it to perform the process), equipment, or tools, etc. Alternatively or additionally, the at least one object may include a marker or reference point, a support (including a hanger), etc. Alternatively or additionally, the at least one object may include an object that a person must not interact with while performing the process, for example, because the object may represent a hazard.

[0018] In some embodiments, detecting at least one object includes determining a bounding box, and optionally determining the type of at least one object. The bounding box may be represented as a polygon (e.g., a rectangle). The object type may be selected from a predetermined list of possible object types.

[0019] In some embodiments, detecting human posture includes detecting multiple body joints or body parts. Body joints or body parts (hereinafter referred to as "body features") can be labeled, for example, corresponding to the head, left hand, right knee, foot, etc. Therefore, a precise assessment of geometric relationships can be performed.

[0020] In some embodiments, the monitoring information includes at least one of the following metrics: whether a step of the process has been performed by a human, whether a human is in danger, whether a human has made an error, human ergonomics, human efficiency, processing duration, or a combination thereof. Therefore, the performance, ergonomics, and safety of the process can be derived from the metrics (multiple metrics) output from the monitoring method, thereby enabling improvements to process definitions and guidelines.

[0021] In some embodiments, at least one geometric relationship includes the distance and / or overlap rate between the human pose and an object, and / or the human pose being within a region defined by a reference detected object, and the monitoring information is returned based on comparing the geometric relationship with predetermined rules. The distance, overlap rate, or being within a specific region may be determined for one or more body features, as appropriate, and the predetermined rules may be specifically defined based on some objects and some body features. An object can be represented by its bounding box. The distance may be the shortest distance between two items (e.g., a body feature and an object) detected in an image. The overlap rate may be defined as the surface ratio of two items in an image. However, other mathematical definitions are also possible, as long as they match the processing specifications of which body parts should or should not interact with which object.

[0022] In some embodiments, the process includes a manufacturing step of an object on a production line. In some embodiments, the above-described cycle includes a manufacturing step of an object on a production line. The production line may be an assembly line.

[0023] In some embodiments, at least one object includes a support for the object. The support for the object can provide a more stable or reliable reference than the object itself.

[0024] This disclosure also relates to an apparatus for monitoring processes to be performed by a person, the apparatus comprising:

[0025] A module for obtaining at least one image of the person performing the processing;

[0026] A module for detecting human pose in at least one image;

[0027] A module for detecting at least one object in at least one image;

[0028] A module for returning monitoring information about the processing based on at least one geometric relationship between the detected human pose and at least one detected object.

[0029] The device can be configured to perform the above-described monitoring method and can have some or all of the above-described features. The device can have a computer hardware structure.

[0030] This disclosure also relates to a system comprising the aforementioned apparatus, which is equipped with a video or image acquisition module to acquire at least one image. The video or image acquisition module may be a camera, etc.

[0031] This disclosure also relates to a computer program that includes instructions for performing the steps of the monitoring method described above when the program is executed by a computer.

[0032] The program can use any programming language and can take the form of source code, object code, or intermediate code between source code and object code, such as partially compiled code or any other desired form.

[0033] This disclosure also relates to a recording medium that can be read by a computer and on which a computer program is recorded, the computer program including instructions for performing the steps of the above-described monitoring method.

[0034] The recording medium can be any entity or device capable of storing programs. For example, the medium can include storage devices such as ROMs (e.g., CD ROMs or microelectronic circuit ROMs) or magnetic storage devices (e.g., disks (floppy disks) or hard disks).

[0035] Alternatively, the recording medium may contain a programmed integrated circuit adapted to perform the method in question or to be used in its execution. Attached Figure Description

[0036] The invention and its advantages will be better understood by reading the following detailed description of embodiments given as non-limiting examples. This description refers to the accompanying drawings, in which:

[0037] Figure 1 This is a diagram illustrating the steps of a computer-implemented method for monitoring processing according to an embodiment;

[0038] Figure 2 It is a diagram illustrating the geometric relationships based on the example;

[0039] Figure 3 This is a diagram illustrating the operation of a computer-implemented method for monitoring processing according to an embodiment. Detailed Implementation

[0040] Reference Figure 1A computer-implemented method (hereinafter referred to as the "monitoring method") for monitoring processes to be performed by a human, according to an embodiment, is described. As previously described, the monitoring method 10 includes an acquisition step 12 of obtaining at least one image of the person performing the processing. If the method is to be implemented in real time, the at least one image can be acquired in real time by an image acquisition module (such as a video acquisition module of a camera, for example, a video camera, a photographic camera, etc.). Alternatively or additionally, for example in the case of post-processing of photographic processing, the at least one image can be acquired in advance and obtained later by the monitoring method 10.

[0041] In the following, it is assumed that the at least one image comprises multiple consecutive frames of a video clip. Nevertheless, as described in detail above, other cases are envisioned, and the method can be adapted to one or more images, regardless of their origin.

[0042] At least one image (or consecutive frames of a video segment) obtained is provided as input to object detection step 14 and human pose detection step 16, either as is or after intermediate image processing. As will be described in detail below, object detection step 14 and human pose detection step 16 are configured to extract information about the processing. Object detection step 14 and human pose detection step 16 can be performed serially or in parallel. In one embodiment, as illustrated, object detection step 14 and human pose detection step 16 are independent, i.e., neither depends on the processing performed by the other to perform its own processing.

[0043] As previously described, object detection step 14 includes detecting at least one object in at least one image. In the case of multiple images (e.g., multiple frames), object detection step 14 may include detecting at least one object in one, some, or all of the multiple images. The object may be the same or may differ from one image to another.

[0044] Object detection step 14 may include performing a computer vision algorithm. More specifically, object detection step 14 may include using a deep learning-based object detector, for example, which may be trained to detect objects of interest from an image. In an example, the deep learning-based object detector may include YOLOv3 (J. Redmon, A. Farhadi, YOLOv3: An incremental improvement, arXiv:1804.02767, 2018)). However, other object detectors can also be used, such as EfficientNet (M. Tan, R. Pang, QV Le, EfficientDet: Scalable and efficient object detection, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), doi:10.1109 / cvpr42600.2020.01079,2020), RetinaNet (T. Lin, P. Goyal, R. Girshick, K. He, P. Dollár, Focal Loss for Dense Object Detection, Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp.2999-3007, doi:10.1109 / ICCV.2017.324,2017), and SSD (W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Fu, AC Berg,SSD:Single shot multibox detector,Proceedings of the European Conference on Computer Vision(ECCV),pp.21–37,doi:10.1007 / 978-3-319-46448-0_2,2016),FCOS(Z.Tian,C.Shen,H.Chen,T.He,FCOS:Fullyconvolutional one-stage object detection,Proceedings of the IEEEInternational Conference on Computer Vision(ICCV), pp.9627–9636, doi:10.1109 / ICCV.2019.00972, 2019), CenterNet (K.Duan, S.Bai, L.Xie, H.Qi, Q.Huang, Q.Tian, ​​CenterNet: Keypoint triplets for object detection, Proceedings of the IEEEInternational Conference on Computer Vision(ICCV), pp.6568–6577, doi:10.1109 / ICCV.2019.00667, 2019) etc. .

[0045] The object detection step 14 may include determining a bounding box and optionally determining the type of at least one object. Figure 3 The middle figure illustrates an example where object detection step 14 has detected the top-front portion 38 and the bottom-rear portion 40 of the hanger 36. As shown, the bounding box can be polygonal, such as rectangular. The bounding box can be returned by object detection step 14 as a list of vertex coordinates or in any other suitable format.

[0046] The object type can be selected from a predetermined number of object types to be detected on the image. These object types can be, for example, explicitly input into the computer vision algorithm during training, or learned from a deep learning model. See also Figure 3 These multiple object types can be "Hanger Top-Front" and "Hanger Bottom-Rear". However, other objects can be identified.

[0047] It is worth noting that object detectors can be quite general and only need to be trained on the objects to be detected. Therefore, the required annotation effort is minimal.

[0048] As previously described, the human pose detection step 16 includes detecting the human pose of the person performing the processing in at least one image. In the case of multiple images (e.g., multiple frames), the human pose detection step 16 may include detecting the human pose in one, some, or all of the multiple images. The person may be the same or different from one image to another. One or more people may be detected in at least one image.

[0049] Human pose detection step 16 may include performing computer vision algorithms. More specifically, human pose detection step 16 may include using a deep learning-based human pose estimator. Detecting human pose may include detecting a person's body features, such as one or more body joints and / or body parts, or the person themselves. In one example, human pose detection step 16 may include calculating a 3D and / or 2D skeleton for each person in the image. Body features may include at least one hand, at least one arm, at least one elbow, at least one shoulder, at least one foot, at least one leg, at least one knee, neck, and / or head for each person.

[0050] In one example, a deep learning-based human pose estimator could include LCR-Net (Rogez, Weinzaepfel, Schmid, LCR-Net: Real-time multi-person 2E and 3D human poseestimation, IEEE Trans. PAMI, 2019). However, other human pose estimators can be used, such as DOPE (Weinzaepfel, P., Brégier, R., Combaluzier, H., Leroy, V., Rogez, G., DOPE: Distillation of Part Experts for whole-body 3D pose estimation in the wild, ECCV, 2020), OpenPose (Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, YA Sheikh. OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019), and DeepCut (Leonid Pishchulin, Eldar Insafutdinov, Siyu Tang, Bjoern Andres, Mykhaylo Andriluka, Peter Gehler, Bernt Schiele, DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation, IEEE Conference on Computer Vision and Pattern Estimation). (e.g., Recognition (CVPR), 2016), AlphaPose (Fang, HS, Xie, S., Tai, YW, Lu, C., RMPE:Regional Multi-person Pose Estimation, ICCV, 2017), etc. It is worth noting that the human pose estimator can be quite general and only requires training using a general human pose. Therefore, a publicly available training set can be used without special adaptation. As a result, the setup of monitoring method 10 is fast and simple.

[0051] Figure 2 An example of a detected human pose 24 is shown, which includes multiple body parts such as the neck 26 and multiple body joints such as the left knee 28. In this figure, the detected body parts and the detected body joints connecting the body parts to each other form a skeleton. The detected body features 26, 28 can be returned by the human pose detection step 16 as point or line coordinates or in any other suitable format.

[0052] Human posture detection step 16 provides a representation of a person's posture or pose while performing this process. This representation can be used to perform ergonomic studies and may be used to adapt the process to provide better ergonomics for people.

[0053] See you again Figure 1 The detected objects and detected human poses, output from the corresponding detection steps 14 and 16, are provided as input to a determination step 18, which is configured to return monitoring information about the process based on at least one geometric relationship between the detected human pose and at least one detected object. The detected objects and detected human poses form a digital representation of the process to be monitored.

[0054] Figure 2 The diagram illustrates an example of how step 18 can be performed. Figure 2 The detected human posture 24 and the detected object 32, represented by its bounding box 32a, are shown. In this example, the object type is designated as hazardous, for example, because the object type corresponds to an object that a person should not approach. Determination step 18 evaluates the geometric relationship between the human posture 24 and the detected object 32. For example, it is specified that if any part of the human posture 24 overlaps with the bounding box 32a, the person is in danger while performing the action. In the current case, since the right hand 30 overlaps with the bounding box 32a, determination step 18 will return monitoring information indicating a hazard.

[0055] Step 18 can rely on a rule engine that includes one or more predefined rules and evaluates whether the geometric relationship between the detected object and the detected human pose satisfies one or more of the rules. In other words, the rule engine can include geometric reasoning logic.

[0056] Figure 2The example illustration depicts a possible rule. Those skilled in the art will understand that many different rules can be used as the basis for determining step 18 in a rule engine. For example, at least one geometric relationship may include the distance and / or overlap rate between the human pose and the object, and / or the human pose being within a region defined by the detected object. The geometric relationship may be applied to the entire human pose or only to a portion of it, such as when examining which hand is performing or not performing a specific action. Relevant body features may or may not be specified: the rule may apply only to some predetermined body features, or it may be satisfied as long as any body feature meets the condition. For example, depending on the human pose and how the object is detected, the geometric relationship may be determined in 2D and / or 3D. In one embodiment, even if the object is initially detected in 2D, its 3D position can be estimated based on given data from the process to be monitored, such as the object actually always having the same size, the camera used to acquire the image being fixed, etc. The geometric relationship can then be determined in 3D. 3D determination allows for more precise and representative monitoring.

[0057] Instead of or attached to a dangerous object that a person should avoid, at least one object may include an object that a person must interact with while performing the processing. For example, this would correspond to an object that a person must manipulate or work on while processing it.

[0058] These rules can be derived from the processing criteria: since the processing (especially in the case of industrial processing) is well defined, these definitions can be transformed into mathematical rules that will or will not satisfy geometric relations. This is easier to implement than other methods where rules must be developed from scratch, because these rules already exist and only geometric transformation is required.

[0059] Based on this, the monitoring information is determined and returned. The monitoring information output by determination step 18 may include information about video frames during which a specific event (corresponding to one or more predetermined rules) has occurred, and for each of these frames, the location where such an event occurred. This location may be determined based on the location of the detected human pose and the location of the detected object.

[0060] Furthermore, this monitoring information can be determined based on at least one geometric relationship between at least two frames in the frame. These frames can be continuous or discontinuous. Considering the time dimension provides richer monitoring information, such as determining how much time a person spends on which tasks, and ultimately detecting potential quality defects stemming from non-compliant processing. Alternatively or additionally, the determination step can include time rules to check whether actions were properly performed during a normally applicable period of time, as opposed to unintentional redundant gestures by a person that might accidentally meet the rule for a relatively short period.

[0061] Back Figure 1 Optionally, the monitoring method 10 may include a sorting step 20 to evaluate the monitoring information predictions returned by the determination step 18 and improve these predictions by removing potential errors, such as inaccurate assessments of geometric relationships.

[0062] Sorting step 20 may include executing a classifier. More specifically, sorting step 20 may include using a human activity recognition algorithm. In one example, this algorithm may be based on dense trajectory (Wang Heng, Alexander, Schmid Cordelia, Liu Cheng-Lin. (2014). Wang H2013-dense trajectories-IJCV), although other methods are also possible. Once trained, the classifier is able to distinguish the monitored processing from other pseudo-behaviors. In one example, training can be based on a manually labeled dataset that determines the output of step 18, and thus from multiple samples (e.g., 700) showing the processing and multiple samples (e.g., 700) showing the pseudo-behavior. However, other training methods are also covered: after several iterations, if necessary, the selection of an appropriate classifier and the definition of the number of training samples can be performed by someone skilled in the art based on their knowledge of the field.

[0063] Reference Figure 3 An example of applying the monitoring method to actual processing is described. Figure 3 The process includes the manufacturing steps of objects (in this case, motor vehicles) on the production line. However, it also covers other processes, whether on or off the production line, and alternatives to manufacturing may include, for example, repair or utilization.

[0064] In this case, the manufacturing steps include the installation of the cable grommets in the taillights of the vehicle. However, other steps are covered: the vehicle does not need to be a car, and the step does not need to be installation, or it could be the installation of another component.

[0065] On the assembly line, car 34 moves forward, that is... Figure 3 The process moves from left to right, followed by another car, on which similar (if not identical) steps are typically performed. Each repetition of these steps constitutes a loop, and this example illustrates a case where the process involves repeatedly executed loops. In these cases, the method may include, for example, identifying at least one occurrence of the loop within a video clip, and returning monitoring information for each of those at least one occurrence. Therefore, two sub-problems must be addressed: detecting the occurrence (loop segment) and returning monitoring information for each occurrence.

[0066] For forward movement, in this example, the car 34 can be supported by a support (e.g., a hanger 36). At least one object to be detected may include a portion of the car (object) 34 itself or a portion of the support (here, the hanger 36): as previously described, in this example, object detection step 14 is configured to detect the top-front portion 38 and the bottom-rear portion 40 of the hanger as objects of interest. However, other portions may be detected additionally or alternatively. Furthermore, even if the car 34 does not move, it can still be supported by the support. While the car model may vary, the support can be the same; therefore, the support can provide an unchanging reference for estimating its position in the image.

[0067] Specifically, in Figure 3 In the example, monitoring method 10 should determine the start and end of each occurrence. One possibility is to consider a cyclic boundary line 46 on the image. The cyclic boundary line 46 can be an imaginary line (e.g., the edge of the image or a line at a set distance therefrom) or a real line (e.g., a boundary marker on an assembly line). The cyclic boundary line 46 can be straight or curved. It also covers other boundaries besides lines.

[0068] Based on the fact that the gantry 36 moves together with the vehicle 34, the monitoring method can determine the start or end of a cycle whenever a given portion of the gantry 36 (and / or the vehicle 34) crosses the cycle boundary line 46. For example, in Figure 3 In the example, a new occurrence is determined when the top-front portion 38 of the hanger crosses the cycle boundary line 46, and the current occurrence is determined to end when the bottom-rear portion 40 of the hanger crosses the cycle boundary line 46. Other rules can also be set; specifically, the cycle boundary line 46 does not need to be the same for detecting the start and end of an occurrence. Furthermore, the end can be not explicitly detected, but can be set to correspond to the start of the next cycle. Conversely, the start can be not explicitly detected, but can be set to correspond to the end of the previous cycle.

[0069] The detection of the start and / or end of an event typically triggers very few errors (if any), making it possible that the resulting prediction does not need to be processed through sorting step 20, although the remaining monitoring information does. This results in improved computational efficiency.

[0070] To determine the remaining monitoring information, in this example, the geometric relationships are defined as follows: a hand region 42 and a foot region 44 are defined based on the position of the hanger bottom-rear 40. These regions are defined as polygons, for example, at a set coordinate distance from the hanger bottom-rear 40. In determination step 18, when the person's hand is in the hand region 42 and his foot is in the foot region 44, it is determined that the grommets are being installed. More generally, the geometric relationship between the detected human posture and the detected object can include the human posture being partially or entirely within the regions defined by reference to the detected object.

[0071] In this embodiment, the segmentation of several video segments occurring simultaneously with the determination of surveillance information can be performed: the determination step 18 may include specific rules for identifying which occurrence the current image belongs to, while other rules are designed to determine the remaining surveillance information related to the detected human pose. In other embodiments, the segmentation of several video segments occurring in a loop can be performed between the acquisition step 12 and the detection steps 14, 16, or even before the acquisition step 12, in which case the acquisition step 12 may take a single occurrence as input.

[0072] As detailed above, the monitoring information can then proceed through sorting step 20.

[0073] The monitoring information output by determining step 18 and / or sorting step 20 may include at least one of the following indicators: whether the processing steps have been performed by a person (e.g., if hands and feet are in the hand area and foot area respectively), and whether the person is in danger (e.g., as referenced). Figure 2 The metrics may include: whether the person made an error (e.g., if the error occurred for an unusual duration, or if the person's posture was not the expected posture); the person's ergonomics (e.g., based on posture); the person's efficiency (e.g., based on unnecessary gestures or processing completion time); processing duration; or a combination thereof. This metric may be output as a continuous or discrete value, or in any other suitable format.

[0074] although Figure 1 The diagram has already been described in terms of the method steps, which can equivalently represent the architecture of an apparatus for monitoring a process to be performed by a human, the apparatus comprising: a module 12 for acquiring at least one image of the human performing the process; a module 14 for detecting the human body posture in the at least one image; a module 16 for detecting at least one object in the at least one image; and a module 18 for returning monitoring information about the process based on at least one geometric relationship between the detected human body posture and the detected at least one object. The apparatus may be a computer or a computer-like system. Figure 1As illustrated, the device may be equipped with a video acquisition module, shown as a camera in acquisition module 12, to acquire at least one image.

[0075] While this disclosure relates to specific exemplary embodiments, variations may be provided with these examples without departing from the general scope of the invention as defined by the claims. In particular, various features of different illustrated / mentioned embodiments may be combined in additional embodiments. Therefore, the specification and drawings should be considered in an illustrative rather than restrictive manner.

Claims

1. A method for monitoring computer-implemented processes to be performed by a human, comprising: Obtain at least one image of the person performing the processing, the at least one image comprising multiple consecutive frames of a video clip; Detect the human posture of the person in the at least one image; Detect at least one object in the at least one image; Determine at least one geometric relationship between the detected human pose and at least one detected object; Based on at least one geometric relationship between the detected human pose and the detected at least one object, monitoring information about the processing is returned; The processing includes a cycle of repeatedly executed manufacturing steps of an object on a production line, and the method includes: identifying at least one occurrence of the cycle in the video clip, and returning the monitoring information for each of the at least one occurrence; and the method further includes: evaluating the prediction of the monitoring information outside the start or end of the at least one occurrence, and improving the prediction of the monitoring information by removing potential errors.

2. The method according to claim 1, wherein, The monitoring information is determined based on at least one geometric relationship in at least two of the consecutive frames.

3. The method according to any one of claims 1 or 2, wherein, The at least one object includes an object that the person interacts with while performing the processing.

4. The method according to any one of claims 1 or 2, wherein, The detection of at least one object includes determining a bounding box and optionally determining the type of the at least one object.

5. The method according to any one of claims 1 or 2, wherein, The detection of human posture includes detecting multiple body joints or body parts of the person.

6. The method according to any one of claims 1 or 2, wherein, The monitoring information includes at least one of the following indicators: whether the steps of the process have been performed by the person, whether the person is in danger, whether the person has made a mistake, the person's ergonomics, the person's efficiency, the duration of the process, or a combination thereof.

7. The method according to any one of claims 1 or 2, wherein, The at least one geometric relationship includes the distance and / or overlap rate between the human pose and the object, and / or the human pose being within a region defined by the detected object, and the monitoring information is returned based on comparing the geometric relationship with predetermined rules.

8. The method of claim 1, wherein the at least one object comprises a support for the object.

9. A device for monitoring a process to be performed by a person, the device comprising: A module for obtaining at least one image of the person performing the processing, the at least one image comprising multiple consecutive frames of a video clip; Module for detecting the human posture of the person in the at least one image; A module for detecting at least one object in the at least one image; A module for determining at least one geometric relationship between a detected human pose and at least one detected object; A module for returning monitoring information about the processing based on at least one geometric relationship between the detected human pose and the detected at least one object; The process includes a cycle of repeatedly executed manufacturing steps of an object on a production line. The apparatus includes a module for identifying at least one occurrence of the cycle in the video clip and returning the monitoring information for each of the at least one occurrence. The apparatus also includes a module for evaluating predictions of the monitoring information outside the start or end of the at least one occurrence and improving the predictions of the monitoring information by removing potential errors.

10. A system comprising the apparatus of claim 9, the apparatus being equipped with a video acquisition module to acquire the at least one image.

11. A recording medium readable by a computer, and having thereon recorded a computer program comprising instructions for performing the steps of the method according to any one of claims 1 to 8.

Images