Computer-implemented method, system and computer program for providing an examination record related to a technical device

Abstract

A human operator (201A) performs activities within a facility that interacts with technical equipment (301A, 301B, 301C). A series (700) of images (701, 702) visualize activities performed by the same human operator (201A) or different human operators. In the images, the computer separates regions (701x, 702x) that include biometric image data from regions (701y, 702y) that have data indicative of interactions. The computer obtains an identifier (ALPHA, ALPHA) of the operator (201A) but removes the biometric image. The computer then combines the image regions by storing the modified first and second images (701', 701') and the identifier and provides an audit record (500).

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to industrial processes; and more specifically, the present disclosure relates to providing review of records of image processing of industrial product handling processes using a computer-implemented method, computer system or computer program for image processing. BACKGROUND

[0002] Handling products in facilities such as factories, plants, assembly lines, warehouses, laboratories, etc. is traditionally a task of human operators interacting with technical equipment such as machines, tools, gadgets, vehicles, etc. Automation does not change this. Human activities remain part of the product handling process.

[0003] Product handling, in the broadest sense, includes manufacturing products, testing or analyzing products, storing products, distributing products, or even destroying products. The term "product" used herein applies to all stages of the product life cycle and includes: product components, intermediate products, and samples extracted from production.

[0004] Furthermore, there can be discrete things (e.g. a product is a bottle filled with a beverage) and continuous streams of matter (e.g. supplying water to a beverage through a pipe).

[0005] Interactions between humans and technical equipment (human-device interactions) create risks. One risk is related to product quality. For example, at the end of a production line, a product can look like it can be shipped to a consumer, but in fact the product can have been damaged beyond repair. Risks can be industry-specific. Human operators can affect product performance by applying incorrect parameters (e.g. quantity of material, time settings, pressure or temperature settings, etc.). For example, in the general chemical industry and its branches (e.g. fragrances, pharmaceuticals, human and animal nutrition, health industry), specifically, human operators can inadvertently come into contact with products during production, rendering the products useless. As a result, the products can be contaminated with bacteria, making it possible for the end consumer to contract an illness. In a laboratory, test samples can be prepared or analyzed incorrectly due to mishandling.

[0006] Additional risks include safety hazards for the operators themselves.

[0007] To mitigate such risks and others, product handling needs to comply with requirements summarized by a large number of technical standards. Such requirements relate to proper instructions for operators, parameters for operating facilities and / or equipment, procedures for tracking production by batch or load, cleaning equipment (before or after use), safety precautions, etc.

[0008] Just to name a few introductory examples, in the context of the chemical industry: standards can require personnel to adhere to predefined procedures and create records, e.g. batch records, cleaning protocols, test protocols, etc.; standards can require operators to wear appropriate protective gloves (or generally protective clothing) in order to minimize the risk of coming into contact with products; standards can require materials to be delivered via a parameterized supply chain; standards can specify air quality within a clean room; and standards can require production to be tracked by computer, by filling, by batch, etc.

[0009] Just to name a few examples, for one of the numerous standards for the handling of industrial products, the following is mentioned:

[0010] • COMMISSION DIRECTIVE 2003 / 94 / EC of 8 October 2003 laying down principles and guidelines of good manufacturing practice for medicinal products for human use and investigational medicinal products, Official Journal of the European Union, 14 October 2003, L 262 / 22,

[0011] • ISO 9001 :2015-09 Quality management systems - Requirements,

[0012] • ISO 14001 :2015-09 Environmental management systems - Requirements and guidance for use,

[0013] • Food Safety System Certification (FSSC) 22000, available from FSSC 22000 Foundation,

[0014] • Food safety in the production process systematic prevention methods from biological, chemical and physical hazards, known as "Hazard Analysis and Critical Control Point (HACCP)".

[0015] A standard can refer to another standard. As used herein, the term "standard" also includes any collection of specific requirements for a specific production batch.

[0016] Despite standardization, the interaction between human operators and equipment is not free from errors or malfunctions. Therefore, it is necessary from time to time to monitor and check whether the product handling complies with the standards. Traditionally, this is the task of so-called auditors (sometimes referred to as "inspectors" or "checkers"). The auditor must distinguish between information relevant to the assessment of compliance with the standards (i.e. information relevant to the requirements to be included in the report) and information that must not leave the facility (i.e. information to be excluded from any report).

[0017] As of today, the review cannot be automated. However, consider the review officer to be physically located far from the facility. US2016 / 0306172 Al explains the use of a wearable camera device to transfer data from the facility to an officer located outside. Since the data is sent in real time, the problem of selectively including / excluding information remains.

[0018] There is a need to provide a technique that solves the inclusion / exclusion dilemma. SUMMARY

[0019] To comply with technical standards, it is often required that subsequent product handling activities are performed by the same operator. However, it is irrelevant who the operator is. More specifically, the operator's personal data is irrelevant (personal data is any information related to an identified or identifiable operator as a natural person by reference to such as name, identification number, location data, online identifiers, or by reference to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person).

[0020] The present invention solves the inclusion / exclusion dilemma by using image processing. According to an embodiment of the invention, a review system receives images from a camera located within a facility. The images (i.e. pictures or photos in electronic form, having pixels) visualize an operator performing a predefined activity. The review system processes biometric image data to obtain an identifier of the operator, but then ignores or removes the biometric image data. The review system then provides a review record with a modified image incorporating the identifier. Due to the modification, no personal data remains in the review record. The identifier cannot be converted back to personal data.

[0021] An assessor inspecting the review record (i.e. looking at the identifier in the record) can still identify whether subsequent activities were performed by the same operator.

[0022] This approach exploits the fact that human operators typically perform activities with their hands and that a viewer (of the images) recognizes a particular person by looking at the face rather than at the hands.

[0023] In more detail, a computer-implemented method for providing a review record involves an activity performed by a human operator within a facility. The activity is an interaction of the human operator with a technical device. The review record indicates the performance of a sequence of activities having a first activity and a subsequent second activity by the same human operator or by different human operators. The method comprises the following:

[0024] A series of images is received. The images visualize an activity performed by a human operator. The series has at least a first image of a first activity of interacting with a first technical device and a second image of a second activity of interacting with a second technical device.

[0025] By applying image processing on the first and second images, at least a first and a second area are separated within the first and second images. The first area comprises biometric image data indicative of a specific human operator within a group of human operators (e.g. operators belonging to a manufacturing facility). The second area comprises image data indicative of an interaction with the technical device.

[0026] By applying image processing on the first area of the first image and on the first area of the second image, an identifier of a human operator among the plurality of human operators is obtained. The identifier is thereby indicative of the execution of the first and second activities by the specific human operator.

[0027] By applying image processing on the first area of the first image and on the first area of the second image, biometric image data indicative of the specific human operator is removed from the first area of the first image and from the first area of the second image. The first area of the first image is modified into a modified first area of the first image and the first area of the second image is modified into a modified first area of the second image.

[0028] The modified first area of the first image is combined with the second area of the first image into a modified first image and the modified first area of the second image is combined with the second area of the second image into a modified second image.

[0029] Providing an audit record comprises storing the modified first image and the modified second image together with the identifier.

[0030] Optionally, the receiving, separating, obtaining and removing steps are performed by a processor of a computer system using only volatile memory.

[0031] Optionally, the receiving comprises receiving a series of images with a timestamp of the images.

[0032] Optionally, the receiving comprises receiving the series of images by receiving the first image from a first camera and the second image from a second camera. The first and the second cameras are respectively associated with a first technical device and a second technical device.

[0033] Optionally, the receiving comprises receiving technical parameters related to the technical device. The technical parameters are associated with the images as part of the images or as in a metadata attachment to the images.

[0034] Optionally, receiving comprises receiving technical parameters related to the technical device from a device interface. The technical parameters are related to the first image or the second image.

[0035] Optionally, receiving the series of images is triggered by a production management system providing an identification of a specific batch. As a result of providing the review record, the review record is related to the specific batch.

[0036] Optionally, receiving the series of images is triggered by a production management system providing an identification of a first batch and an identification of a second batch. Receiving the first image of the first activity is related to both batches. Receiving the second image of the second activity is performed in batch-specific versions for the first batch and for the second batch. Providing the review record thus comprises providing separate review records for the first batch and for the second batch.

[0037] Optionally, the biometric image data comprises data related to a face of the human operator.

[0038] Optionally, the biometric image data comprises data related to a face of the human operator. In the step of obtaining an identifier, an image of an operator in a specific group of operators is used as a reference.

[0039] In one embodiment, a computer program product, when loaded into the memory of a computer and executed by at least one processor of the computer, performs the steps of the computer-implemented method.

[0040] In one embodiment, the review system is a computer system adapted to perform the method according to any of the above examples. Such a system has modules dedicated to the steps of the method. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1A and Figure 1B An overview of a facility with people and with devices is shown, wherein Figure 1A A conventional method is shown, and wherein Figure 1B A way of using a computer system and a computer-implemented method is shown;

[0042] Figure 2 An operator interacting with a device is shown;

[0043] Figure 3 A case of non-compliance with a standard in cases (A) to (D) is shown;

[0044] Figure 4 A computer system performing image processing leading to a review record in an example with a first scenario is shown;

[0045] Figure 5 A computer system is shown performing image processing that leads to a review report in an example with a second scenario;

[0046] Figure 6 A flowchart of a computer-implemented method is shown, which is used to provide audit logs related to activities performed by human operators and technical equipment within a facility;

[0047] Figure 7 The steps for performing the method separately for different production batches are shown;

[0048] Figure 8A , Figure 8B and Figure 8C An image of a device with technical parameters is shown;

[0049] Figure 9 An embodiment with multiple cameras transmitting images to a computer system is shown;

[0050] Figure 10 An embodiment of a camera with a modified lens that transmits images to a computer system is shown; and

[0051] Figure 11 A series of multiple images captured by a camera are shown. Detailed Implementation

[0052] Facilities and Systems Overview

[0053] Figure 1A and Figure 1B An overview of facility 100 with 200 personnel and 300 technical equipment is shown. Figure 1A Suitable for traditional methods, while Figure 1B Applicable to computer system 400 and computer-implemented method 600 (in which...) Figures 4-6 (Detailed explanation below). For convenience, computer system 400 will be labeled "Review System".

[0054] In very simple terms, facility 100 can be a factory, assembly line, warehouse, laboratory, etc. Technical equipment 300 is collectively referred to as equipment 301A, 301B, 301C...301Z. In the chemical industry, examples include glove dispensers, cleanroom locks, tank reactors, bags containing materials (such as ingredients), heaters, ovens, conveyor belts, etc.

[0055] As used herein, the standard applies to the situation within facility 100. The description and figures refer to facility 100 and equipment 300 in the cleanroom example.

[0056] In this example, the standard for the clean room involves technical equipment 300, such as air conditioning equipment, locks between rooms, but also activities that the human operators must perform (see Figure 2 ).

[0057] As used herein, the personnel 200 can have multiple roles (or functions), including operators 201, auditors 202, and evaluators 203.

[0058] The operators 201 perform activities within the manufacturing facility. There are typically multiple operators in a single facility, distinguished here as 201A, 201B, 201C,... 201Z. It is not required that all operators be present at the same point in time. Since the operators 201 are human beings, each person has specific biometric characteristics: a specific face, a specific body shape or build, a specific weight, a specific voice, and so on.

[0059] The biometric characteristics that humans can recognize by looking at each other are relevant here: these are visual biometric characteristics (i.e., face, body shape / build). In the figures, they are represented with different faces. For simplicity, it is assumed that there are no identical twins among the operators.

[0060] The operators 201 perform interactive activities with the equipment 300. There is no one-to-one relationship between a specific operator and a specific piece of equipment. A specific operator 201 (e.g., 201A) can interact with different pieces of equipment (e.g., 301A, 301B, 301C) at the same time or at different points in time. All operators can interact with a specific piece of equipment (e.g., 301A), but not at the same time.

[0061] The auditors 202 Figure 1A ) visit the manufacturing facility and audit the performance of the activities there. In terms of content, the audit typically includes determining whether and how the operators (in general) perform the activities that interact with the equipment (i.e., equipment components). The ways of auditing can include visual inspection (i.e., looking at the operators and / or equipment) and verbal inspection (i.e., talking to the operators and supervisors, listening to voices, etc.). As mentioned above, traditionally, the auditors collect their findings in an audit report 402 in writing and / or by taking photographs.

[0062] As in Figure 1A , the evaluators 203 examine the audit report 402 and evaluate it for compliance with the standard 450. The standard 450 is here represented by a document with multiple requirements (A), (B), (C), and (D). The description and the figures use a very simplified example of a fictitious standard that requires the operators to do the following (e.g., in a nutrition or pharmaceutical environment):

[0063] • (A) wear gloves when locking the clean room,

[0064] • (B) putting on two gloves when putting raw material into the reactor,

[0065] • (C) changing gloves within an interval (e.g. every 2 hours), and

[0066] • (D) performing these activities by the same operator.

[0067] In this example, the fictitious criterion is written with "positive activities", but it is also possible to use undefined.

[0068] In this example, criterion 450 focuses on the behavior of the operator 201 (i.e. the execution or non-execution of pre-defined activities), but there can be further requirements (from different criteria) that are only related to the equipment.

[0069] Due to the nature of the review, the reviewer 202 and the evaluator 203 are usually not the same person as the operator 201. The functions of the reviewer and the evaluator can be performed by the same person. These figures illustrate the evaluator 203 as being located outside the facility.

[0070] This does not exclude the possibility that the computer can support the evaluation. This will be explained hereinafter.

[0071] Figure 1B The way in which the review system 400 with the execution method 600 is illustrated. Figure 1B The reviewer 202 is illustrated by a dashed line, because this function is no longer needed. However, this way is not an automation of the review, but a way of generating a review record, which allows for an offline and / or remote review of the handling of the industrial product.

[0072] The cameras 310-1 and 320-2 are associated with the equipment 300 to take images (or "capture images", i.e. data structures representing reality). Real-world objects from which images are taken include the equipment and / or the operator. The review system 400 processes the images and provides the review record 500 as a data structure.

[0073] In the example of Figure 1B , the camera 310-1 is associated with the equipment 301A to take images when any operator interacts with this equipment 301A; the camera 310-2 is associated with the equipment 301B and 301C to take images when any operator interacts with this equipment 301A, 301B or both.

[0074] The operator has no access to the cameras 310, the review system 400 and the review record 500.

[0075] Figure 1B It is also shown that the review system 400 can be coupled to a production management system 410 (see also Figure 7and the audit record 500 can be evaluated by a further computer system, e.g. the evaluation system 800 (which uses the standard requirements as rules for evaluating the audit record 500).

[0076] Figure 2 The operator 201A is shown to interact with equipment at least partially by way of example. In this example, it is assumed that the facility is a manufacturing facility in the chemical industry. The applicable standard requires that manufacturing is performed at a clean room location. Activities 210 and 220 are typical activities frequently performed within such a facility, as used herein. In view of the standard, the performance (or non-performance) is relevant. Activities 210 and 220 can be identified by visual inspection.

[0077] In a simplified example of Figure 2 At time point tl, the operator 201A performs (first) activity 210 by carefully checking the clean room lock. He washes his hands and dons gloves 399 from a glove dispenser (here: equipment 301A). From the standard, there are special requirements for the gloves (i.e. time limit to be observed after opening the dispenser, sterile gloves indicated by a specific color, etc.) and for the activity (e.g. wearing them, changing them regularly according to a change interval TINTERVAL, etc.).

[0078] At time point t2, the operator 201A has been in the clean room, and the operator 201A performs the second activity 220 by opening a bag (or "pocket", equipment 301B) containing a food ingredient (e.g. sugar or flour) and pouring the food ingredient into a tank reactor (equipment 301C), performing the "add ingredient" activity. It is assumed that time points tl and t2 are within the replacement interval TINTERVAL.

[0079] At time point t3, the operator 201A has left the clean room to perform an irrelevant activity 230 (e.g. resting, details not shown). It is assumed that this activity is not relevant to the standard. Note that the operator 201A has taken off the gloves. There is no activity to monitor, and there is also no human-machine interaction (at least no interaction of interest).

[0080] At time point t4, the operator 201A re-performs the activity of tl, here the first activity 210* is repeated.

[0081] Looking briefly at Figure 1A The auditor 202 will look at the operator 201A and write the following audit report 402:

[0082] (A) Wearing gloves when locking? Yes

[0083] (B) Wearing both gloves when loading the reactor with the ingredient? Yes

[0084] (C) Considered glove change interval? Yes

[0085] (D) Same operator? Yes

[0086] The assessor 203 (possibly the same person) will determine if the criteria are met (all Yes). Potentially, the review report 402 will become a pass statement (enhanced by a note "PASSED" or the like).

[0087] In other words, there needs to be evidence that the specific operator performed the pre-manufacturing activity (e.g. put on gloves) before the (subsequent) manufacturing activity. This proof is available in the proof document (i.e. the review report 402).

[0088] Note: The reviewer 202 cannot see the operator all the time (i.e. t1, t2, t4, not t3). It can be difficult for the reviewer 202 to keep track of the activities of various operators working in the same facility. The identity of the operator is needed in the sense that the same operator actually performed these activities (see (D)).

[0089] It should be noted that the activities within the facility 100 can be do-not-activities (or "negative activities") in the sense that certain activities are not allowed within the facility 100 according to the criteria. Take the clean room as an example. It is possible to comb your hair or clean your nails, but it is not allowed to do so in the clean room (i.e. further requirement (E) as a negative activity). There is a dedicated room - the bathroom or similar - where this can be done. Going to the toilet, by contrast, is an example of an unrelated activity 230 (at t3) that is not monitored by (the method described here). In other words, the performance of a do-not-activity is of interest for the assessment, but the performance of an unrelated action is not.

[0090] However, there are many other cases that can lead to a non-compliance. The reviewer 202 will have to keep an eye on this. This is explained in the following:

[0091] Figure 3 The non-compliance with the criteria in cases (A) to (D) is illustrated by way of example. It is noted that not adhering to at least one of the criteria requirements leads to a non-compliance with the criteria.

[0092] • In case (A), the operator did not wear gloves when locking. The operator wore gloves, but possibly different ones.

[0093] • As in case (B), the operator did not wear two gloves when loading the ingredients into the reactor. The operator lost one.

[0094] • As in case (C), the operator did not consider a glove change (in case of a change interval).

[0095] • In case (D), there are different operators interacting.

[0096] Method for providing an audit record

[0097] Figures 4-6 A computer system 400 (i.e. an audit system) is shown performing a computer-implemented method 600 for providing an audit record 500. As the method 600 includes image processing, Figures 4-5 An image is shown. Figure 6 A flowchart is shown (with steps 610 to 660). For simplicity, Figures 4-6 Computer components such as processors or memories are not shown. Performing the method 600 does not necessarily result in an audit record with a standard compliance statement (e.g. PASSED) or with a standard non-compliance statement (e.g. FAILED). The method 600 is applicable to scenarios resulting in PASSED (see Figure 4 ) and to scenarios resulting in FAILED (see Figure 5 ).

[0098] Figure 4 A computer system performing image processing resulting in an audit record 500 is illustrated in an example with a first scenario. Reference numerals (e.g. "301") are not part of the image.

[0099] An audit system 400 (see Figure 1B ) receives a series 700 of images 701 and 702 (from cameras). In this example, image 701 is from camera 310-1 and image 702 is from camera 310-2. The cameras take images at subsequent points in time, and (in embodiments) the images have timestamps.

[0100] In the series 700 of images 701 and 702 (two images are simplified for ease of explanation), image 701 has captured a first activity (interacting with a first device (see 210 in Figure 2 ) and image 702 has captured a second activity (also interacting, see 220 in Figure 2 ).

[0101] The audit system 400 then applies image processing techniques to split the images into ID regions 701x, 702x and activity regions 701y, 702y), to obtain an identifier (e.g. ALPHA) and to remove biometric data.

[0102] The review system 400 divides the image into (see step 620) the ID region 701x, 702x (of the image 701, 702) and the activity region 701y, 702y (of the image 701, 702). A region is a contiguous plurality of pixels. A region has a shape, e.g., a rectangle, a circle, or other shape. Coordinates can be used to process the regions, e.g., XY coordinates of the pixels. Similar to a geographically known exclave / enclave, one region can enclose another region.

[0103] The ID region includes biometric data (e.g., a facial image), while the activity region shows activity (i.e., operator interaction with the device).

[0104] The review system 400 then processes the ID region and obtains (step 630) the identifier ID1, ID2 for the ID regions 701x and 702x, respectively. In the example, the identifiers ID1 and ID2 are ALPHA. Figure 4

[0105] The review system 400 removes (step 640) the biometric data (see step 640). This is illustrated in the figure using a graphical anonymizer. But other methods can be used as well (e.g., by switching substantially all pixels of the ID region to a single color, e.g., black or white). The removal is in one direction, without opportunity to reestablish the content (i.e., to retrieve the biometric data). The removal of the biometric data is a one-way data conversion. The identifier (ALPHA) cannot be converted back to the personal data.

[0106] In the last step, the removal of the content can reduce the number of bits and bytes used to store the image. This is a further benefit. The saved bits and bytes can be used to transmit data (e.g., to transmit parameters in Figure 8A , Figure 8B , Figure 8C ).

[0107] The review system 400 combines (650) the modified first region 701xx of the first image 701 with the second region 701y of the first image 700 into a modified first image 701'. The review system 400 combines (650) the modified first region 702xx of the second image 702 with the second region 702y of the second image 702 into a modified second image 702'.

[0108] The review system 400 provides a review record 500 by storing the modified first image 701', the modified second image 702' (i.e., the modified series 700') and storing the identifier for each modified image (e.g., ID1 = ALPHA, ID2 = ALPHA). ​

[0109] For convenience, the identifier (e.g., ALPHA) is shown as part of the modified image (i.e., the identifier is embedded), but the identifier can be related to the modified image in other ways (e.g., as metadata, using known data binding techniques).

[0110] Note that the review record 500 does not provide an assessment (pass, fail) according to the standard, but the review record 500 contains all the information needed to perform the assessment.

[0111] The assessor 203 (see Figure 1B ) can examine the modified series 700' (i.e., the review record 500) and check if the requirements are met:

[0112] • (A) Gloves on when locking? Yes, this is visible in image 701'

[0113] • (B) Both gloves on when loading the reactor with raw material? Yes, this is visible in image 702'

[0114] • (C) Consider the change interval? Yes, this is not shown in the figure, but images 701, 702 have timestamps, and the timestamps are also taken over to the modified images 701' and 702'. Looking at the timestamps, the assessor 203 can determine if the interval has passed.

[0115] • (D) Same operator? Yes, the assessor 203 reads ALPHA associated with both images 701' and 702'.

[0116] Looking at the review record 500, the assessor 203 can make a judgment according to the standard (non-) compliance. But unlike the traditional reviewer (see Figure 1A 202), the assessor does not have to exclude information. In other words, information that is not allowed to leave the facility (e.g., the operator's facial data). The above inclusion / exclusion dilemma has been solved.

[0117] Figure 5 Computer system image processing leading to a review report is shown in the example with a second scenario. Figure 5 The process in Figure 4 is the same as in Figure 3 , but the content of the review record 500 is slightly different. In case (D) of there are activities performed by different operators (see 201A and 201Z).

[0118] Figure 5 AlthoughThe review record 500 does not indicate compliance, but it does instruct ALPHA (as the operator performing the first activity) and BETA (as the operator performing the second activity). The evaluator 203 will determine non-compliance (e.g., failure). The inclusion / exclusion dilemma has also been resolved.

[0119] It should be noted that images 701 and 702 (in) Figures 4-5 (In the middle) sufficient for the person acting as reviewer / evaluator to determine whether ( Figure 4 ) or does not conform to ( Figure 5 However, inclusion / exclusion remains an obstacle.

[0120] Figure 6 A flowchart of a computer-implemented method 600 is shown for providing a review record 500 of activities 210, 220 involving interaction between a human operator 201 and technical equipment 300 within facility 100.

[0121] Review record 500 indicates an activity sequence having a first activity (e.g., 210) and a subsequent second activity (e.g., 220) (see [link]). Figure 2 The execution of (210, 220) in the above. These activities are performed by the same human operator (e.g., Figure 4 201A) or different human operators (e.g., Figure 5 Method 600 is performed by review system 400 (see 201A and 201Z). Figure 1B )implement.

[0122] In receiving step 610, the review system receives a series of images 700, 701, 702, which visualize the activities performed by a human operator (i.e., the images have captured images of the activities). The series 700 has at least a first image 701 of a first activity 210 interacting with device 301A, and a second image 702 of a second activity 220 interacting with second devices 301B, 301C.

[0123] By applying image processing to the first image 701 and the second image 702, the review system performs separation step 620. Within the first image 701 and the second image 702, the review system separates at least the first regions 701x and 702x and the second regions 701y and 702y. The first regions 701x and 702x include biometric image data indicating a specific human operator 201A within a group of human operators 201 belonging to the manufacturing facility 100. The second regions 701y and 702y include indications and technical equipment 301A, 301B, and 301C (see...). Figures 4-5 (Image data of interaction).

[0124] By applying image processing to the first region 701x of the first image 701 and the first region 702x of the second image 702, the review system performs the identifier obtaining step 630. It obtains the identifiers ALPHA, BETA of the human operators 201 among the plurality of human operators. The identifiers thus indicate the performance of the first activity and of the second activity by a particular human operator (as Figure 4 In the example of figure 7, ALPHA performs both activities; as Figure 5 In the example of figure 7, ALPHA performs the first activity and BETA performs the second activity.

[0125] By applying image processing to the first region 701x of the first image 701 and the first region 702x of the second image 702, the review system performs the biometric image data removing step 640. The biometric data indicates the particular human operator from the first region 701x of the first image 701 (ID region) and from the first region 702x of the second image 702. The first region 701x of the first image 701 is modified into a modified first region 701xx of the first image 701 and the first region 702x of the second image 702 is modified into a modified first region 702xx of the second image 702. The second regions (activity regions) remain unchanged.

[0126] In the combining step 650, the review system combines the modified first region 701xx of the first image 701 with the second region (701y) of the first image 701 into a modified first image 701’ and the modified first region 702xx of the second image 702 with the second region 702y of the second image into a modified second image 702’.

[0127] In the providing step 660, the review system provides a review record 500 by storing the modified first image 701’, the modified second image 702’ with the corresponding identifiers (i.e. the identifiers for the first and second images). The review record 500 does not necessarily indicate whether the standard is met or not.

[0128] Figure 6 A computer program or computer program product is also shown. The computer program product, when loaded into the memory of a computer and executed by at least one processor of the computer, performs the steps of the computer-implemented method. In other words, the computer program product comprises program code which, when executed by the computer, causes the computer to perform the steps of the computer-implemented method. Figure 6 The blocks in the figure 8 show the modules of a computer as part of the review system 400. In more detail, the computer program comprises an image receiver module, an image separator module, an identifier obtainer module, a biometric data removing module, an image region combiner module and a record provider module.

[0129] ■Multiple assessors

[0130] Although Figure 1A And Figure 1B In the illustration, single facility and single assessor are simplified, but note that a single facility can manufacture products for multiple customers and thus also for multiple assessors. Data segregation is important: the review data for the first assessor must be separated from the review data for the second assessor. The present description explains how to use production control signals (in coordination) to separate the records.

[0131] ■Embodiment with batch separation

[0132] Figure 7 The execution of the method steps is illustrated for different production batches (or "fills"). The steps are illustrated by rightward large arrows with text.

[0133] The production management system 410 is a production management system (e.g. distributed control system DCS, process control system PCS) that is communicatively coupled to the review system 400 (see Figure 1B ). The production management system 410 sends triggers (indicated by vertical arrows Tr1 to Tr5) to the review system 400 (and / or to the camera). Creating an interface between the systems 400 / 410 for the triggers is within the expertise of the skilled person.

[0134] In the simplified illustration, it is assumed that there is one manufacturing process that distinguishes two separate production batches (or fills). Certain activities (human-machine interaction) apply to both batches, e.g. activity 210.

[0135] In the method 600, the receiving step 610 can be triggered for each image by the production management system 410 that provides the identification of the specific batch. As a result, in the step 660 of providing the review records 500, the review records 500#1, 500#2 are related to the specific batch.

[0136] In the example of Figure 7 , the production management system 410 sends the triggers Tr1 to Tr5 continuously. By sending Tr1, the production management system 410 causes the system 400 to continuously receive images 701 (first activity, camera 310-1), e.g. each time the operator enters a lock. For convenience, Figure 7 Only one image is shown. The image (in modified form, see Figures 4-6 ) will go into the review records 500#1 and 500#2 for batch #1 and batch #2. This activity applies to all batches. There is no need to hide (modify) the image, because the first activity is the same for all batches.

[0137] The trigger Tr2 causes the production management system 410 to process the image 702-1 (second activity) into an inspection record 500#1 for batch #1; the trigger Tr3 causes the production management system 410 to process the image 702-2 into an inspection record 500#2 for batch #2; the trigger Tr4 causes the production management system 410 to process the image 702-3 into an inspection record 500#1; the trigger Tr5 causes the production management system 410 to process the image 702-4 into an inspection record 500#2, and so on.

[0138] More generally, the reception 610 of the series of images is triggered by the production management system providing the identification of a first batch #1 and a second batch #2, wherein the first image 701 of the first activity 210 is received in relation with both batches #1 and #2, wherein the second image 702 of the second activity 220 is received in a batch-specific version: second image 702-1, 702-3 for the first batch and second image 702-2, 702-4 for the second batch. The provision of the inspection records comprises the provision of separate inspection records 500#1, 500#2 for the first batch and for the second batch.

[0139] ■Embodiment with parameters

[0140] Figure 8A 、 Figure 8B and Figure 8C Figures illustrate images showing a representation of a device with parameters 351, 352, 353. These parameters are device parameters.

[0141] In the example of Figure 8A , the device is represented by a glove dispenser (301A) with a label. The label is captured by the image 701 (partially displayed). The label indicates the consumption expiration date as a parameter 351 (e.g. "Valid until 12 / 2018").

[0142] In the example of Figure 8B , the image 702 has a metadata attachment (702-DATA) to indicate a parameter 352 of the reactor tank (e.g. a specific temperature 50°C inside the tank).

[0143] In the example of Figure 8C , a parameter 353 is embedded into the image (here for the temperature).

[0144] The evaluator can be interested in technical parameters. For the evaluator, this information is available in relation with the checking of the compliance with the standards.

[0145] The parameters can be available at the device interface, e.g. as a communication device that can be part of the device (e.g. a data output, an interface for an industrial data bus, an interface for a computer network, etc.). Such parameters can include measured values for the device (e.g. from sensors, meters, etc. for pressure, temperature, ingredient amounts, etc.). The parameters 351, 353 can be color-coded parameters.

[0146] Figure 8A , Figure 8B and Figure 8C Examples of the above can conveniently be used to further illustrate the batch separation (see Figure 7 ). Just to give an example, the activity 220 can require that the ingredient is type 405 flour in the first batch and type 550 flour in the second batch. This type of parameter is typically printed on the bag (see parameter 351 in Figure 8A ). Different flour types can represent the above mentioned set of specific requirements for a specific production batch (e.g. batches of different flour).

[0147] The way the parameters are communicated (in the image Figure 8A , in the metadata Figure 8B or embedded Figure 8C ) is not important. In terms of the method 600, the receiving 610 can comprise receiving technical parameters 351, 352, 353. The parameters are associated with the image as part of the image (figs. 8a, 8c) or as part of the metadata attachment (702-data) to the image (see Figure 8B ).

[0148] Since the parameters are technical parameters, they are not related to biometric data. Therefore, the image part with the parameters belongs to the second area (or metadata) that is substantially invariant to processing. The position where the parameters are embedded (if they are embedded in the image) is expected to be sufficiently far from the ID area to prevent accidental removal of the embedded parameters.

[0149] ■Embodiments with multiple cameras

[0150] Figure 10 Embodiments with multiple cameras sending images to the computer system are shown. The figure shows images 701-C1 and 701-C2 taken simultaneously by two cameras. One image shows the operator in a frontal view, while the other image shows the operator in a side view. The method 600 can be applied to both images, so that the review record 500 (see Figures 4-5 ) is augmented by images from different directions. The multiple camera approach can also be used with other cameras in the facility. In other words, the image (to be processed, to be provided to obtain the record) can be a collection of multiple images from multiple cameras.

[0151] In view of the method 600, the receiving 610 can comprise receiving a plurality of images taken from different directions, wherein the images are taken simultaneously. As used herein, "simultaneously" is understood as a time frame in which the operator performs the activity.

[0152] Not all method steps have to be applied to both images. The front view in image 701-C1 can be more suitable for obtaining identifiers (ALPHA, BETA, see step 630), while the side view in image 701-C2 can be more suitable for displaying the activity. The need to perform method steps differs: image 701-C1 is being processed, including obtaining identifiers; while image 701-C2 is only partially processed (possibly not requiring obtaining identifiers at all). This difference can provide higher computational efficiency for the system 400 performing the method.

[0153] ■Embodiment to optimize optical resolution

[0154] Figure 10 Embodiments are shown with cameras sending images to the computer system 400 (see Figure 1B ) can be optimized by assigning relatively higher resolution (i.e. pixels per area) to objects with relatively higher relevance to the activity. The choice of resolution depends on multiple factors, including the ability of the evaluator to view the actions performed by the operator, and / or the ability of the computer to recognize the content, for example by optical character recognition (OCR) to recognize whether a computer (e.g. system 800) is used to identify parameters (see Figure 8A 、 8C ).

[0155] The skilled person can install a camera with suitable optics. For example, the focal length of the camera lens can be chosen such that one image includes the human operator (at least with a biological feature) and the technical equipment.

[0156] Depending on the optical elements of the camera, the scaling of real-world objects (i.e. the operator and equipment, as described above) to pixels on the camera sensor can be distinguished by the optical elements of the camera lens. In the following, camera 310-1 (camera in the clean room lock) is exemplified.

[0157] The normal image 701a (left side of the figure) is the image 701 as explained above in Figures 4-5 The scaling from real-world objects to pixels is essentially constant throughout the image. In other words, the proportions are constant. A person is depicted with a head-to-body ratio corresponding to reality, a square is still a square, and so on.

[0158] The scaled modified image 701b is captured with a special optical lens that magnifies the area of interest, here the face, the hands and the glove dispenser (see device 301A). The magnification is an optical magnification, so that the area of interest is captured with a relatively large number of pixels. In other words, the pixel resolution is increased. On the other hand, areas of less interest, such as the operator's feet, can be captured (if at all) with a relatively small number of pixels.

[0159] As the areas of different interest can be identified in advance, the optical magnification can be set accordingly by the person skilled in the art. Depending on the type of activity, different parts of the body can be monitored. For example, if the standardized activity requires the operator to wear special shoes for clean room use, the operator's feet can indeed be monitored. In this case, the magnification will be highest for the foot area.

[0160] The scaled modified image 701b looks distorted, but this distortion is irrelevant for the computer. The degree of distortion is limited by the technical ability of the review system to obtain (step 630) the identifiers (ALPHA, BETA) and the cognitive ability of the evaluator to view the activity (by inspecting the review record 500). In other words, it does not matter whether the operator and / or the device appear on the image in a disturbed form. Since the evaluator knows where to look, the burden is low.

[0161] The review computer 400 can use either version (normal image 701a or scaled modified image 701b) to perform the processing steps (i.e. separation 620, obtaining 630, combining 650). Using the scaled modified version 701b can have the advantage of showing more detail. For example, the version 701b can be more suitable than the version 701a for displaying technical parameters (see Figure 8A , Figure 8C ).

[0162] As shown on the right, the scaled modified image 701b can be pre-processed back to a normal image 701c (before, during or after processing).

[0163] It is noted that an image captured with special optics can be used as a modified image (see 701' and 702' in Figure 5 ). For the evaluator, the content is important (such as the added identifiers ALPHA, BETA, etc., or the technical parameters), not the shape of the operator or the shape of the device.

[0164] The special optical element can be implemented by a wide-angle lens, the lens angle being large enough to capture both the operator and the equipment. The wide-angle lens can be implemented as a so-called fish-eye lens (or frog-eye lens). These lenses are known in the art for several decades (in the patent literature, for example in GB 225,398 and US 2,247,068 A).

[0165] In other words, the image provided by the optical device is such that some parts appear to be enlarged, while some parts appear to be reduced. Since the activity is performed repeatedly (see Figure 2 ), there is no need to re-adjust the optical element for each image.

[0166] ■Embodiment where the image is part of a video

[0167] Figure 11 A series of multiple images taken by a camera, for example camera 310-1 (associated with the clean room lock) and camera 310-2 (associated with the tank reactor), is shown. Since the series 700 (of images) can be stored as a video, it is possible to provide a review record 500 in which image 701' is part of a first sub-series (showing a first activity) and image 701' is part of a second sub-series (showing a second activity).

[0168] It is noted that for a review record 500 with a video, the removal (step 640) of biometric data must be performed for all images of the video containing such data. However, obtaining the identifier (step 630, potentially more computationally intensive) would have to be performed only for (at least) one image.

[0169] Figure 11 The example shows only some video frames. They are numbered from 1 to 11. The figure is greatly simplified in terms of number of images (or video frames). For example, using a video with 24 image frames per second, and assuming that the gloves are put on for 2 seconds, 48 frames would show the activity (i.e. activity 210, see Figure 2 ).

[0170] Assuming synchronization (1st frame of both cameras having the same timestamp, same inter-frame time distance), the 3rd frame of camera 310-1 would correspond to image 701 (received by system 400), and the 4th-6th frames of camera 310-2 would correspond to image 702 (also received by system 400).

[0171] Establishing the series 700 (of first image 701 and second image 702) can be performed after the video is taken (assuming the video is stored), and the two images can be correlated by the operator identification (e.g. ALPHA).

[0172] It is also possible to check whether a particular image does not comply with the criteria and identify the subsequent activity. For example, in the 9th frame of the lock camera, the operator forgets to put on gloves. He or she can be identified (i.e. BETA) and the image of the BETA (e.g. a frame from the reactor tank camera) in the subsequent activity can be used to identify the non-compliance (see requirement (B), which can be identified from a single frame in this case).

[0173] The description continues discussing other aspects related to the general figures.

[0174] There is a risk that the operator or other people (in the facility 100) interact with the review system 400 to manipulate the report. The system 400 can be protected by security measures. It is also possible to add graphical data to the (modified) images in the review record 500 by techniques such as watermarking to identify the system 400 (as a processing entity).

[0175] From a broader perspective and business-related considerations, the evaluator 203 (in Figure 1B ) can work for a customer. To remain in the chemical industry, the customer can be a different industrial company, but also a retailer. This customer will receive the review record 500. There are many ways to deliver such data, for example by using a computer portal (at the review system 400) to which the customer has access. The customer can operate an evaluation system 800 that helps perform the evaluation.

[0176] In this sense, the customer can (at least virtually) view the facility, but with great differences. Unlike the reviewer 202, the customer will see the data of interest (related to the evaluation criteria compliance, but not the information that must be kept within the facility, see above). Again, the inclusion / exclusion dilemma is solved.

[0177] ■Additional embodiments

[0178] The description now investigates potential further limitations and ways to solve them.

[0179] ■Operator on the image

[0180] The first constraint relates to the applicability of the image to the operator.

[0181] As shown, the clean room operator wears gloves. But in many industries, the operator also wears special clothing. The clothing can include some kind of headgear that covers the hair so that the hair does not appear on the image. Finally, the operator can wear a mask (as a surgeon) so that the face is only fully shown on the image.

[0182] However, it is not necessary to use complex face recognition techniques applicable to identify the face (e.g. in social networks) to obtain the identifier (ALPHA, BETA).

[0183] Since the number of operators within the facility is limited (e.g. 1000 operators), artificial intelligence can be used to train a computer to recognize only the faces of these operators. For example, the operators can be photographed (wearing work equipment) and an identifier (i.e. from ALPHA_1, ALPHA_2... ALPHA_1000) can be used for training (supervised training).

[0184] In terms of the method 600, the biometric image data comprises data related to the face of a human operator, and in the step of obtaining 630 an identifier, an image of a predefined set of operators is used as a reference. This set of operators can comprise operators associated with the facility (e.g. its employees) or operators associated with a particular part of the facility (e.g. operators working in a particular sub-part of the facility) or operators associated with a particular equipment (e.g. operators operating a reactor tank) or operators working in the facility at a particular time interval (e.g. during a particular shift of a product batch being produced).

[0185] Some operators wear protective glasses (or generally goggles), but the glasses are clear enough for face recognition to be possible.

[0186] ■Technical measures for isolating the image data

[0187] The second limitation relates to protecting the data from accidental misuse. The series of images 700 (received by the review system from the camera) contain information that can allow a viewer to identify a particular person.

[0188] In order to limit the risk of such information accidentally leaving the facility, the method steps receiving 610, separating 620, obtaining 630 and removing 640 can be performed by a processor of a computer system 400 that uses only volatile memory. In other words, as long as the ID areas 701x, 702x (see Figures 4-5 ) have not been removed, the image data remains in the random access memory RAM of the computer processor. The combining 650 (i.e. by accessing the modified areas 701xx, 702xx) can be performed in non-volatile storage (e.g. hard drive storage, solid state drive storage). From a different perspective, the technical measure of intercepting data from memory is more complex than the technical measure of intercepting data from storage, which exploits this difference.

[0189] In view of the above-mentioned batch separation (see Figure 7 ), the use of volatile memory further reduces the potential risk of cross-feeding information from one batch to another.

[0190] However, these steps can be performed in non-volatile memory (i.e. with a database) and separated in batches at a later point in time (when the second image is taken).

[0191] ■ Combined embodiments

[0192] As mentioned above, the camera can be implemented in different ways, and the present description has explained two main directions:

[0193] • by the number of images taken simultaneously to obtain images from different directions (see Figure 9 ),

[0194] • by the difference between the optical elements of the camera and the field of view area to select the appropriate area-related image resolution (see Figure 10 )

[0195] Further implementations are possible, for example using a camera with a zoom lens. The camera can be controlled according to signals obtained from the production management system 410. In this case, it is even possible to differentiate between batches. As Figure 7 shown, the review record 500#1 for the first batch can use different camera settings than the review record 500#2 for the second batch.

[0196] The camera settings (direction, movement, e.g. zoom, pan, tilt) can be obtained as predefined settings. The predefined settings can be provided by the production management system (410 in Figure 1B ). The production activities for different batches can be recorded differently, either by a record with still images or by a record with video.

[0197] It is also considered to post-process the images, for example by applying a digital zoom when presenting the review record 500 to the evaluator 203.

[0198] ■Detecting conformity at an early stage

[0199] The recording can be simplified by storing data sets representing activities performed by the operators. As an example, the simplified recording can include the following data sets: Figure 2

[0200] • ALPHA's first activity "put on gloves" at 09:00,

[0201] • ALPHA's second activity "add ingredients" at 09:10,

[0202] • ALPHA's first activity "put on gloves" at 10:10, and so on.

[0203] ​In such embodiments, it can not be necessary to store the (modified) image. The simplified record can serve as basis for the assessment. However, if non-compliance becomes possible, the simplified record can be used outside the review context to alert the operator.

[0204] ■Manufacturing-centric activities

[0205] The review system 400 provides review records 500 for activities in the facility. These activities can include pre-manufacturing activities (e.g. activity 210 "put on gloves"), manufacturing activities (e.g. activity 220 "add ingredient") and post-manufacturing activities (e.g. clean equipment after use).

[0206] ■Location of the review system

[0207] In Figure 1B , the review system 400 is shown as belonging to the facility 100 (see the large rectangle). However, the review system 400 does not have to be physically implemented on the premises of the facility, it can be implemented by a computer system remote from the facility.

[0208] ■Real-time aspects

[0209] Although the review system 400 provides results as records (i.e. data structures to be stored), some real-time or contemporaneous aspects can also be considered. For example, the trigger to start receiving images (or a series of images) can also come from another computer system, e.g. from the assessment system 800 (see Figure 1B ). The assessor can decide to monitor production from a specific point in time (e.g. when a specific batch is produced).

[0210] ■Implementation details

[0211] The camera is communicatively coupled to the review system. For simplicity, Figure 1B The connection is not shown, but suitable connections are known in the art (e.g. via a network).

[0212] Since image processing is known in the art, the skilled person can select computer routines and code from libraries. For example, for the separation 620, the computer system can use face recognition techniques known from hand-held consumer cameras, in the sense that it is recognized that a face is present on the image, not in the sense that the person with that face is detected. Although the mentioned cameras typically frame the face with a rectangular box (on the viewer screen of the camera), it is not necessary to show such a box or the like to the user. No user interaction is required at this stage.

[0213] Face recognition is known in the art from many publications, including the following publications:

[0214] • Qin Zhou, Heng Fan, Hang Su, Hua Yang, Shibao Zheng, Haibin Ling: "Weighted Bilinear Coding over Salient Body Parts for Person Re-identification", Pattern Recognition Letters, preprint "arXiv:1807.00975" by Cornell University 2018, explains the re-identification of persons.

[0215] • Allyson Rice, P. Jonathon Phillips, Vaidehi Natu, Xiaobo An, and Alice J. O'Toole: "Unaware Person Recognition From the Body When Face Identification Fails", Psychological Science 24(11) 2235-2243, provides guidance on how to recognize persons when face identification fails.

[0216] It can be expected that the operator 201 wears a helmet such as an industrial safety helmet (see DIN EN 397:2013-04). Since the helmet can cover certain parts of the face, the images can be processed in addition to the face images also on the basis of other biometric features such as the shoulder and / or the height.

[0217] It is to be noted that once the ID region with biometric data is identified (see Figures 4-5 701x, 702x), other regions with activity can also be identified simply by subtraction (activity region = image minus ID region). No specific processing needs to be provided to identify specific activities.

[0218] To ensure that the images 701, 702 are taken at subsequent points in time, a time stamp can be added. In other words, the receiving 610 can comprise receiving a series 700 of images 701, 702 with time stamps of the images 701, 702. In principle, there are two options.

[0219] • In an embodiment, the time stamp is added by the camera (assuming the camera has a synchronized clock).

[0220] • It is also possible to add the time stamp when the image is received by the review system.

[0221] The signal propagation delay between the camera and the review system can be negligible, as the subsequent activity is performed by a human operator in a time interval (from the first activity to the second activity) that is greater (i.e. longer) than the signal propagation time (from the camera to the system). Time stamps of images are known in the art. They can be embedded into the image (graphically) and can be transmitted as metadata.

[0222] These activities are industrial activities that have the following properties: (i) they are often repeated and (ii) they are standardized. Continuing the example explained above, when putting on gloves (activity 210, on image 701), the operator moves his / her hand near the glove dispenser, but the head does not move much. When adding an ingredient (activity 220, on image 702), the operator moves the hand (more than the feet) and can look inside the reactor. In both cases, for most activities, the viewing direction is more or less predefined (i.e. in the direction of the dispenser, inside the reactor). The ID area on the image that the camera (when positioned) has is more or less the same (repeated) for all images.

[0223] Taking into account that different pixels change differently over time, a dynamic way can be implemented to distinguish the areas 701x, 702x and 701y, 702y. For example, when the operator (in the facility) performs an activity, he or she moves the hand more than the feet (for example). Calculating these differences (for example, between images taken a fraction of a second apart) and identifying the potential areas should be enough.

[0224] It is to be noted that here also auto-focus techniques (known from digital cameras) can be used.

[0225] ■Calibration biometric data

[0226] As explained above, in the step of acquiring the identifier, an image of an operator 201 from a specific group of operators is taken as a reference. The specific group of operators 201 can be defined by a variety of cases, for example: operators working in the facility, operators working in the facility during a specific shift, operators working in the facility using a specific equipment (for example, operators of reactor tanks are different from operators or drivers of vehicles). As the appearance of an operator changes from time to time (for example, different hair styles on different days; beard style), the image can be taken at appropriate points in time (for example, periodically).

[0227] ■Non-biometric image data treated like biometric image data

[0228] Operators can carry their name on a lanyard, a badge, an embroidery on their clothing, etc. The name is usually written in a color that contrasts sharply with the color of the clothing (for example, black on white). This enables co-workers to familiarize each other.

[0229] Optical character recognition technology (OCR) allows a computer to convert an image with letters into data. It is also expected that such OCR reading will be used to acquire (step 630) the identifier (e.g. ALPHA, BETA). The area of the image with the name will be considered as the area with biometric image data (i.e. ID area) and can also be removed (step 640). This approach can even be mandatory as the name (of the operator) is personal data (as mentioned above) and is to be masked to the assessor. It is to be noted that OCR processing of the name helps to distinguish between operators that look very similar (e.g. in the extreme case of identical twins).

[0230] ■Assessment computer

[0231] Figure 1B It is also illustrated an assessment system 800. This computer system can be used as a user interface for the assessor 203. As mentioned above, the identifier (ALPHA, BETA) can be embedded into the (modified) images 701', 702', the same applies to the parameters 351, 353 (see Figure 8A , 8C ). Image viewing software is typically available on computers. It is also expected that the computer file formats allowing image storage (*.jpg, *.bmp, *.tiff, *.pdf, *.png) corresponding to the potential requirements of the archival record will remain available for the next decades.

[0232] Once the review record 500 is available, the review system 400 can push the review record 500 to the assessment system 800. It is possible to notify the assessor that the record is available (by various techniques, e.g. email, short message service SMS, instant messaging, e.g. WHATSAPP(TM), etc.).

[0233] Although the assessment system 800 is illustrated as a single computer, the review system 400 can send the review record to different computers depending on the batch or depending on other circumstances.

[0234] The systems 400 and 800 can be communicatively coupled by internet technology, with / without access technology through a website, a web portal, a relationship management system, a database system, etc.

[0235] The review record 400 can be enhanced by process documentation (e.g. of the process being performed at the time of receiving the image) and / or by documentation indicating the facility details (e.g. a map or a plan of the facility). Such additional files can help the assessor.

[0236] ■Other use scenarios

[0237] For example, a standard can require that a transport device (e.g. a fork lift moving luggage) moves at a speed below a speed limit (e.g. 10 km / h). However, it can be allowed to exceed the limit for a short time (e.g. 12 km / h for 2 seconds). By taking timestamped images 701 and 702 at two different locations (with a known distance), the average speed can be calculated.

[0238] ■Location stamp

[0239] It can be required that an operator must perform certain activities within a predefined location. For example, a bag of opened flour should be poured into a nearby tank reactor, but must not be carried through the facility. To make the evaluation as easy as possible, images can optionally be associated with a location within the facility with a predefined granularity (e.g. identifying a specific device, a specific room within the facility, specific coordinates, a specific production line, etc.). In other words, images can come with a location stamp.

[0240] ■General computer system

[0241] Embodiments of the invention can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The invention can be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier (e.g., in a machine-readable storage device), for execution by, or to control the operation of, data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). The computer program can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network. Each of the described methods can be performed by a corresponding computer program product on each of the respective devices, e.g. the first and second computers, the trusted computer and the communication unit.

[0242] The method steps of the invention can be performed by one or more programmable processors executing a computer program to perform the functions of the invention by operating on input data and generating output. The method steps can also be performed by, and the invention can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0243] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Such storage devices can also be removable. A computer can also receive data and instructions from or transfer data and instructions to, or both, a communications network coupled to the computer. A communications network can comprise any one or more networks or combinations of networks selected from a local area network (LAN) and a wide area network (WAN), e.g., the Internet, and telecommunications, mobile or cellular networks. A network can operate according to one or more protocols. A network can also be implemented as part of a cloud of computers. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0244] To provide for interaction with a user, the application can be implemented on a computer having a display device, e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, for displaying information to the user and a input device, e.g., keyboard, touch screen, or touch pad, a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0245] The application can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the application, or any combination of such back end, middleware, or front end components. The client computer can also be a mobile device, e.g., a smart phone, tablet, or any other hand held or wearable computing device. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet or an intranet, or a wireless network or a telecommunication network.

[0246] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0247] ■Reference signs

[0248] 100 Facility

[0249] 200 Person

[0250] 201 Operator (A...Z)

[0251] 202 Reviewer

[0252] 203 Assessor

[0253] 300 Equipment

[0254] 301 Equipment (A...Z)

[0255] 310 Camera (-1, -2)

[0256] 399 Glove

[0257] 400 Review System

[0258] 402 Review Report

[0259] 410 Production Management System

[0260] 450 Standard

[0261] 600 Method

[0262] 6x0 Method Step

[0263] 700 Series of Images

[0264] 701, 702 Image (-x, -xx, -y, etc.)

[0265] 800 Assessment System

[0266] Tr1...Tr5 Trigger

Claims

1. A computer-implemented method (600) for providing review records (500), said review records relating to activities (210, 220) performed by human operators interacting with technical equipment within a facility (100), wherein, The review record (500) indicates the execution of an activity sequence (210, 220) having a first activity (210) and a subsequent second activity (220) by the same human operator or by different human operators, the method (600) comprising: Receive (610) a series (700) of images (701, 702) that visualize activities (210, 220) performed by a human operator, the series (700) having at least a first image (701) of a first activity (210) interacting with a first technical device and a second image (702) of a second activity (220) interacting with a second technical device; By applying image processing to the first image (701) and the second image (702), at least a first region (701x, 702x) and a second region (701y, 702y) are separated (620) within the first image (701) and within the second image (702), wherein the first region (701x, 702x) includes biometric image data indicating a specific human operator within a group of human operators belonging to the facility (100), and wherein the second region (701y, 702y) includes image data indicating interaction with the technical equipment; By applying image processing to the first region (701x) of the first image (701) and the first region (702x) of the second image (702), (630) identifiers (ALPHA, BETA) of human operators among a plurality of human operators are obtained, the identifiers (ALPHA, BETA) thereby indicating the execution of the first activity and the second activity by a specific human operator; By applying image processing to the first region (701x) of the first image (701) and the first region (702x) of the second image (702), biometric image data indicating a specific human operator is removed (640) from the first region (701x) of the first image (701) and the first region (702x) of the second image (702), such that the first region (701x) of the first image (701) is modified to the modified first region (701xx) of the first image (701), and the first region (702x) of the second image (702) is modified to the modified first region (702xx) of the second image (702); The modified first region (701xx) of the first image is combined (650) with the second region (701y) of the first image (701) to form a modified first image (701'), and the modified first region (702xx) of the second image (702) is combined with the second region (702y) of the second image (702) to form a modified second image (702'); and The review record (500) is provided by storing a modified first image (701') and a modified second image (702') with corresponding identifiers (ALPHA, BETA).

2. The method (600) according to claim 1, wherein, The steps of receiving (610), separating (620), obtaining (630), and removing (640) are performed by the processor of a computer system (400) that uses only volatile memory.

3. The method (600) according to any one of the preceding claims, wherein, Receiving (610) includes receiving a series of (700) images (701, 702) having timestamps of the images (701, 702).

4. The method (600) according to any one of claims 1-2, wherein, Receiving (610) includes receiving the series of (700) images (701, 702) by receiving the first image (701) from the first camera (310-1) and by receiving the second image (702) from the second camera (310-2), wherein the first camera and the second camera are associated with a first technical device and a second technical device, respectively.

5. The method (600) according to any one of claims 1-2, wherein, Receiving (610) includes receiving technical parameters (351, 352, 353) associated with the technical device, wherein the technical parameters are received as part of the image or as part of the image's metadata attachment (702-DATA) and associated with the image.

6. The method (600) according to claim 5, wherein, Receiving (610) includes receiving technical parameters (351, 352, 353) related to the technical device from the device interface, wherein the technical parameters are related to the first image (701) or the second image (702).

7. The method (600) according to any one of claims 1-2, wherein, The receipt (610) of the series (700) images (701, 702) is triggered by the production management system for a specific image, the production management system providing an identifier for a specific batch, such that, as a result of providing the review record (500), the review record (500#1, 500#2) is associated with the specific batch.

8. The method (600) according to claim 7, wherein, Receiving (610) the series (700) images is triggered by a production management system that provides identifiers for a first batch (#1) and a second batch (#2), wherein receiving the first image (701) of the first activity (210) is associated with the two batches (#1, #2), wherein receiving the second image (702) of the second activity (220) is performed in a batch-specific version for the second image of the first batch and the second image of the second batch, such that providing the review record includes providing separate review records for the first batch and for the second batch.

9. The method (600) according to any one of claims 1-2, wherein, The biometric image data includes data related to the human operator's face.

10. The method (600) according to claim 9, wherein, The biometric image data includes data related to the human operator's face, and wherein, in the step of obtaining the (630) identifier, an image of an operator (201) from a specific group of operators is used as a reference.

11. A computer program product, when loaded into the memory of a computer and executed by at least one processor of the computer, performs the steps of the computer-implemented method according to any one of claims 1-10.

12. A review computer system (400) adapted to perform the method (600) according to any one of claims 1-10.

Images