Computer-implemented method, system and computer program for providing review records related to technical installation

Abstract

A human operator (201A) performs an activity within a facility that interacts with the technical equipment (301A, 301B, 301C). A series (700) of images (701, 702) visualizes activities performed by the same human operator (201A) or different human operators. In the image, the computer separates a region (701x, 702x) including biometric image data from a region (701y, 702y) having data indicative of the interaction. The computer obtains an identifier (ALPHA, ALPHA) of the operator (201A) but removes the biometric image. The computer then combines the image regions and provides a review record (500) by storing the modified first and second images (701 ', 701') and the identifier.

Description

[0001] This application is a divisional application of the original application filed with the China Patent Office on July 16, 2021, with the application number 202080009629.6 and the title “Computer- implemented method, system and computer program for providing an audit trail related to technical equipment”. TECHNICAL FIELD

[0002] The present disclosure relates generally to industrial processes; and more specifically, the present disclosure relates to image processing for providing an audit trail using a computer-implemented method, computer system or computer program for image processing, wherein the trail is related to an industrial product handling process. BACKGROUND

[0003] Handling products in facilities such as factories, plants, assembly lines, warehouses, laboratories, etc. is traditionally a task of human operators interacting with technical equipment such as machines, tools, gadgets, vehicles, etc. Automation does not change this. Human activity is still part of the product handling process.

[0004] Product handling in the broadest sense includes manufacturing products, testing or analyzing products, storing products, distributing products, or even destroying products. The term “product” as used herein applies to all stages of the product life cycle and includes: product components, intermediate products, and samples extracted from production.

[0005] Furthermore, there can be discrete things (e.g. a product is a bottle filled with a beverage) and continuous streams of matter (e.g. supplying water for a beverage through a pipe).

[0006] The interaction between humans and technical equipment (human-device interaction) creates risks. One risk is related to product quality. For example, at the end of a production line, a product can look like it can be shipped to a consumer, but in fact the product can have been damaged beyond repair. The risks can be industry specific. Human operators can influence product performance by applying incorrect parameters (e.g. amount of material, time settings, pressure or temperature settings, etc.). For example, in the general chemical industry and its branches (e.g. fragrances, pharmaceuticals, human and animal nutrition, health industry), specifically, a human operator can accidentally come into contact with a product during production, rendering the product useless. As a result, the product can be contaminated with bacteria, making it possible for an end consumer to contract a disease. In a laboratory, a test sample can be prepared or analyzed incorrectly due to mishandling.

[0007] Additional risks include safety hazards for the operators themselves.

[0008] To mitigate such risks and other risks, product handling needs to comply with requirements summarized by a large number of technical standards. Such requirements relate to proper operating instructions for operators, parameters of the operating facilities and / or equipment, procedures to track production by batch or load, cleaning of equipment (before or after use), safety precautions, etc.

[0009] Just to name a few introductory examples, in the context of the chemical industry: a standard can require personnel to adhere to predefined procedures and create records, e.g. batch records, cleaning protocols, testing protocols, etc.; a standard can require operators to wear appropriate protective gloves (or generally protective clothing) in order to minimize the risk of coming into contact with the product; a standard can require materials to be delivered via a parameterized supply chain; a standard can specify air quality within a clean room; and a standard can require production to be tracked by computer, by load, by batch, etc.

[0010] Just to name a few examples, for one of the numerous standards for industrial product handling, the following is mentioned:

[0011] • COMMISSION DIRECTIVE 2003 / 94 / EC of 8 October 2003 laying down principles and guidelines of good manufacturing practice for medicinal products for human use and investigational medicinal products, Official Journal of the European Union, 14 October 2003, L 262 / 22,

[0012] • ISO 9001 :2015-09 Quality management systems - Requirements,

[0013] • ISO 14001 :2015-09 Environmental management systems - Requirements and guidance for use,

[0014] • Food Safety System Certification (FSSC) 22000, available from FSSC 22000 Foundation,

[0015] • Food safety in the production process systematic prevention methods against biological, chemical and physical hazards, known as "Hazard Analysis and Critical Control Points (HACCP)".

[0016] A standard can refer to another standard. As used herein, the term "standard" also includes any collection of specific requirements for a specific production batch.

[0017] Despite standardization, the interaction between human operators and equipment is not free from errors or malfunctions. Therefore, it is necessary from time to time to monitor and check whether the product handling complies with the standards. Traditionally, this is the task of so-called auditors (sometimes referred to as "inspectors" or "checkers"). The auditors have to distinguish between information relevant for the assessment of compliance with the standards (i.e. information relevant to the requirements to be included in the report) and information that must not leave the facility (i.e. information to be excluded from any report).

[0018] As of today, the audit cannot be automated. However, it is considered to physically station the auditors at a location remote from the facility. US 2016 / 0306172 A1 explains the use of a wearable camera device to transfer data from the facility to an auditor located outside. Since the data is sent in real time, the problem of selectively including / excluding information remains.

[0019] There is a need to provide a technique that solves the inclusion / exclusion dilemma. SUMMARY

[0020] For compliance with the technical standards, it is often required that subsequent product handling activities are performed by the same operator. However, it is irrelevant who this operator is. More specifically, the personal data of the operator are irrelevant (personal data are any information related to an identified or identifiable operator as a natural person by reference to such as name, identification number, location data, online identifier, or by reference to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person).

[0021] The present invention solves the inclusion / exclusion dilemma by using image processing. According to an embodiment of the invention, an audit system receives images from a camera located within the facility. The images (i.e. pictures or photos in electronic form, with pixels) show the operator performing a predefined activity. The audit system processes the biometric image data to obtain an identifier of the operator, but then ignores or removes the biometric image data. The audit system then provides an audit record with a modified image incorporating the identifier. Due to the modification, no personal data remain in the audit record. The identifier cannot be converted back to personal data.

[0022] An assessor checking the audit record (i.e. looking at the identifier in the record) can still identify whether subsequent activities were performed by the same operator.

[0023] This approach exploits the fact that human operators typically perform activities with their hands and that a viewer (of the images) recognizes a particular person by looking at the face rather than at the hands.

[0024] In more detail, the computer-implemented method for providing an audit record involves activities performed by human operators within a facility. An activity is an interaction of a human operator with a technical device. An audit record is indicative of the performance of a sequence of activities having a first activity and a subsequent second activity by the same human operator or by different human operators. The method comprises the following:

[0025] receiving a series of images. The images visualize an activity performed by a human operator. The series has at least a first image of a first activity of interacting with a first technical device and a second image of a second activity of interacting with a second technical device.

[0026] separating at least a first and a second region within the first and the second image by applying image processing to the first and the second image. The first region comprises biometric image data indicative of a particular human operator within a group of human operators, e.g. operators belonging to a manufacturing facility. The second region comprises image data indicative of an interaction with the technical device.

[0027] obtaining an identifier of a human operator of the plurality of human operators by applying image processing to the first region of the first image and to the first region of the second image. The identifier is thereby indicative of the performance of the first activity and the second activity by the particular human operator.

[0028] removing biometric image data indicative of the particular human operator from the first region of the first image and from the first region of the second image by applying image processing to the first region of the first image and to the first region of the second image. The first region of the first image is modified to a modified first region of the first image and the first region of the second image is modified to a modified first region of the second image.

[0029] combining the modified first region of the first image with the second region of the first image to a modified first image and combining the modified first region of the second image with the second region of the second image to a modified second image.

[0030] providing the audit record comprises storing the modified first image and the modified second image together with the identifier.

[0031] Optionally, the receiving, separating, obtaining and removing steps are performed by a processor of a computer system using only volatile memory.

[0032] Optionally, the receiving comprises receiving a series of images having a time stamp of the images.

[0033] Optionally, the receiving comprises receiving the series of images by receiving the first image from a first camera and receiving the second image from a second camera. The first and the second camera are associated with a first technical device and a second technical device, respectively.

[0034] Optionally, the receiving comprises receiving technical parameters related to the technical device. The technical parameters are associated with the images as part of the images or as received in a metadata attachment to the images.

[0035] Optionally, the receiving comprises receiving technical parameters related to the technical device from a device interface. The technical parameters are related to the first image or the second image.

[0036] Optionally, the receiving the series of images is triggered by a production management system providing an identification of a specific batch for a specific image. The review record is related to the specific batch as a result of providing the review record.

[0037] Optionally, the receiving the series of images is triggered by a production management system providing an identification of a first batch and an identification of a second batch. The receiving the first image of the first activity is related to both batches. The receiving the second image of the second activity is performed in batch specific versions for a second image of the first batch and a second image of the second batch. Thus, the providing the review record comprises providing separate review records for the first batch and for the second batch.

[0038] Optionally, the biometric image data comprises data related to a face of the human operator.

[0039] Optionally, the biometric image data comprises data related to a face of the human operator. In the step of obtaining an identifier, an image of an operator in a specific group of operators is used as a reference.

[0040] In one embodiment, a computer program product, when loaded into the memory of a computer and executed by at least one processor of the computer, performs the steps of the computer-implemented method.

[0041] In one embodiment, the review system is a computer system adapted to perform the method according to any of the above examples. Such a system has modules dedicated to the steps of the method. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1A and Figure 1B An overview of a facility with people and with devices is shown, wherein Figure 1A A conventional method is shown, and wherein Figure 1BThe use of a computer system and computer-implemented method is shown;

[0043] Figure 2 An operator interacting with the equipment is shown;

[0044] Figure 3 A case where the criteria are not met in cases (A) to (D) is shown;

[0045] Figure 4 A computer system performing image processing resulting in an audit record in an example with a first scenario is shown;

[0046] Figure 5 A computer system performing image processing resulting in an audit report in an example with a second scenario is shown;

[0047] Figure 6 A flowchart of a computer-implemented method for providing an audit record related to activities performed by a human operator interacting with technical equipment within a facility is shown;

[0048] Figure 7 The method steps are shown being performed separately for different production batches;

[0049] Figure 8A , Figure 8B and Figure 8C An image of the equipment with a representation of the technical parameters is shown;

[0050] Figure 9 An embodiment with multiple cameras sending images to the computer system is shown;

[0051] Figure 10 An embodiment with a modified lens of the camera sending images to the computer system is shown; and

[0052] Figure 11 A series of multiple images taken by the camera is shown. DETAILED DESCRIPTION

[0053] Facility and system overview

[0054] Figure 1A and Figure 1B An overview of a facility 100 with personnel 200 and technical equipment 300 is shown. Figure 1A The conventional approach is suitable for Figure 1B The approach using a computer system 400 and a computer-implemented method 600 (to be explained in detail in Figures 4-6 The computer system 400 will be labeled "audit system" for convenience.

[0055] Very simply, the facility 100 can be a factory, an assembly line, a warehouse, a laboratory, etc. The technical equipment 300 is collectively referred to as equipment 301A, 301B, 301C... 301Z. In the chemical industry, examples are glove dispensers, clean room locks, tank reactors, bags containing material (e.g. ingredients), heaters, ovens, conveyor belts, etc.

[0056] As used herein, the standards apply to situations within the facility 100. The description and the figures relate to a facility 100 and equipment 300 in a clean room example.

[0057] In this example, the standards for a clean room relate to the technical equipment 300, e.g. air conditioning equipment, locks between rooms, but the standards also relate to activities that a human operator must perform (see Figure 2 ).

[0058] As used herein, the personnel 200 can have multiple roles (or functions), including operators 201, auditors 202 and assessors 203.

[0059] The operators 201 perform activities within the manufacturing facility. There are typically multiple operators in a single facility, here distinguished as 201A, 201B, 201C... 201Z. It is not required that all operators are present at the same point in time. Since the operators 201 are humans, each person has specific biological characteristics: a specific face, a specific body shape or build, a specific weight, a specific voice, etc.

[0060] The biological characteristics that humans can recognize by looking at each other are relevant here: these are visual biological characteristics (i.e. face, body shape / build). In the figures, they are represented with different faces. For simplicity, it is assumed that there are no identical twins among the operators.

[0061] The operators 201 perform interactive activities with the equipment 300. There is no one-to-one relationship between a specific operator and a specific equipment. A specific operator 201 (e.g. 201A) can interact with different equipment (e.g. 301A, 301B, 301C) at the same time or at different points in time. All operators can interact with a specific piece of equipment (e.g. 301A), but not at the same time.

[0062] The auditors 202 ( Figure 1A) access the manufacturing facility and review the performance of the actions there. From a content point of view, the review typically includes determining whether and how the operator (in general) performed the activities that interact with the equipment (i.e. equipment components). The way of the review can include visual inspection (i.e. looking at the operator and / or equipment) and oral inspection (i.e. talking to the operator and supervisors, listening to sounds, etc.). As mentioned above, traditionally, the reviewer collects their findings in a review report 402 in writing and / or by taking pictures.

[0063] As in the Figure 1A The assessor 203 examines the review report 402 and assesses it for compliance with the criteria 450. The criteria 450 are here characterized by a document with multiple requirements (A), (B), (C) and (D). The description and the figures use a very simplified example of fictitious criteria (e.g. in a nutrition or pharmaceutical environment) that require the operator to do the following:

[0064] • (A) wear gloves when locking the clean room,

[0065] • (B) wear two pairs of gloves when putting raw material into the reactor,

[0066] • (C) change gloves within an interval (e.g. every 2 hours), and

[0067] • (D) perform these activities by the same operator.

[0068] In this example, the fictitious criteria are written in terms of "positive activities", but it is also possible to use no definition.

[0069] In this example, the criteria 450 focus on the behavior of the operator 201 (i.e. the performance or non-performance of predefined activities), but there can be further requirements (from different criteria) that are only related to the equipment.

[0070] Due to the nature of the review, the reviewer 202 and the assessor 203 are typically not the same person as the operator 201. The functions of the reviewer and the assessor can be performed by the same person. The figures illustrate the assessor 203 as being located outside the facility.

[0071] This does not exclude the possibility that the computer can support the assessment. This will be explained below.

[0072] Figure 1B The figures illustrate ways of having a review system 400 that performs the method 600. Figure 1B The reviewer 202 is shown by a dashed line, as this function is no longer needed. However, this way is not an automation of the review, but a way of generating a review record that allows for an offline and / or remote review of the industrial product handling.

[0073] Cameras 310-1 and 320-2 are associated with the equipment 300 to take images (or "capture images", i.e. data structures representing reality). Real-world objects from which images are taken include the equipment and / or the operators. The review system 400 uses the images and provides review records 500 as data structures.

[0074] In the example of Figure 1B , camera 310-1 is associated with equipment 301A to take images when any operator interacts with the equipment 301A; camera 310-2 is associated with equipment 301B and 301C to take images when any operator interacts with the equipment 301A, 301B or both.

[0075] The operators have no access to the cameras 310, the review system 400 and the review records 500.

[0076] Figure 1B It is also shown that the review system 400 can be coupled to a production management system 410 (see also Figure 7 , production management system) and that the review records 500 can be evaluated by a further computer system, e.g. an evaluation system 800 (which uses standard requirements as rules for evaluating the review records 500).

[0077] Figure 2 The operator 201A is shown to interact at least partially with equipment by way of example. In this example, it is assumed that the facility is a manufacturing facility of the chemical industry. The applicable standard requires that manufacturing is performed at a clean room location. As used herein, activities 210 and 220 are typical activities frequently performed within such a facility. In view of the standard, the performance (or non-performance) is relevant. Activities 210 and 220 can be identified by visual inspection.

[0078] In the simplified example of Figure 2 , at a point in time tl, the operator 201A performs (a first) activity 210 by carefully checking the clean room lock. He washes his hands and puts on gloves 399 from a glove dispenser (here: equipment 301A). From the standard, there are special requirements for the gloves (i.e. time limit to be observed after opening the dispenser, sterile gloves indicated by a specific color, etc.) and for the activity (e.g. wearing them, changing them regularly according to a change interval TINTERVAL, etc.).

[0079] At time point t2, operator 201A is already in the clean room, and operator 201A performs the second activity 220 by opening a bag (or "pocket", device 301B) containing a food ingredient (e.g. sugar or flour) and pouring the food ingredient into a tank reactor (device 301C), performing an "add ingredient" activity. Assume that time points t1 and t2 are within the replacement interval TINTERVAL.

[0080] At time point t3, operator 201A has left the clean room to perform an unrelated activity 230 (e.g. rest, details not shown). Assume that this activity is not relevant to the standard. Note that operator 201A has taken off the gloves. There is no activity to monitor, and also no human-machine interaction (at least no interaction of interest).

[0081] At time point t4, operator 201A re-performs the activity of t1, here the first activity 210* again.

[0082] Briefly looking Figure 1A , the inspector 202 will look at operator 201A and write the following inspection report 402:

[0083] (A) Wearing gloves when locking? Yes

[0084] (B) Wearing both gloves when loading the reactor with ingredients? Yes

[0085] (C) Considering the replacement interval? Yes

[0086] (D) Same operator? Yes

[0087] The evaluator 203 (possibly the same person) will determine whether the standard is met (all yes). Potentially, the inspection report 402 will become a compliance statement (enhanced by a note "PASSED" or the like).

[0088] In other words, there needs to be evidence that the particular operator performed the pre-manufacturing activity (e.g. wearing gloves) before the (subsequent) manufacturing activity. This proof is available in the proof document, namely the inspection report 402.

[0089] Note: the inspector 202 cannot look at the operator all the time (i.e. t1, t2, t4, not including t3). It can be difficult for the inspector 202 to keep track of the activities of various operators working in the same facility. The identity of the operator is needed in the sense that the same operator actually performed these activities (see (D)).

[0090] It should be noted that the activity within the facility 100 can be a do-not-activity (or "negative activity") in the sense that, according to the standard, certain activities are not allowed within the facility 100. Take the clean room as an example. It is possible to comb the hair or clean the nails, but it is not allowed to do so within the clean room (i.e. further requirement (E) as negative activity). There are dedicated rooms - bathrooms or the like - where this can be done. By contrast, going to the toilet is an example of a non-relevant activity 230 (at t3) that is not monitored by the method (here described). In other words, the execution of a do-not-activity is assessed, but the execution of an action that is not relevant is not of interest.

[0091] However, there are many other cases that can lead to non-compliance. The inspector 202 will have to pay close attention to this. This is explained below:

[0092] Figure 3 The non-compliance with the standard in cases (A) to (D) is illustrated by way of example. It is noted that non-compliance with at least one of the standard requirements leads to non-compliance with the standard.

[0093] • In case (A), the operator does not wear gloves when locking. The operator wears gloves, but they can be different.

[0094] • As in case (B), the operator does not wear two gloves when loading the ingredients into the reactor. The operator has lost one.

[0095] • As in case (C), the operator does not take into account the change of gloves (in case of change interval).

[0096] • In case (D), there is a different operator who is performing the interaction.

[0097] Method for providing an inspection record

[0098] Figures 4-6 A computer system 400 (i.e. an inspection system) is shown to execute a computer-implemented method 600 for providing an inspection record 500. Since the method 600 comprises image processing, Figures 4-5 An image is shown. Figure 6 A flowchart is shown (with steps 610 to 660). For simplicity, Figures 4-6 Computer components such as a processor or a memory are not shown. The execution of the method 600 does not necessarily result in an inspection record with a standard compliance statement (e.g. PASSED) or with a standard non-compliance statement (e.g. FAILED). The execution of the method 600 is applicable to scenarios leading to PASSED (see Figure 4 ) and to scenarios leading to FAILED (see Figure 5 ).

[0099] Figure 4 The computer system that performs the image processing that results in the review record 500 is illustrated in an example with a first scene. Reference numbers (e.g., “301”) are not part of the image.

[0100] The review system 400 (see Figure 1B ) receives a series 700 of images 701 and 702 (from the cameras). In this example, image 701 is from camera 310-1 and image 702 is from camera 310-2. The cameras take images at subsequent points in time, and (in embodiments) the images have timestamps.

[0101] In the series 700 of images 701 and 702 (two images are simplified for ease of explanation), image 701 has captured a first activity (interacting with a first device (see 210 in Figure 2 ) and image 702 has captured a second activity (also interacting, see 220 in Figure 2 ).

[0102] The review system 400 then applies image processing techniques to divide the images into ID regions 701x, 702x and activity regions 701y, 702y) to obtain identifiers (e.g., ALPHA) and remove biometric data.

[0103] The review system 400 divides the images into (see step 620) ID regions 701x, 702x (of images 701, 702) and activity regions 701y, 702y (of images 701, 702). A region is a contiguous plurality of pixels. A region has a shape, e.g., a rectangle, a circle, or other shape. Coordinates can be used to process the regions, e.g., XY coordinates of the pixels. Similar to a geographically known exclave / enclave, one region can enclose another region.

[0104] The ID regions include biometric data (e.g., facial images), while the activity regions show the activity (i.e., the operator’s interaction with the device).

[0105] The review system 400 then processes the ID regions and obtains (step 630) identifiers ID1, ID2 for ID regions 701x and 702x, respectively. In Figure 4 the example, the identifiers ID1 and ID2 are ALPHA.

[0106] The review system 400 removes (step 640) the biometric data (see step 640). This is illustrated using the graphic anonymizer. But other methods can be used (e.g., by switching substantially all pixels of the ID area to a single color, such as black or white). The removal is in one direction, with no opportunity to reestablish the content (i.e., to retrieve the biometric data). The removal of the biometric data is a one-way data conversion. The identifier (ALPHA) cannot be converted back to the personal data.

[0107] In the last step, the removal of the content can reduce the number of bits and bytes used to store the image. This is a further benefit. The saved bits and bytes can be used to transmit data (e.g., to transmit the parameters in Figure 8A , Figure 8B , Figure 8C .

[0108] The review system 400 combines (650) the modified first area 701xx of the first image 701 with the second area 701y of the first image 700 into a modified first image 701'. The review system 400 combines (650) the modified first area 702xx of the second image 702 with the second area 702y of the second image 702 into a modified second image 702'.

[0109] The review system 400 provides a review record 500 by storing the modified first image 701', the modified second image 702' (i.e., the modified series 700') and storing an identifier for each modified image (e.g., ID1 = ALPHA, ID2 = ALPHA).

[0110] For convenience, the identifier (e.g., ALPHA) is shown as part of the modified image (i.e., the identifier is embedded), but the identifier can be related to the modified image in other ways (e.g., as metadata, using known data binding techniques).

[0111] Note that the review record 500 does not provide an assessment (pass, fail) according to the standard, but the review record 500 contains all the information needed to perform the assessment.

[0112] The assessor 203 (see Figure 1B ) can examine the modified series 700' (i.e., the review record 500) and check if the requirements are met:

[0113] • (A) gloves on when locking? Yes, this is visible in image 701'

[0114] • (B) both gloves on when loading the reactor with raw material? Yes, this is visible in image 702'

[0115] • (C) Consider replacement interval? Yes, this is not shown in the figure, but the images 701, 702 have timestamps, and the timestamps are also taken over to the modified images 701' and 702'. Looking at the timestamps, the evaluator 203 can determine if the interval has passed.

[0116] • (D) Same operator? Yes, the evaluator 203 reads ALPHA associated with both images 701' and 702'.

[0117] Looking at the review record 500, the evaluator 203 can make a determination of (in) compliance with the standard. But unlike the traditional reviewer (see 202 in Figure 1A ), the evaluator does not have to exclude information. In other words, information that is not to leave the facility (e.g., the face data of the operator). The above inclusion / exclusion dilemma has been solved.

[0118] Figure 5 A computer system image processing is shown that results in a review report in the example with a second scenario. Figure 5 The process in Figure 4 is the same as in Figure 3 , but the content of the review record 500 is slightly different. In case (D), there are activities performed by different operators (see 201A and 201Z).

[0119] While the review record 500 of Figure 5 does not indicate compliance, it indicates ALPHA (as the operator performing the first activity) and BETA (as the operator of the second activity). The evaluator 203 will determine non-compliance with the standard (e.g., a failure). The inclusion / exclusion dilemma has also been solved.

[0120] It should be noted that the images 701 and 702 (in Figures 4-5 ) are sufficient for a person in the role of reviewer / evaluator to determine compliance ( Figure 4 ) or non-compliance ( Figure 5 ), but inclusion / exclusion remains an obstacle.

[0121] Figure 6 A flowchart of a computer-implemented method 600 is shown for providing a review record 500 involving activities 210, 220 performed by a human operator 201 within a facility 100 interacting with a technical device 300.

[0122] The review record 500 indicates performance of a sequence of activities (see 210, 220 in Figure 2 ) with a first activity (e.g., 210) and with a subsequent second activity (e.g., 220). These activities are performed by the same human operator (e.g., Figure 4the same human operator (e.g. ALPHA) in the first image 701 and in the second image 702. Figure 5 The method 600 is executed by the review system 400 (see Figure 1B ).

[0123] In a receiving step 610, the review system receives a series 700 of images 701, 702 visualizing activities performed by human operators (i.e. the images have captured images of activities). The series 700 has at least a first image 701 of a first activity 210 interacting with a device 301A and a second image 702 of a second activity 220 interacting with a second device 301B, 301C.

[0124] By applying image processing on the first image 701 and on the second image 702, the review system performs a separating step 620. Within the first image 701 and within the second image 702, the review system separates at least a first area 701x, 702x and a second area 701y, 702y. The first area 701x, 702x comprises biometric image data indicative of a specific human operator 201A belonging to a group of human operators 201 of the manufacturing facility 100. The second area 701y, 702y comprises image data indicative of an interaction with a technical device 301A, 301B, 301C (see Figures 4-5 ).

[0125] By applying image processing on the first area 701x of the first image 701 and on the first area 702x of the second image 702, the review system performs an obtaining identifier step 630. It obtains an identifier ALPHA, BETA of a human operator 201 among the plurality of human operators. The identifier is thus indicative of the performance of the first activity and of the second activity by the specific human operator (as Figure 4 in the first image 701 and in the second image 702). Figure 5 in the first image 701 and in the second image 702).

[0126] By applying image processing on the first area 701x of the first image 701 and on the first area 702x of the second image 702, the review system performs a removing biometric image data step 640. The biometric data is indicative of the specific human operator from the first area 701x (ID area) of the first image 701 and from the first area 702x of the second image 702. The first area 701x of the first image 701 is modified into a modified first area 701xx of the first image 701 and the first area 702x of the second image 702 is modified into a modified first area 702xx of the second image 702. The second area (activity area) remains unchanged.

[0127] In a combining step 650, the review system combines the modified first region 701xx of the first image 701 with the second region (701y) of the first image 701 into a modified first image 701', and the modified first region 702xx of the second image 702 with the second region 702y of the second image into a modified second image 702'.

[0128] In a providing step 660, the review system provides a review record 500 by storing the modified first image 701', the modified second image 702' with the respective identifiers (i.e. the identifiers for the first and second image). The review record 500 does not necessarily indicate whether the standard was met or not.

[0129] Figure 6 A computer program or computer program product is also shown. The computer program product, when loaded into the memory of a computer and executed by at least one processor of the computer, performs the steps of the computer-implemented method. In other words, the computer program product comprises program code which, when executed by the computer, causes the computer to perform the steps of the computer-implemented method. Figure 6 The blocks in the diagram of

[0130] ■Multiple assessors

[0131] Although Figure 1A and Figure 1B The diagram is simplified in terms of a single facility and a single assessor, but it is noted that a single facility can manufacture products for multiple customers, and thus also for multiple assessors. Data segregation is important: the review data for a first assessor must be separated from the review data for a second assessor. The present description explains how the production control signal (coordination) is used to separate the records.

[0132] ■Embodiment with batch separation

[0133] Figure 7 The execution of the method steps is illustrated for different production batches (or "loads") is illustrated. The steps are illustrated by rightward large arrows with text.

[0134] The production management system 410 is a production management system (e.g. a distributed control system DCS, a process control system PCS) that is communicatively coupled to the review system 400 (see Figure 1B ). The production management system 410 sends triggers (indicated by the vertical arrows Tr1 to Tr5) to the review system 400 (and / or to the camera). Creating an interface between the systems 400 / 410 for the triggers is within the expertise of the skilled person.

[0135] In a simplified illustration, assume a manufacturing process that distinguishes two separate production batches (or fills). Certain activities (human-machine interactions) apply to both batches, e.g. activity 210.

[0136] In the method 600, the receiving step 610 can be triggered by the production management system 410 providing the identity of a particular batch for each image. As a result, in the step 660 of providing the review records 500, the review records 500#1, 500#2 are related to the particular batch.

[0137] In the example of Figure 7 , the production management system 410 sends the triggers Tr1 to Tr5 consecutively. By sending Tr1, the production management system 410 causes the system 400 to receive consecutively the images 701 (first activity, camera 310-1), e.g. each time the operator enters a lock. For convenience, Figure 7 Only one image is shown. The images (in modified form, see below) will enter the review records 500#1 and 500#2 for batch #1 and batch #2. This activity applies to all batches. There is no need to hide (modify) the images, as the first activity is the same for all batches. Figures 4-6

[0138] The trigger Tr2 causes the production management system 410 to process the image 702-1 (second activity) into the review record 500#1 for batch #1; the trigger Tr3 causes the production management system 410 to process the image 702-2 into the review record 500#2 for batch #2; the trigger Tr4 causes the production management system 410 to process the image 702-3 into the review record 500#1; the trigger Tr5 causes the production management system 410 to process the image 702-4 into the review record 500#2, and so on.

[0139] More generally, the receiving 610 of the series of images is triggered by the production management system providing the identity of a first batch #1 and a second batch #2, wherein a first image 701 of a first activity 210 is received in relation to both batches #1 and #2, wherein a second image 702 of a second activity 220 is received in a batch-specific version: second image 702-1, 702-3 for the first batch and second image 702-2, 702-4 for the second batch. The providing of the review records comprises providing separate review records 500#1, 500#2 for the first batch and for the second batch.

[0140] ■Embodiment with parameters

[0141] Figure 8A , Figure 8B and Figure 8C ​Figures illustrate images showing devices with representations of parameters 351, 352, 353. These parameters are device parameters.

[0142] In the example of Figure 8A , the device is represented by a glove dispenser (301A) with a label. The label is captured by image 701 (partially shown). The label indicates a consumption expiration date as parameter 351 (e.g. "Valid until 12 / 2018").

[0143] In the example of Figure 8B , image 702 has a metadata attachment (702-DATA) to indicate a parameter 352 of the reactor tank (e.g. a specific temperature of 50°C inside the tank).

[0144] In the example of Figure 8C , parameter 353 is embedded into the image (here for temperature).

[0145] An assessor can be interested in technical parameters. For the assessor, this information can be relevant for checking compliance with standards.

[0146] Parameters can be available at a device interface, e.g. as a communication device (e.g. data output, interface for an industrial data bus, interface for a computer network, etc.) that can be part of the device. Such parameters can include measured values for the device (e.g. from sensors, meters, etc. measured values for pressure, temperature, ingredient amounts, etc.). Parameters 351, 353 can be color-coded parameters.

[0147] Figure 8A , Figure 8B and Figure 8C Examples can be conveniently used to further illustrate batch separation (see Figure 7 ). Just to give an example, activity 220 can require that the ingredient is type 405 flour in a first batch and type 550 flour in a second batch. This type of parameter is typically printed on the bag (see parameter 351 in Figure 8A ). Different flour types can represent a set of specific requirements for a specific production batch (e.g. batches of different flour).

[0148] The way the parameters are delivered (in the image Figure 8A , in metadata Figure 8B or embedded Figure 8C ) is not important. In terms of method 600, receiving 610 can include receiving technical parameters 351, 352, 353. The parameters are associated with the image as part of the image (Figures 8a, 8c) or as received in a metadata attachment (702-data) of the image Figure 8B .

[0149] Since the parameter is a technical parameter, it is not related to the biometric data. Therefore, the image portion with the parameter belongs to a second area (or metadata) that is processed substantially invariant. The location where the parameter is embedded (if it is embedded in the image) is expected to be far enough from the ID area so as to prevent accidental removal of the embedded parameter.

[0150] ■Embodiment with multiple cameras

[0151] Figure 10 An embodiment is shown with multiple cameras sending images to the computer system. The figure shows images 701-C1 and 701-C2 taken simultaneously by two cameras. One image shows the operator in a frontal view, while the other image shows the operator in a side view. Method 600 can be applied to both images, whereby review record 500 (see Figures 4-5 ) is augmented with images from different directions. The multiple camera approach can also be used with other cameras in the facility. In other words, the image (to be processed, to be provided to obtain the record) can be a collection of multiple images from multiple cameras.

[0152] In view of method 600, receiving 610 can include receiving multiple images taken from different directions, wherein the images are taken simultaneously. As used herein, "simultaneously" is understood as the time frame in which the operator performs the activity.

[0153] Not all method steps have to be applied to both images. The frontal view in image 701-C1 can be more suitable for obtaining the identifier (ALPHA, BETA, see step 630), while the side view in image 701-C2 can be more suitable for showing the activity. The need to perform the method steps is different: image 701-C1 is being processed, including obtaining the identifier; while image 701-C2 is only partially processed (possibly not requiring obtaining the identifier at all). This difference can provide higher computational efficiency for the system 400 performing the method.

[0154] ■Embodiment to optimize optical resolution

[0155] Figure 10 An embodiment is shown with a camera sending an image to the computer system 400 (see Figure 1B ). The image taking can be optimized by assigning a relatively higher resolution (i.e. pixels per area) to objects with a relatively higher relevance to the activity. The resolution selected depends on a variety of factors, including the ability of the evaluator to view the operation performed by the operator, and / or the ability of the computer to recognize the content, for example by optical character recognition (OCR) to identify whether the computer (e.g. system 800) is used to identify the parameter (see Figure 8A 、 8C ).

[0156] The person skilled in the art can install a camera with suitable optics. For example, the focal length of the camera objective can be chosen such that one image comprises the human operator (at least with a biological feature) and the technical equipment.

[0157] Depending on the optical elements of the camera, the scaling of real-world objects (i.e. the operator and the equipment as described above) to pixels on the camera sensor can be distinguished by the optical elements of the camera lens. In the following, the camera 310-1 (camera in the clean room lock) is exemplified.

[0158] The normal image 701a (left side of the figure) is the image 701 as explained above in Figures 4-5 The scaling from real-world objects to pixels is essentially constant over the whole image. In other words, the scale is constant. A person is depicted with a head-to-body ratio corresponding to reality, a square is still a square, and so on.

[0159] The scaled modified image 701b is captured with a special optical lens that magnifies the areas of interest, here the face, the hands and the glove dispenser (see equipment 301A). The magnification factor is an optical magnification factor, so that the areas of interest are captured with a relatively large number of pixels. In other words, the pixel resolution is increased. On the other hand, areas of less interest (e.g. the operator's feet) can be captured (if at all) with a relatively small number of pixels.

[0160] Since the areas of different interest can be identified in advance, the person skilled in the art can set the optical magnification factor accordingly. Depending on the type of activity, different parts of the body can be monitored. For example, if a standardized activity requires the operator to wear special shoes for use in the clean room, it can indeed be monitored that the operator's feet. In this case, the magnification factor will be highest for the foot area.

[0161] The scaled modified image 701b looks distorted, but this distortion is irrelevant for the computer. The degree of distortion is limited to the technical ability of the review system to obtain (step 630) the identifiers (ALPHA, BETA) and the cognitive ability of the evaluator to view the activity (by inspecting the review record 500). In other words, it does not matter whether the operator and / or the equipment appear on the image in a disturbed form. Since the evaluator knows where to look, the burden is low.

[0162] The review computer 400 can use either version (normal image 701a or scaled modified image 701b) to perform the processing steps (i.e. separation 620, obtaining 630, combining 650). Using the scaled modified version 701b can have the advantage of showing more details. For example, the version 701b can be more suitable than the version 701a to show technical parameters (see Figure 8A , Figure 8C ).

[0163] As shown on the right, the scaled modified image 701b can be pre-processed back to a normal image 701c (before, during or after the processing).

[0164] It is noted that an image captured using special optics can be used as the modified image (see 701' and 702' in Figure 5 . For the evaluator, the content is important (like the added identifiers ALPHA, BETA, etc., or technical parameters), not the shape of the operator or the shape of the equipment.

[0165] The special optical element can be realized by a wide-angle lens, a lens with an angle large enough to capture both the operator and the equipment at the same time. The wide-angle lens can be realized as a so-called fisheye lens (or frog-eye lens). These lenses are known in the art for decades (in the patent literature, for example in GB 225,398 and US 2,247,068 A).

[0166] In other words, the image provided by the optics is such that certain parts seem to be enlarged, while certain parts seem to be shrunk. Since the activity is performed repeatedly (see Figure 2 ), there is no need to re-adjust the optical element for each image.

[0167] Embodiment where the image is part of a video

[0168] Figure 11 A series of multiple images is shown, taken by a camera, for example camera 310-1 (associated with the clean room lock) and camera 310-2 (associated with the tank reactor). Since the series 700 (of images) can be stored as a video, a review record 500 can be provided, wherein the image 701' is part of a first sub-series (showing a first activity) and the image 701' is part of a second sub-series (showing a second activity).

[0169] It is noted that for a review record 500 with a video, the removal (step 640) of biometric data must be performed for all images of the video containing such data. However, obtaining the identifier (step 630, potentially more computationally intensive) would have to be performed only for (at least) one image.

[0170] Figure 11 The examples show only some video frames. They are numbered from 1 to 11. The figure is greatly simplified in terms of number of images (or video frames). For example, using a video with 24 image frames per second, and assuming the gloves are put on for 2 seconds, 48 frames would show the activity (i.e. activity 210, see Figure 2 ).

[0171] Assuming synchronization (1st frame of both cameras has the same timestamp, same inter-frame time distance), the 3rd frame of camera 310-1 would correspond to image 701 (received by system 400), the 4th-6th frames of camera 310-2 would correspond to image 702 (also received by system 400).

[0172] Establishing the series 700 (of the first image 701 and the second image 702) can be performed after the video is taken (assuming the video is stored), and both images can be correlated by an operator identification (e.g. ALPHA).

[0173] It is also possible to check whether a particular image does not comply with the standard and identify a subsequent activity. For example, in the 9th frame of the lock camera, the operator forgets to put on the gloves. He or she can be identified (i.e. BETA), and the images (e.g. frames from the reactor tank camera) showing BETA in the subsequent activity can be used to identify the non-compliance (see requirement (B), which can be identified from a single frame in this case).

[0174] The description continues to discuss other aspects related to the general figures.

[0175] There is a risk that the operator or other people (in the facility 100) interact with the review system 400 to manipulate the report. The system 400 can be protected by security measures. It is also possible to add graphical data to the (modified) images in the review record 500 by techniques such as watermarking to identify the system 400 (as a processing entity).

[0176] From a broader perspective and business-related considerations, the assessor 203 (in Figure 1B ) can work for a customer. To remain in the chemical industry, the customer can be a different industrial company, but also a retailer. The customer will receive the review record 500. There are many ways to deliver such data, for example by using a computer portal (at the review system 400) to which the customer has access. The customer can operate an assessment system 800 that helps perform the assessment.

[0177] In this sense, the customer can (at least virtually) view the facility, but with great differences. Unlike the reviewer 202, the customer will see the data of interest (related to the compliance with the assessment standard, but not the information that must be kept within the facility, see above). Again, the inclusion / exclusion dilemma is solved.

[0178] ■Additional embodiments

[0179] The description now investigates potential further limitations and ways to address them.

[0180] ■Operators on the images

[0181] A first limitation concerns the suitability of the images for the operators.

[0182] As shown, the cleanroom operators wear gloves. But in many industries, the operators also wear special clothing. The clothing can include some kind of headwear that covers the hair so that the hair does not appear on the images. Eventually, the operators can wear a mask (as surgeons) so that the face is only fully shown on the images.

[0183] However, it is not necessary to use complex face recognition techniques suitable for recognizing faces (e.g. in social networks) to obtain the identifiers (ALPHA, BETA).

[0184] Since the number of operators within the facility is limited (e.g. 1000 operators), it is possible to use artificial intelligence to train a computer to recognize only the faces of these operators. For example, the operators can be photographed (wearing the working equipment) and the identifiers (i.e. from ALPHA_1, ALPHA_2... ALPHA_1000) can be used for training (supervised training).

[0185] In terms of the method 600, the biometric image data comprises data related to the face of a human operator and, in the step of obtaining 630 the identifier, an image of a predefined set of operators is used as a reference. The set of operators can include operators associated with the facility (e.g. its employees) or operators associated with a particular part of the facility (e.g. operators working in a particular sub-part of the facility) or operators associated with a particular equipment (e.g. operators operating a reactor tank) or operators working in the facility at a particular time interval (e.g. during a particular shift of a product batch being produced).

[0186] Some operators wear protective glasses (or generally goggles), but the glasses are clear enough that face recognition is possible.

[0187] ■Technical measures for isolating the image data

[0188] A second limitation concerns the protection of the data from accidental misuse. The series of images 700 (received by the review system from the camera) contain information that can allow a viewer to identify a particular person.

[0189] To limit the risk of such information accidentally leaving the facility, the method steps receiving 610, separating 620, acquiring 630 and removing 640 can be performed by a processor of a computer system 400 using only volatile memory. In other words, as long as the ID areas 701x, 702x (see Figures 4-5 ) have not been removed, the image data remains in the random access memory RAM of the computer processor. The combining 650 (i.e. by accessing the modified areas 701xx, 702xx) can be performed in non-volatile storage (e.g. hard disk drive storage, solid state drive storage). From a different perspective, the technical measure of intercepting data from memory is more complex than the technical measure of intercepting data from storage, which exploits this difference.

[0190] In view of the above introduced batch separation (see Figure 7 ), the use of volatile memory further reduces the potential risk of cross-feeding information from one batch to another.

[0191] However, these steps can be performed in non-volatile memory (i.e. with a database) and separate the batches at a later point in time (when the second image is taken).

[0192] ■Combining embodiments

[0193] As mentioned above, the camera can be implemented in different ways, and the present description has explained two main directions:

[0194] • by the number of images taken simultaneously to obtain images from different directions (see Figure 9 ),

[0195] • by the difference in optical elements of the camera and the field of view area to select a suitable area-related image resolution (see Figure 10 )

[0196] Further implementations are possible, for example using a camera with a zoom lens. The camera can be controlled according to signals obtained from the production management system 410. In this case, it is even possible to distinguish between batches. As Figure 7 indicated, the review record 500#1 for the first batch can use a different camera setting than the review record 500#2 for the second batch.

[0197] The camera settings (direction, movement, e.g. zoom, pan, tilt) can be obtained as predefined settings. The predefined settings can be provided by the production management system (410) in Figure 1B . The production activities for different batches can be recorded differently by having a record with still images or a record with video.

[0198] It is also contemplated to post-process the images, for example by applying digital zooming when presenting the audit record 500 to the assessor 203.

[0199] ■Detecting compliance at an early stage

[0200] The recording can be simplified by storing a data set representing the activities performed by the operator. For example, the simplified recording can include the following data set: Figure 2

[0201] • ALPHA's first activity "put on gloves" at 09:00,

[0202] • ALPHA's second activity "add ingredient" at 09:10,

[0203] • ALPHA's first activity "put on gloves" at 10:10, and so on.

[0204] In such embodiments, it can not be necessary to store the (modified) images. The simplified recording can serve as a basis for the assessment. However, if non-compliance becomes possible, the simplified recording can be used outside the audit context to alert the operator.

[0205] ■Manufacturing-centric activities

[0206] The audit system 400 provides an audit record 500 for activities in the facility. These activities can include pre-manufacturing activities (e.g. activity 210 "put on gloves"), manufacturing activities (e.g. activity 220 "add ingredient") and post-manufacturing activities (e.g. cleaning the equipment after use).

[0207] ■Location of the audit system

[0208] In the Figure 1B , the audit system 400 is shown as belonging to the facility 100 (see the large rectangle). However, the audit system 400 does not have to be physically implemented on the premises of the facility, it can be implemented by a computer system remote from the facility.

[0209] ■Real-time aspects

[0210] Although the audit system 400 provides the results as a record (i.e. a data structure to be stored), it is also contemplated to consider some real-time or simultaneous aspects. For example, the trigger to start receiving images (or a series of images) can also come from another computer system, for example from the assessment system 800 (see Figure 1B ). The assessor can decide to monitor the production starting from a specific point in time (e.g. when a specific batch is produced).

[0211] ■Implementation details

[0212] ​The camera is communicatively coupled to the review system. For simplicity, Figure 1B The connection is not shown, but suitable connections are known in the art (e.g. via a network).

[0213] Since image processing is known in the art, the computer routines and code can be selected by the person skilled in the art from libraries. For example, for the separation 620, the computer system can use face recognition techniques known from hand-held consumer cameras, in the sense of recognizing that a face is present on the image, not in the sense of detecting the person having that face. While the mentioned cameras typically frame the face with a rectangular box (on the viewer screen of the camera), it is not necessary to show such a frame or the like to the user. No user interaction is required at this stage.

[0214] Face recognition is known in the art from many publications, including the following publications:

[0215] • Qin Zhou, Heng Fan, Hang Su, Hua Yang, Shibao Zheng, Haibin Ling: "Weighted Bilinear Coding over Salient Body Parts for Person Re-identification", Pattern Recognition Letters, Preprint Cornell University 2018 "arXiv: 1807.00975", explaining the re-identification of persons.

[0216] • Allyson Rice, P. Jonathon Phillips, Vaidehi Natu, Xiaobo An, and Alice J. O'Toole: "Unaware Person Recognition From the Body When Face Identification Fails", Psychological Science 24(11) 2235-2243, providing guidance on how to recognize persons when face recognition fails.

[0217] It can be expected that the operator 201 wears a helmet such as an industrial safety helmet (see DIN EN 397:2013-04). Since the helmet can cover certain parts of the face, the image can also be processed in addition to the face image on the basis of other biometric features, such as the shoulders and / or the height.

[0218] It is to be noted that, once the ID region with biometric data is recognized (see Figures 4-5The ID region can also identify other regions with activity (by subtraction (active region = image minus ID region) from the image). No specific processing is needed to identify specific activities.

[0219] To ensure that the images 701, 702 are taken at subsequent points in time, a time stamp can be added. In other words, the receiving 610 can comprise receiving a series 700 of images 701, 702 with a time stamp of the images 701, 702. In principle, there are two options.

[0220] • In an embodiment, the time stamp is added by the camera (assuming the camera has a synchronized clock).

[0221] • It is also possible to add the time stamp when the image is received by the review system.

[0222] The signal propagation delay between the camera and the review system can be negligible, as the subsequent activity is performed by a human operator in a time interval (from the first activity to the second activity) that is larger (i.e. longer) than the signal propagation time (from the camera to the system). Time stamps of images are known in the art. They can be embedded into the image (graphically) and can be transmitted as metadata.

[0223] These activities are industrial activities with the following properties: (i) they are often repeated and (ii) they are standardized. Continuing the example explained above, when putting on gloves (activity 210, on image 701), the operator moves his / her hand near the glove dispenser, but the head does not move much. When adding the ingredient (activity 220, on image 702), the operator moves the hand (more than the feet) and can look inside the reactor. In both cases, for most activities, the viewing direction is more or less predefined (i.e. in the direction of the dispenser, inside the reactor). The ID region on the image that the camera (at the time of positioning) has is more or less the same for all images (repeated).

[0224] Taking into account that different pixels change differently over time, a dynamic way can be implemented to distinguish between the regions 701x, 702x and 701y, 702y. For example, when the operator (in the facility) performs an activity, he or she moves the hand more than the feet (for example). Calculating these differences (for example, between images taken a fraction of a second apart) and identifying the potential regions should be sufficient.

[0225] It is noted that also here auto-focus techniques (known from digital cameras) can be used.

[0226] ■Calibration of biometric data

[0227] As mentioned above, in the step of acquiring the identifier, an image of an operator 201 from a specific group of operators is taken as a reference. The specific group of operators 201 can be defined by various circumstances, for example: operators working in the facility, operators working in the facility during a specific shift, operators working in the facility using a specific equipment (for example, operators of reactor tanks are different from operators or drivers of vehicles). Since the appearance of the operators changes from time to time (for example, different hairstyle on different days; beard style), it is possible to take the image at an appropriate point in time (for example, periodically).

[0228] ■Non-biometric image data treated like biometric image data

[0229] Operators can carry their name on a name tag, badge, embroidery on their clothing, etc. The name is usually written in a color that contrasts sharply with the color of the clothing (for example, black on white). This allows colleagues to familiarize each other.

[0230] Optical character recognition technology (OCR) allows a computer to convert an image with letters into data. It is expected that such an OCR reading is also used to acquire (step 630) the identifier (for example, ALPHA, BETA). The area on the image with the name will be considered as an area with biometric image data (i.e. ID area) and can also be removed (step 640). This approach can even be mandatory since the name (of the operator) is personal data (as mentioned above) and is to be masked to the assessor. It is noted that OCR processing of the name helps to distinguish between operators with similar appearance (for example, identical twins in extreme cases).

[0231] ■Assessment computer

[0232] Figure 1B An assessment system 800 is also illustrated. This computer system can be used as a user interface for the assessor 203. As mentioned above, the identifier (ALPHA, BETA) can be embedded into the (modified) images 701', 702', the same applies to the parameters 351, 353 (see Figure 8A 、 8C ). Image viewing software is usually available on computers. It is also expected that the computer file formats allowing image storage (*.jpg, *.bmp, *.tiff, *.pdf, *.png) corresponding to the potential requirements of the archival record will remain available for the next decades.

[0233] Once the review record 500 is available, the review system 400 can push the review record 500 to the assessment system 800. It is possible to notify the assessor that the record is available (by various techniques, for example, e-mail, short message service SMS, instant messaging, for example, WHATSAPP(TM), etc.).

[0234] Although the evaluation system 800 is shown as a single computer, the review system 400 can send review records to different computers depending on the batch or depending on other circumstances.

[0235] The systems 400 and 800 can be communicatively coupled by internet technology, with / without access technology through a website, a web portal, a relationship management system, a database system, etc.

[0236] The review records 400 can be enhanced by process documentation (e.g., of the process being performed at the time the image is received) and / or by documentation indicating facility details (e.g., a map or plan of the facility). Such additional files can help the evaluator.

[0237] ■Other use cases

[0238] For example, a standard can require that a transport device (e.g., a fork lift moving luggage) move at a speed below a speed limit (e.g., 10 km / h). However, a short time above the limit (e.g., 12 km / h for 2 seconds) can be allowed. By taking timestamped images 701 and 702 at two different locations (with a known distance), the average speed can be calculated.

[0239] ■Location stamp

[0240] An operator can be required to perform certain activities within a predefined location. For example, a bag of opened flour should be poured into a nearby tank reactor, but must not be carried through the facility. To make the evaluation as easy as possible, images can optionally be associated with a location within the facility with a predefined granularity (e.g., identifying a particular device, a particular room within the facility, particular coordinates, a particular production line, etc.). In other words, images can come with a location stamp.

[0241] ■General computer system

[0242] Embodiments of the invention can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The invention can be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers. Tangible computer-readable storage devices include all computer-readable media, for example, volatile and non-volatile storage devices, including but not limited to, RAM (random access memory), ROM (read only memory), EPROM (erasable programmable ROM), EEPROM (electrically erasable programmable ROM), flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The computer-readable storage devices can also be distributed over network-coupled computer systems so that the instruction and data is stored and executed in a distributed fashion. The embodiments of methods described herein are all equally applicable to combined processing by one or more programmed processors and computer(s). The computer program can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed by one computer or by multiple computers in one site or distributed across multiple sites and interconnected by a communication network. The described methods can all be performed by a respective computer program product, e.g., by the first and second computer, the trusted computer and the communication unit.

[0243] The steps of methods of the invention can be performed by one or more programmable processors executing a computer program to perform the functions of the invention by operating on input data and generating output. The methods steps can also be performed by, and apparatus of the invention can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0244] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Such storage devices can also be connected to the computer through a network as indicated, e.g., via the Internet, an Intranet, an Extranet, or a local area network. The tangible computer-readable storage devices include all computer-readable media for storing software instructions, including portable or fixed storage devices, optical storage devices, and

[0245] To provide for interaction with a user, the application can be implemented on a computer having a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user and an input device (e.g., a keyboard, a touch screen or touchpad, a pointing device (e.g., a mouse or a trackball)) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0246] The application can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the application), or any combination of such back-end, middleware, or front-end components. The client computer can also be a mobile device, such as a smart phone, tablet, or any other hand held or wearable computing device. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet or an intranet, or a wireless LAN or a telecommunication network.

[0247] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0248] ■Reference signs

[0249] 100 Facility

[0250] 200 Person

[0251] 201 Operator (A...Z)

[0252] 202 Reviewer

[0253] 203 Assessor

[0254] 300 Equipment

[0255] 301 Equipment (A...Z)

[0256] 310 Camera (-1, -2)

[0257] 399 Gloves

[0258] 400 Review System

[0259] 402 review report

[0260] 410 production management system

[0261] 450 standard

[0262] 600 method

[0263] 6x0 method step

[0264] 700 series of images

[0265] 701, 702 image (-x, -xx, -y, etc.)

[0266] 800 evaluation system

[0267] Tr1...Tr5 trigger

Claims

1. A computer-implemented method (600) for providing an audit record (500) relating to an activity (210) performed by a human operator interacting with technical equipment within a facility (100), wherein, The review record (500) is indicative of an execution of an activity (210) having a first activity (210), the method (600) comprising: receiving (610) an image (701), the image visualizing an activity (210) by a human operator, the activity comprising an interaction with a first technical device, wherein receiving (610) comprises receiving at least one technical parameter (351, 352, 353) related to the technical device, wherein the at least one technical parameter is received as part of the image or in a metadata attachment (702-DATA) to the image in association with the image; separating (620) within the image (701) at least a first region (701x) and a second region (701y) by applying image processing to the image (701), wherein the first region (701x) comprises biometric image data indicative of a specific human operator (201A) and wherein the second region (701y) comprises image data indicative of an interaction with the technical device; obtaining (630) an identifier (ALPHA) of the human operator among a plurality of human operators by applying image processing to the first region (701x) of the image (701), the identifier (ALPHA) thereby being indicative of an execution of the first activity by the specific human operator; removing (640) from the first region (701x) of the image (701) the biometric image data indicative of the specific human operator by applying image processing to the first region (701x) of the image (701), such that the first region (701x) of the image (701) is modified into a modified first region (701xx) of the image (701); combining (650) the modified first region (701xx) of the image (701) with the second region (701y) of the image (701) into a modified image (701'); providing (660) the review record (500) by storing the modified image (701') with the respective identifier (ALPHA); wherein the steps of receiving (610), separating (620), obtaining (630), removing (640) are performed by at least one processor using only volatile memory.

2. The method (600) of claim 1, wherein Receiving (610) comprises receiving the image (701) with a timestamp of the image (701).

3. The method (600) of claim 1, wherein Receiving (610) comprises receiving the image (701) from a first camera (310-1), wherein the first camera is associated with the first technical device.

4. The method (600) of claim 1, wherein Receiving (610) comprises receiving the technical parameter (351, 352, 353) related to the technical device from a device interface, wherein the technical parameter is associated with the image (701).

5. The method (600) of claim 1, wherein The receiving (610) of the image (701) is triggered by a production management system for a specific image, the production management system providing an identification of a specific batch, such that as a result of providing the review record (500), the review record (500#1, 500#2) is related to the specific batch.

6. The method (600) of claim 5, wherein The receiving (610) of the image is triggered by a production management system providing an identification of a first batch (#1) and a second batch (#2), wherein the receiving of the image (701) of the activity (210) is related to both batches (#1, #2).

7. The method (600) of claim 1, wherein The biometric image data comprises data related to a face of the human operator.

8. The method (600) of claim 7, wherein In the step of obtaining (630) the identifier, an image of an operator (201) from a specific group of operators is used as a reference.

9. A computer program product, which, when loaded into the memory of a computer and executed by at least one processor of the computer, carries out the steps of the computer-implemented method according to any one of claims 1-8.

10. A review computer system, which is adapted to carry out the method according to any one of claims 1-8.

Images