Methods, apparatus, and computer readable media for measuring productivity

Abstract

A method, apparatus (70), and program(s) for measuring productivity are provided. The method includes identifying a first movement based on at least one image frame, wherein the first movement matches a start action that defines a cycle of movement (S1); identifying a second movement based on at least one image frame, wherein the second movement matches an end action that defines the cycle (S2); and determining a time period between the identified first movement and the identified second movement to measure the productivity (S3).

Description

TECHNICAL FIELD

[0001] The present invention relates broadly, but not exclusively, to a method, device(s) and program(s) for measuring productivity. BACKGROUND

[0002] For manufacturers, cycle time is an important indicator to measure the productivity of their assembly lines.

[0003] One cycle on each workstation usually includes a series of actions such as installing components on a board, tightening screws or placing packaging covers, etc.

[0004] LIST OF CITATIONS

[0005] PATENT LITERATURE

[0006] PTL 1: International Patent Publication No. WO2018 / 191555A1 SUMMARY

[0007] TECHNICAL PROBLEM

[0008] Conventionally, cycle time is manually measured by a line manager using a stopwatch. In this case, since the measurement is made by sampling, it is difficult to make statistics based on long-term and continuous monitoring results.

[0009] Video analysis can help estimate cycle time instead of relying solely on manual effort. In particular, behavior analysis has the potential to detect a series of actions related to the work process in the assembly line.

[0010] The present disclosure relates to a cycle time estimation method, cycle time estimation device and program(s) for cycle time estimation using the position of hands of a factory assembly line, but its application can be extended to cover other scenarios, such as food preparation in a kitchen.

[0011] Disclosed herein are example embodiments of a device(s), method(s) and program(s) for measuring productivity that solve one or more of the above problems.

[0012] Furthermore, other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background of the disclosure.

[0013] SOLUTION TO THE PROBLEM

[0014] According to a first aspect, there is provided a method for measuring productivity, performed by a computer, the method comprising:

[0015] identifying, based on at least one image frame, a first movement, wherein the first movement matches a starting action defining a cycle of movements;

[0016] identify, based on the at least one image frame, a second movement, wherein the second movement matches an end action defining a cycle; and

[0017] determine a time period between the identified first movement and the identified second movement to measure the production rate.

[0018] According to a second aspect, there is provided an apparatus for measuring a production rate, the apparatus comprising:

[0019] at least one processor; and

[0020] at least one memory including computer program code;

[0021] the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to:

[0022] identify, based on the at least one image frame, a first movement, wherein the first movement matches a start action defining a cycle of movements;

[0023] identify, based on the at least one image frame, a second movement, wherein the second movement matches an end action defining a cycle; and

[0024] determine a time period between the identified first movement and the identified second movement to measure the production rate.

[0025] According to a third aspect, there is provided a non-transitory computer readable medium storing a program for measuring a production rate, the program causing a computer to at least:

[0026] identify, based on the at least one image frame, a first movement, wherein the first movement matches a start action defining a cycle of movements;

[0027] identify, based on the at least one image frame, a second movement, wherein the second movement matches an end action defining a cycle; and

[0028] determine a time period between the identified first movement and the identified second movement to measure the production rate. BRIEF DESCRIPTION OF DRAWINGS

[0029] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various example embodiments and together with the description given above and the claims serve to explain the principles and advantages of various example embodiments. Figures 1 to 8 The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various example embodiments and together with the description given above and the claims serve to explain the principles and advantages of various example embodiments.

[0030] The example embodiments will be better understood and readily apparent to one of ordinary skill in the art from the following written description taken in conjunction with the accompanying drawings.

[0031] [ Figure 1 ] Figure 1 A system for measuring productivity is shown in accordance with aspects of the disclosure.

[0032] [ Figure 2 ] Figure 2 A method of measuring productivity is shown in accordance with example embodiments.

[0033] [ Figure 3 ] Figure 3 How interpolation is performed to fill gaps of missing movements is shown.

[0034] [ Figure 4 ] Figure 4 How various ground truths are received in accordance with example embodiments is described.

[0035] [ Figure 5 ] Figure 5 How the various ground truths received are averaged to obtain a ground truth is described.

[0036] [ Figure 6 ] Figure 6 Major components of a method of measuring productivity are shown.

[0037] [ Figure 7 ] Figure 7 Major components of an apparatus for measuring productivity in accordance with example embodiments are shown.

[0038] [ Figure 8 ] Figure 8 An example computing device that can be used to perform a method of measuring productivity is shown. DETAILED DESCRIPTION

[0039] Term Description

[0040] Subject - A subject can be any suitable type of entity, which can include a person, worker, and user.

[0041] The term target or target subject is used herein to identify a person, user, or worker of interest. The target subject can be a target subject selected by user input or identified as a target subject of interest.

[0042] Subject or identified subject as used herein relates to a person (e.g., a partner or a person with a similar skill set) that is related to the target subject. For example, in the context of measuring productivity, a subject is a person that can be considered to have a similar skill set or experience as the target.

[0043] A user registered to the productivity measurement server will be referred to as a registered user. A user not registered to the productivity measurement server will be referred to as an unregistered user. A user can obtain productivity measurements of any subject.

[0044] Productivity measurement server - The productivity measurement server is a server that hosts a software application for receiving inputs, processing data, and objectively providing graphical representations. The productivity measurement server communicates with any other server (e.g., remote assistance server) to manage requests. The productivity measurement server communicates with the remote assistance server to receive benchmark true values or predetermined movements. The productivity measurement server can use various different protocols and processes to manage data and provide graphical representations.

[0045] The productivity measurement server is typically managed by a provider, which can be an entity (e.g., a company or organization) whose operations are process requests, manage data, and receive / display graphical representations useful to the situation. The server can include one or more computing devices for processing graphical representation requests and providing customizable services according to the situation.

[0046] Productivity measurement account - The productivity measurement account is an account of a user registered to the productivity measurement server. In some cases, the productivity measurement account does not require the use of a remote assistance server. The productivity measurement account includes details of the user (e.g., name, address, vehicle, etc.). An indicator of productivity is the cycle time, which is the period of time between the identified first movement and the second movement pair.

[0047] The productivity measurement server manages the productivity measurement account of the user and the interactions between the user and other external servers, as well as the data exchanged.

[0048] Detailed description

[0049] Where reference is made in any one or more of the accompanying drawings to steps and / or features having the same reference numerals, those steps and / or features have for the purposes of this description the same function(s) or operation(s) unless the contrary intention appears otherwise.

[0050] It should be noted that the discussion contained in the Background section and above relating to prior art arrangements relates to discussion of devices that form part of the common general knowledge by use of the devices. This should not be construed as the inventor(s) or patent applicant(s) admitting that these devices form part of the common general knowledge.

[0051] System 100

[0052] Figure 1A block diagram of a system 100 for measuring productivity of a target is shown. The system 100 includes a requester device 102, a productivity measurement server 108, a remote assistance server 140, remote assistance hosts 150A to 150N, and sensors 142A to 142N.

[0053] The requester device 102 is in communication with the productivity measurement server 108 and / or the remote assistance server 140 via connections 116 and 121, respectively. The connections 116 and 121 can be wireless (e.g., via NFC communication, Bluetooth (TM), etc.) or through a network (e.g., the Internet). The connections 116 and 121 can also be connections of a network (e.g., the Internet).

[0054] The productivity measurement server 108 is also in communication with the remote assistance server 140 via a connection 120. The connection 120 can be through a network (e.g., a local area network, a wide area network, the Internet, etc.). In one arrangement, the productivity measurement server 108 and the remote assistance server 140 are combined, and the connection 120 can be an interconnect bus. The productivity measurement server 108 can access a database 109 via a connection 118. The database 109 can store various data processed by the productivity measurement server 108.

[0055] The remote assistance server 140, in turn, is in communication with the remote assistance hosts 150A to 150N via respective connections 122A to 122N. The connections 122A to 122N can be of a network (e.g., the Internet).

[0056] The remote assistance hosts 150A to 150N are servers. The term host is used herein to distinguish the remote assistance hosts 150A to 150N from the remote assistance server 140. The remote assistance hosts 150A to 150N are collectively referred to herein as remote assistance hosts 150, while a remote assistance host 150 refers to one of the remote assistance hosts 150. The remote assistance hosts 150 can be combined with the remote assistance server 140.

[0057] In one example, the remote assistance hosts 150 can be managed by a factory, and the remote assistance server 140 is a central server that manages productivity at an organizational level and decides which one of the remote assistance hosts 150 to forward data or retrieve data (similar to image input). The remote assistance hosts 150 can access the database 109 via a connection 119. The database 109 can store various data processed by the remote assistance hosts 150.

[0058] The sensors 142A-142N are connected to the remote assistance server 140 or the productivity measurement server 108 via respective connections 144A-144N or 146A-146N. The sensors 142A-142N are collectively referred to herein as the sensors 146A-146N. The connections 144A-144N are collectively referred to herein as the connections 144, and a connection 144 refers to one of the connections 144. Similarly, the connections 146A-146N are collectively referred to herein as the connections 146, and a connection 146 refers to one of the connections 146. The connections 144 and 146 can be wireless (e.g., via NFC communication, Bluetooth, etc.) or over a network (e.g., the Internet). The sensors 142 can be one of an image capture device, a video capture device, and a motion sensor, and can be configured to send inputs to at least one of the productivity measurement server 108 depending on the type of input.

[0059] In the illustrative example embodiments, each of the following provides an interface to enable communication with other connected devices 102 and 142 and / or servers 108, 140, and 150: the devices 102 and 142; and the servers 108, 140, and 150. Such communication is facilitated by an application programming interface ("API"). Such an API can be part of a user interface, which can include a graphical user interface (GUI), a web-based interface, a programming interface such as an application programming interface (API), and / or a set of remote procedure calls (RPCs) corresponding to interface elements, a messaging interface corresponding to messages of a communication protocol, and / or suitable combinations thereof.

[0060] The use of the term "server" herein can refer to a single computing device or multiple interconnected computing devices that operate together to perform a particular function. That is, a server can be contained in a single hardware unit or distributed across several or many different hardware units.

[0061] Remote assistance server 140

[0062] The remote assistance server 140 is associated with an entity (e.g., a factory or company or organization or host of a service). In one arrangement, the remote assistance server 140 is owned and operated by the entity that operates the server 108. In such an arrangement, the remote assistance server 140 can be implemented as part of the server 108 (e.g., computer program modules, computing devices, etc.).

[0063] The remote assistance server 140 can also be configured to manage registration of users. A registered user has a contact tracking account (see discussion above) that includes user details. The registration step is called login. A user can use a requester device 102 to perform a login to the remote assistance server 140.

[0064] The functionality of accessing the remote assistance server 140 does not require a productivity measurement account of the remote assistance server 140. However, some functionality is available to registered users. For example, graphical representations of target subjects and potential subjects can be displayed in other jurisdictions. These additional functionalities will be discussed below.

[0065] The login process for a user is performed by the user through one of the requestor devices 102. In one arrangement, the user downloads an application (which includes an API to interact with the remote assistance server 140) to the sensor 142. In another arrangement, the user accesses a website (which includes an API to interact with the remote assistance server 140) on the requestor device 102.

[0066] The registration details include, for example, the user's name, the user's address, emergency contacts, or other important information, and the sensor 142 that is authorized to update the remote assistance account, etc.

[0067] After login, the user will have a contact tracking account that stores all the details.

[0068] Requestor device 102

[0069] The requestor device 102 is associated with a subject (or requestor), which is the party that initiates a contact tracking request at the requestor device 102. The requestor can be a relevant member of the public whose assistance is required to obtain the necessary data to obtain a graphical representation of a network graph. The requestor device 102 can be a computing device, such as a desktop computer, an interactive voice response (IVR) system, a smartphone, a laptop, a personal digital assistant computer (PDA), a portable computer, a tablet, etc.

[0070] In one example arrangement, the requestor device 102 is a watch or similar wearable computing device, and is equipped with a wireless communication interface.

[0071] Productivity measurement server 108

[0072] The productivity measurement server 108 is as described above in the terminology section.

[0073] The productivity measurement server 108 is configured to handle processes related to determining a time period between an identified first movement and an identified second movement to measure productivity.

[0074] Remote assistance host 150

[0075] The remote assistance host 150 is a server associated with an entity (e.g., a company or organization) that manages (e.g., establishes, governs) productivity information regarding information about subjects or members of the organization.

[0076] In one arrangement, the entity is an organization. Thus, each entity operates the remote assistance host 150 to manage resources of the entity. In one arrangement, the remote assistance host 150 receives an alert signal that a target subject is in motion. The remote access host 150 can then arrange to send resources to a location identified by location information contained in the alert signal. For example, the host can be a host configured to obtain relevant video or image input for processing.

[0077] Advantageously, such information is valuable for detecting exact start and end times of cycle time estimates of a factory assembly line. The present disclosure uses the correlation between the position of the hand and the start / end times of the cycle. In this way, more accurate estimates of cycle time can be obtained.

[0078] The position of the hand is more suitable for identifying the start / end times of the cycle in a factory setting, as objects move from left to right or right to left on a belt conveyor at the assembly line. Thus, the actual position can generate better features for these cases.

[0079] However, the time series of the actual position will be different from traditional techniques that utilize distance instead of position, resulting in more false matches by the pattern matching (given a query sequence, find similar sequences in a target dataset). Also, the detection of the hand can incorrectly detect the position of the hand, which in turn results in a higher number of false matches.

[0080] Therefore, to make the pattern matching more accurate, the present disclosure identifies which hands have been detected, and subsequently interpolates (replaces the missing positions with substitute values) the missing data due to missed detections or occlusions. Alternatively or additionally, the present disclosure collects sequences that correspond to ground truth values on a sample dataset. For the ground truth values, the start and end actions that constitute a work cycle are predefined.

[0081] For example, a user can define the start and end actions of a work cycle (each action comprising a continuous sequence of movement or predetermined movement of a hand) by providing timestamps of when these actions occur on a video clip obtained from a camera of interest. In an example embodiment, two sets of predetermined movements are defined to cover the start and end of the cycle.

[0082] At the same time, the expected number of hands within the camera view is also specified. This value is directly related to the number of workers / operators expected to be visible working in the camera view (e.g., if there are two operators, then four hands are expected; if there is one operator, then two hands are expected).

[0083] Alternatively or additionally, the present disclosure generates an average query sequence from the collected sequences, so that the query sequence can be used as an input query to find similar sequences in a target dataset that represent the start and end times.

[0084] Sensor 142

[0085] The sensor 142 is associated with a user associated with the requester device 102. More details on how to utilize the sensor will be provided below.

[0086] Figure 2 A method 200 of measuring production rate is demonstrated in accordance with example embodiments of the present disclosure. As shown at 202, hand detection is performed to detect hands on a given image frame, and a time series of hand positions with corresponding frame numbers is generated in 206.

[0087] By taking the positions of the detected hands in the first process 202, interpolation 214 is performed. Specifically, the method includes detecting the number of hands in a frame, and comparing the number of detected hands with the number of expected hands in the frame to detect missing hands in the frame. If missing hands in the frame are detected, interpolation 214 is performed to interpolate the missing hands in the frame, thereby generating a time series of hand positions with hand recognition (which can identify the target) and corresponding frame numbers as shown at 220.

[0088] For example, the interpolation will look at the number of expected hands in the camera view (specified in the ground truth) and compare it with the number of hands detected for each video frame. For example, if there are four hands expected, but only three hands in a given frame, interpolation is performed to fill in the missing data (i.e. missing positions of the missing hands in the missing period with at least one missing hand), making the data “complete” by looking at the average historical position of the hands corresponding to the missing hands.

[0089] The sequence matching 224 checks which one in the second output (i.e. time series of hand positions 220) matches the given query sequence in 218 to detect the start and end time, and then outputs the matching sequence 208 with frame numbers.

[0090] On the third output, the cycle time estimation 216 estimates each cycle of the assembly line, and outputs the estimated cycle time as shown at 222. In various example embodiments, the cycle time is the time period between a pair of identified first and second movements. The first movement corresponds to the beginning of the cycle. On the other hand, the second movement corresponds to the end of the cycle.

[0091] To provide the query sequence as input to the third process 224, the query sequence generation 210 generates a query 218 that detects the start and end times on a given input data based on the given ground truth real values (or predetermined movements) 204 of the start and end times specified on the sample dataset. The query sequence based on the given ground truth real values of the start times corresponds to the first sequence of the hand(s) position, which is the start action in the work of the worker or the start action in the operation of the operator. On the other hand, the query sequence based on the given ground truth real values of the end times corresponds to the second sequence of the hand(s) position, which is the end action in the work or the end action in the operation.

[0092] Figure 3 It is shown how to use the predetermined movements when averaging the ground truth real values. The start and end actions that constitute the work cycle are predefined.

[0093] In various example embodiments, the user will define the start and end actions of the work cycle by providing the timestamps of when these actions occur on the video clip obtained from the camera of interest. Each action comprises a continuous sequence of hand movements or predetermined movements. Two sets of predetermined movements are defined to cover the start and end of the cycle.

[0094] In example embodiments, the number of expected hands within the camera view is also specified. This value is directly related to the number of workers or operators of the work expected to be visible in the camera view. For example, if there are two operators, then four hands are expected. If there is one operator, then two hands are expected.

[0095] For each video frame, the interpolation will look at the number of expected hands in the camera view (specified in the ground truth real values) and compare it to the number of hands detected for each video frame. For example, if four hands are expected but a given frame has only three hands, then interpolation is performed to fill in the missing data (i.e. the missing position of the missing hand at least one missing hand in the missing period) by making the data “complete” by looking at the average historical position of the hand corresponding to the missing hand.

[0096] Each of 301 and 302 shows a possible set of predetermined movements or ground truth real values. In 303, certain movements can be detected and there can be missing movements of hands 310. As shown in 304, 305, and 306, interpolation can be performed to fill in the gaps (e.g. idx: 1, x: 488, y: 323, idx: 1, x: 489, y: 324, idx: 1, x: 491, y: 322) otherwise there will be missing data. Missing data will negatively impact the time series sequence matching due to the increase in false matches.

[0097] The output of averaging the benchmark ground truths is a pair of average predetermined movements that represent the average movement of the action, which is coupled with a known technique called dynamic time warping to match the similar first movement(s) and second movement(s) among multiple movements (detected hand positions converted into time series data) obtained from the same camera view. This can be obtained after the detected hand position data has been "pretreated" by interpolation.

[0098] This coupling allows for variations in the sequence of movements that constitute the start or end action of a cycle of work. For example. The first average predetermined movement can include a movement up followed by a movement down, but the real-world first movement can include a movement up, a movement right, followed by a movement down. In this case, the real-world first movement will still match the first average predetermined movement, despite the obvious differences. Similarly, it is still possible to match if a certain movement is omitted.

[0099] Figure 4 It is described how example implementations according to the present disclosure receive various benchmark ground truths. In Figure 4 In the middle, each time series pattern 402, 404, 406, 408, and 410 shown in 400 is obtained from a user-defined benchmark ground truth or predetermined movement, and is an example of a sequence corresponding to the start of a cycle. To measure the productivity of a target, time series patterns related to subjects with similar experiences are retrieved.

[0100] Figure 5 It is described how to obtain a benchmark ground truth 502 by averaging the received various benchmark ground truths 500. Sequences 402, 404, 406, and 408 are averaged to obtain a final query sequence, which is used as input for sequence matching in order to identify similar sequences within a target dataset.

[0101] Figure 6 The main components of the method of measuring productivity are shown. According to various example embodiments, there is a method of measuring productivity. The method comprises: identifying a first movement based on at least one image frame, wherein the first movement matches a starting action that defines a cycle of movements (SI); identifying a second movement based on at least one image frame, wherein the second movement matches an ending action that defines the cycle (S2); and determining a time period between the identified first movement and the identified second movement to measure the productivity (S3).

[0102] The method further comprises detecting a number of hands in the image frame; comparing the detected number of hands to a number of expected hands in the frame to detect at least one missing hand in the frame; and in response to detecting a missing hand in the frame, performing an interpolation of movement of the missing hand in the frame. Thus, movement of a missing hand can be compensated for even if the hand is not captured in the frame. Thus, the first movement and / or the second movement can be identified in this case.

[0103] Further, the interpolation is performed by filling in missing positions of the missing hand using average historical positions of the hand corresponding to the missing hand in missing periods of the missing hand.

[0104] Further, the method comprises generating a first sequence of positions of the hand corresponding to a start action; and generating a second sequence of positions of the hand corresponding to an end action; wherein the identification of the first movement comprises identifying the first movement matching the first sequence; and the identification of the second movement comprises identifying the second movement matching the second sequence.

[0105] Further, the generation of the first sequence can comprise averaging a plurality of sequences of positions of the hand corresponding to a start action of a cycle of movement to generate the first sequence. The generation of the second sequence can comprise averaging a plurality of sequences of positions of the hand corresponding to an end action of a cycle of movement to generate the second sequence.

[0106] In the method according to the above, the identification of the first movement comprises identifying whether the first movement is performed by a right hand or a left hand.

[0107] Further, the identification of the second movement can be performed when it is identified that the first movement is performed by a right hand. In this case, the right hand can be a dominant hand of the worker / operator.

[0108] Alternatively, the identification of the second movement can be performed when it is identified that the first movement is performed by a left hand. In this case, the left hand can be a dominant hand of the worker / operator.

[0109] Figure 7 The main components of an apparatus for measuring productivity according to an example embodiment are shown. The apparatus 70 comprises at least one processor 71 and at least one memory 72 including a computer program code. The at least one memory 72 and the computer program code are configured to, with the at least one processor 71, cause the apparatus to perform the above described method.

[0110] Figure 8 An example computing device 1300, hereinafter referred to interchangeably as computer system 1300, is described in which one or more such computing devices 1300 can be used to perform the methods shown above. The example computing device 1300 can be used to implement Figure 1The illustrated system 100. The following description of the computing device 1300 is provided as an example only and is not intended to be limiting.

[0111] As Figure 8 shown, the example computing device 1300 includes a processor 1307 for executing software routines. Although a single processor is shown for purposes of clarity, the computing device 1300 can also include multiple processors. The processor 1307 is connected to a communication infrastructure 1306 for communication with other components of the computing device 1300. The communication infrastructure 1306 can include, for example, a communications bus, cross-over bar, or network.

[0112] The computing device 1300 also includes a main memory 1308, such as random access memory (RAM), and a secondary memory 1310. The secondary memory 1310 can include, for example, a storage drive 1312, which can be a hard disk drive, a solid-state drive, or a hybrid drive, and / or a removable storage drive 1317, which can include a tape drive, an optical drive, a solid-state memory drive (such as a USB flash drive, a flash memory device, a solid-state drive, or a memory card), and the like. The removable storage drive 1317 reads from and / or writes to a removable storage media 1377 in a well-known manner. The removable storage media 1377 can include a tape, an optical disk, a non-volatile memory storage medium, and the like, which is read by and written to by the removable storage drive 1317. The relevant skill in the art will appreciate that the removable storage media 1377 includes computer readable storage media on which computer-executable program code instructions and / or data are stored.

[0113] In alternative implementations, the secondary memory 1310 can additionally or alternatively include other similar components for allowing computer programs or other instructions to be loaded into the computing device 1300. Such components can include, for example, a removable storage unit 1322 and an interface 1314. Examples of the removable storage unit 1322 and the interface 1314 include a program cartridge and cartridge interface (such as that found in video game console devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a removable solid-state storage drive (such as a USB flash drive, a flash memory device, a solid-state drive, or a memory card), and other removable storage units 1322 and interfaces 1314, which allow software and data to be transferred from the removable storage unit 1322 to the computing device 1300.

[0114] The computing device 1300 also includes at least one communication interface 1327. The communication interface 1327 allows software and data to be transferred between the computing device 1300 and external devices via a communication path 1326. In various example embodiments, the communication interface 1327 allows data to be transferred between the computing device 1300 and a data communication network, such as a public data or a proprietary data communication network. The communication interface 1327 can be used to exchange data with different computing devices 1300 that form part of an interconnected computer network. Examples of the communication interface 1327 can include a modem, a network interface (such as an Ethernet card), a communication port (such as a serial, parallel, printer, GPIB, IEEE 1394, RJ45, USB), an antenna with associated circuitry, and the like. The communication interface 1327 can be wired or wireless. Software and data transferred via the communication interface 1327 are in the form of signals, which can be electronic, electromagnetic, optical or other signals capable of being received by the communication interface 1327. These signals are provided to the communication interface via the communication path 1326.

[0115] As shown in Figure 8 The computing device 1300 also includes a display interface 1302 that forwards graphics, text, and other data from the communication infrastructure 1302 (or from a frame buffer not shown) for display on the associated display 1350, and an audio interface 1352 that forwards audio and other data from the communication infrastructure 1302 for aural presentation via the associated speaker(s) 1357.

[0116] As used herein, the term "computer program product" can refer, at least partially, to a removable storage medium 1377, a removable storage unit 1322, a hard disk installed in a storage drive 1312, or a carrier wave that transmits software to the communication interface 1327 over the communication path 1326 (wireless link or cable). A computer-readable storage medium refers to any non-transitory, non-volatile tangible storage medium that provides recorded instructions and / or data to the computing device 1300 for execution and / or processing. Examples of such storage media include magnetic tape, CD-ROM, DVD, Blu-ray(TM) disc, hard disk drive, ROM or integrated circuit, solid-state storage drive (such as a USB flash drive, flash memory device, solid-state drive, or memory card), hybrid drive, magneto-optical disk, or computer readable card such as a PCMCIA card, and the like, whether or not such devices are internal or external to the computing device 1300. Examples of transitory or non-tangible computer-readable transmission media that can also participate in the provision of software, application programs, instructions and / or data to the computing device 1300 include wireless radio or infrared transmission channels as well as a network connection to another computer or network device, and the Internet or intranet composition of information such as email transmissions and information recorded on websites, and the like.

[0117] Computer programs (also referred to as computer program code) are stored in main memory 1308 and / or secondary memory 1310. Computer programs can also be received via communication interface 1327. Such computer programs, when executed, enable the computing device 1300 to perform one or more features of the example embodiments discussed herein. In various example embodiments, the computer programs, when executed, enable the processor 1307 to perform the features of the example embodiments described above. Accordingly, such computer programs represent controllers of the computer system 1300.

[0118] The software can be stored in a computer program product and loaded into the computing device 1300 using removable storage drive 1317, storage drive 1312 or interface 1314. The computer program product can be a non-transitory computer readable medium. Alternatively, the computer program product can be downloaded to the computer system 1300 over the communication path 1326. The software, when executed by the processor 1307, causes the computing device 1300 to perform the functions as described above.

[0119] It should be understood that Figure 8 The example embodiments are presented by way of example only, to explain the operation and structure of the system 100. Thus, in some example embodiments, one or more features of the computing device 1300 can be omitted. Further, in some example embodiments, one or more features of the computing device 1300 can be combined together. Additionally, in some example embodiments, one or more features of the computing device 1300 can be split into one or more component parts.

[0120] Those skilled in the art will understand that numerous modifications and / or changes can be made to the specific example embodiments as shown and described without departing from the spirit or ambit of the present application as broadly described. Accordingly, the present example embodiments are to be considered in all respects as illustrative and not restrictive.

[0121] This application is based upon and claims priority to Singapore Patent Application No. 10202109093T, filed on August 19, 2021, the disclosure of which is incorporated herein in its entirety by reference.

[0122] Supplementary notes

[0123] All or part of the above disclosed example embodiments can be described as, but not limited to, the following supplementary notes.

[0124] (Supplementary note 1)

[0125] A method, performed by a computer, for measuring productivity, comprising:

[0126] identifying a first movement based on at least one image frame, wherein the first movement matches a start action defining a cycle of movements;

[0127] identifying a second movement based on at least one image frame, wherein the second movement matches an end action defining the cycle; and

[0128] determining a time period between the identified first movement and the identified second movement to measure productivity.

[0129] (Supplementary note 2)

[0130] The method according to supplementary note 1, further comprising:

[0131] detecting a number of hands in the image frame;

[0132] comparing the detected number of hands to a number of expected hands in a frame to detect at least one missing hand in the frame; and

[0133] performing an interpolation of a movement of the missing hand in the frame in response to detecting a missing hand in the frame.

[0134] (Supplementary note 3)

[0135] The method according to supplementary note 2, wherein the interpolation is performed by filling in missing positions of the missing hand using an average historical position of a hand corresponding to the missing hand in a missing period of the missing hand.

[0136] (Supplementary note 4)

[0137] The method according to supplementary note 1, further comprising:

[0138] generating a first sequence of positions of a hand corresponding to the start action; and

[0139] generating a second sequence of positions of a hand corresponding to the end action; wherein

[0140] the identifying of the first movement comprises identifying the first movement matching the first sequence; and

[0141] the identifying of the second movement comprises identifying the second movement matching the second sequence.

[0142] (Supplementary note 5)

[0143] The method according to supplementary note 4, wherein:

[0144] The generating of the first sequence comprises averaging a plurality of sequences of positions of the hand corresponding to the start action of the cycle of the movement to generate the first sequence; and

[0145] The generating of the second sequence comprises averaging a plurality of sequences of positions of the hand corresponding to the end action of the cycle of the movement to generate the second sequence.

[0146] (Supplementary note 6)

[0147] An apparatus for measuring productivity, the apparatus comprising:

[0148] at least one processor; and

[0149] at least one memory including computer program code; wherein

[0150] the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:

[0151] identify, based on at least one image frame, a first movement, wherein the first movement matches a start action defining a cycle of movements;

[0152] identify, based on at least one image frame, a second movement, wherein the second movement matches an end action defining the cycle; and

[0153] determine a time period between the identified first movement and the identified second movement to measure productivity.

[0154] (Supplementary note 7)

[0155] The apparatus according to supplementary note 6, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:

[0156] detect a number of hands in the image frame;

[0157] compare the detected number of hands to a number of expected hands in the frame to detect at least one missing hand in the frame; and

[0158] in response to detecting the missing hand in the frame, perform interpolation of a movement of the missing hand in the frame.

[0159] (Supplementary note 8)

[0160] The apparatus according to supplementary note 7, wherein the interpolation is performed by filling in missing positions of the missing hand using an average historical position of a hand corresponding to the missing hand in a missing period of the missing hand.

[0161] (Supplementary note 9)

[0162] The apparatus according to supplementary note 6, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:

[0163] generate a first sequence of positions of the hand corresponding to the start action;

[0164] generate a second sequence of positions of the hand corresponding to the end action;

[0165] identify the first movement matching the first sequence; and

[0166] identify the second movement matching the second sequence.

[0167] (Supplementary note 10)

[0168] The apparatus according to supplementary note 6 or 9, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:

[0169] average a plurality of sequences of positions of the hand corresponding to a start action of the cycle of movements to generate the first sequence; and

[0170] average a plurality of sequences of positions of the hand corresponding to an end action of the cycle of movements to generate the second sequence.

[0171] (Supplementary note 11)

[0172] A non-transitory computer readable medium storing a program for measuring productivity, wherein the program causes a computer to at least:

[0173] identify a first movement based on at least one image frame, wherein the first movement matches a start action defining a cycle of movements;

[0174] identify a second movement based on at least one image frame, wherein the second movement matches an end action defining the cycle of movements; and

[0175] determine a time period between the identified first movement and the identified second movement to measure productivity.

[0176] List of figures

[0177] 70 apparatus

[0178] 71 processor

[0179] 72 memory

[0180] 100 system

[0181] 102 requester device

[0182] 108 yield measurement server

[0183] 109 database

[0184] 140 remote assistance server

[0185] 142A-142N sensors

[0186] 150A-150N remote assistance host

Claims

1. A method, performed by a computer, for measuring productivity, comprising: identifying, based on at least one image frame, a first movement, wherein the first movement matches a start action that defines a cycle of movements; identifying, based on at least one image frame, a second movement, wherein the second movement matches an end action that defines the cycle; and determining a time period between the identified first movement and the identified second movement to measure productivity; detecting a number of hands in the image frame; comparing the detected number of hands to an expected number of hands in the frame to detect at least one missing hand in the frame; and in response to detecting the missing hand in the frame, performing an interpolation of a movement of the missing hand in the frame. The interpolation is performed by filling in missing positions of the missing hand using average historical positions of hands corresponding to the missing hand in missing time periods of the missing hand.

2. The method of claim 1, wherein, 3. The method of claim 1, further comprising: generating a first sequence of positions of hands corresponding to the start action; and generating a second sequence of positions of hands corresponding to the end action; wherein the identifying of the first movement comprises identifying the first movement that matches the first sequence; and the identifying of the second movement comprises identifying the second movement that matches the second sequence.

4. The method of claim 3, wherein: the generating of the first sequence comprises averaging a plurality of sequences of positions of hands corresponding to a start action of a cycle of movements to generate the first sequence; and the generating of the second sequence comprises averaging a plurality of sequences of positions of hands corresponding to an end action of a cycle of movements to generate the second sequence.

5. An apparatus for measuring productivity, the apparatus comprising: at least one processor; and at least one memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to: identify, based on at least one image frame, a first movement, wherein the first movement matches a start action that defines a cycle of movements; identify, based on at least one image frame, a second movement, wherein the second movement matches an end action that defines the cycle; and determine a time period between the identified first movement and the identified second movement to measure productivity; detect a number of hands in the image frame; compare the detected number of hands to an expected number of hands in the frame to detect at least one missing hand in the frame; and in response to detecting the missing hand in the frame, perform an interpolation of a movement of the missing hand in the frame. The interpolation is performed by filling in missing positions of the missing hand using average historical positions of hands corresponding to the missing hand in missing time periods of the missing hand.

7. The apparatus of claim 5, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:

6. The apparatus of claim 5, wherein, ​ ​ generating a first sequence of positions of the hand corresponding to the start action; generating a second sequence of positions of the hand corresponding to the end action; identifying the first movement matching the first sequence; and identifying the second movement matching the second sequence.

8. A computer readable medium storing a program for measuring productivity, wherein the program causes a computer to at least: identify a first movement based on at least one image frame, wherein the first movement matches a start action defining a cycle of movements; identify a second movement based on at least one image frame, wherein the second movement matches an end action defining the cycle; and determine a time period between the identified first movement and the identified second movement to measure productivity; detect a number of hands in the image frame; compare the detected number of hands to an expected number of hands in the frame to detect at least one missing hand in the frame; and in response to detecting the missing hand in the frame, perform an interpolation of a movement of the missing hand in the frame.

Images