Information processing device, method, and program

Abstract

To provide a method, device, and program for estimating cycle time of a factory assembly line using positions of hands.SOLUTION: A computer disclosed herein is configured to acquire a video image, receive a setting related to the number of hands of people, acquire the number of hands of people appearing in the video image, and obtain a hand-related history according to the set number of hands and the number of hands of people appearing in the video image.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present invention relates to, but is not limited to, a method, apparatus and program for measuring productivity. [Background technology]

[0002] For manufacturers, cycle time is a key metric for measuring assembly line productivity.

[0003] One cycle at each work table is usually made up of a series of operations, such as mounting a part on the table, tightening screws, and attaching a packaging cover. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 191555 Summary of the Invention [Problem to be solved by the invention]

[0005] Conventionally, the cycle time is measured manually by the line manager using a stopwatch. In such cases, since the measurement is performed by sampling, it is difficult to obtain statistics based on long-term and continuous monitoring results.

[0006] Video analytics can help estimate cycle times rather than relying solely on manual labor. In particular, motion analysis has the potential to detect sequences of actions associated with assembly line work processes.

[0007] The present disclosure relates to a method for estimating cycle time using hand positions, a cycle time estimation device, and a cycle time estimation program for a factory assembly line, but its application can be extended to other situations, such as food preparation in a kitchen.

[0008] Disclosed herein are exemplary embodiments of an apparatus, method and program for measuring productivity that solve one or more of the above problems.

[0009] Furthermore, other desirable features and characteristics will become apparent from the subsequent detailed description and appended claims, taken in conjunction with the accompanying drawings and this background of the disclosure. [Means for solving the problem]

[0010] The information processing device according to the present disclosure includes: a means for acquiring the image; means for receiving a setting regarding the number of hands of the person; A means for obtaining the number of hands of a person appearing in the video; The camera is provided with a means for acquiring a history relating to hands according to the relationship between the set number of hands and the number of hands of a person appearing in the video.

[0011] The method according to the present disclosure includes: Get the video and Accepts settings for the number of hands on the person, Obtain the number of hands of the person in the video, A history of hands is acquired according to the relationship between the set number of hands and the number of hands of the person appearing in the video.

[0012] The program according to the present disclosure is for a computer to: acquiring a video; accepting a setting regarding the number of hands of the person; obtaining the number of hands of a person appearing in the video; and a step of acquiring a history relating to hands according to the relationship between the set number of hands and the number of hands of a person appearing in the video. [Brief explanation of the drawings]

[0013] In the accompanying Figures 1-8, like reference numerals refer to identical or functionally similar components throughout the separate figures, and these figures, together with the following detailed description, are incorporated into or form a part of this specification, and serve to illustrate various exemplary embodiments and to explain, by way of non-limiting example, various principles and advantages based on the exemplary embodiments.

[0014] Exemplary embodiments will be better understood and become more readily apparent to those skilled in the art from the following illustrative description and drawings. [Figure 1] FIG. 1 illustrates a system for measuring productivity according to one embodiment of the present disclosure. [Figure 2] FIG. 2 illustrates a method for measuring productivity according to an exemplary embodiment. [Figure 3] FIG. 3 shows a method of imputation to compensate for the missing movement deficit. [Figure 4] FIG. 4 illustrates a method for receiving various ground truths according to an example embodiment. [Figure 5] Figure 5 shows how the different received ground truths are averaged to obtain the ground truth. [Figure 6] Figure 6 shows the main components of a method for measuring productivity. [Figure 7] FIG. 7 shows the main components of an apparatus for measuring productivity according to an exemplary embodiment. [Figure 8] FIG. 8 illustrates an exemplary computer device that can be used to implement the method for measuring productivity. DETAILED DESCRIPTION OF THE INVENTION

[0015] Terminology Subject—Subject may refer to any suitable type of entity, including, for example, a person, a worker, and a user.

[0016] The terms target or target person are used herein to identify a person, user, or worker of interest. A target person may be a person selected by user input or a person identified as a target.

[0017] The term subject or identified subject herein refers to a person related to the subject (e.g., a partner or a person with similar skills). For example, in the context of measuring productivity, a subject is someone who is believed to have similar skills or experience as the target.

[0018] A user who is registered with the productivity measurement server is called a registered user. A user who is not registered with the productivity measurement server is called a non-registered user. A user can obtain productivity measurements of any subject.

[0019] Productivity Measurement Server - A productivity measurement server is a server that has a software application program that receives input, processes data, and objectively provides a graphical display. The productivity measurement server communicates with other servers (e.g., remote assistance servers) to manage requests. The productivity measurement server communicates with the remote assistance server to receive ground truth or predetermined actions. The productivity measurement server may use a variety of protocols and procedures to manage data and provide a graphical display.

[0020] A productivity measurement server is typically managed by a provider, which is an entity (e.g., a company or organization) that processes requests, manages data, and receives / displays contextually useful graphical displays. The server may include one or more computing devices used to process requests for graphical displays and provide contextually customizable services.

[0021] Productivity Measurement Account - A productivity measurement account is a user's account that is registered with the productivity measurement server. In certain circumstances, a productivity measurement account is not required to use the remote assistance server. A productivity measurement account includes user details (e.g., name, address, vehicle, etc.). A productivity indicator is cycle time, which is the period between a specified pair of first and second movements.

[0022] Productivity Measurement manages the user's productivity measurement account and also manages the interactions between the user and other external servers, along with the data exchanged.

[0023] Detailed Description Where steps and / or functions having the same reference numbers are referenced in any one or more of the accompanying drawings, these steps and / or functions have the same function or process for the purposes of this description, unless indicated to the contrary.

[0024] It should be noted that the descriptions contained in the "Background" section and the above discussion of prior art configurations are directed to descriptions of devices using which form public knowledge, and such should not be construed as a representation by the inventors or patent applicants that such devices form part of the general knowledge in the art in any way.

[0025] System 100 1 shows a block diagram of a system 100 for measuring a target's productivity. The system 100 includes a requester device 102, a productivity measurement server 108, a remote assistance server 140, remote assistance hosts 150A-150N, and sensors 142A-142N.

[0026] The requester device 102 communicates with the productivity measurement server 108 and / or the remote assistance server 140 via connections 116 and 121, respectively. The connections 116 and 121 may be wireless (e.g., via NFC communication, Bluetooth, etc.) or may be over a network (e.g., the Internet, etc.). The connections 116 and 121 may also be over a network (e.g., the Internet, etc.).

[0027] The productivity measurement server 108 further communicates with a remote assistance server 140 via a connection 120. The connection 120 may be via a network (e.g., a local area network, a wide area network, the Internet, etc.). In some configurations, the productivity measurement server 108 and the remote assistance server 140 are combined, and the connection 120 may be an interconnected bus. The productivity measurement server 108 can access a database 109 via a connection 118. The database 109 can store various data processed by the productivity measurement server 108.

[0028] The remote assistance server 140 then communicates with the remote assistance hosts 150A-150N via respective connections 122A-122N, which may be a network (eg, the Internet, etc.).

[0029] The remote support hosts 150A to 150N are servers. Here, the term host is used to distinguish the remote support hosts 150A to 150N from the remote support server 140. Here, the remote support hosts 150A to 150N are collectively referred to as remote support hosts 150, and the remote support host 150 indicates one of the remote support hosts 150. The remote support host 150 can be combined with the remote support server 140.

[0030] In one example, remote support hosts 150 are managed at a factory, and remote support server 140 is a central server that manages productivity at an organizational level and determines which remote support hosts 150 transfer data or acquire data, such as image capture. Remote support hosts 150 can access database 109 via connection 119. Database 109 can store various data processed by remote support hosts 150.

[0031] The sensors 142A-142N are connected to the remote assistance server 140 or the productivity measurement server 108 via individual connections 144A-144N or 146A-146N. The sensors 142A-142N are collectively referred to herein as sensors 146A-146N. The connections 144A-144N are collectively referred to herein as connections 144, where connection 144 denotes one of the connections 144. Similarly, the connections 146A-146N are collectively referred to herein as connections 146, where connection 146 denotes one of the connections 146. The connections 144 and 146 may be wireless (e.g., via NFC communication, Bluetooth, etc.) or may be connected via a network (e.g., the Internet, etc.). The sensors 142 may be imaging devices, video capture devices, or motion sensors, and depending on their type, may be configured to send input to at least one of the productivity measurement servers 108 .

[0032] In an exemplary embodiment, each device 102 and 142 and server 108, 140, and 150 provides an interface that enables communication with other connected devices 102 and 142 and / or servers 108, 140, and 150. Such communication is facilitated by an application programming interface ("API"). Such API may be part of a user interface, which may include a graphical user interface (GUI), a web-based interface, a programmatic interface such as an application programming interface (API) and / or a set of remote procedure calls (RPCs) corresponding to components of the interface, a messaging interface in which components of the interface correspond to messages in a communication protocol, and / or any suitable combination thereof.

[0033] As used herein, the term "server" may refer to a single computing device or to multiple interconnected computing devices that cooperate to perform a particular function, i.e., a server may be contained within a single hardware unit or may be distributed across multiple or many different hardware units.

[0034] Remote support server 140 The remote assistance server 140 is associated with an entity (e.g., a factory, a company, an organization, a service moderator). In some configurations, the remote assistance server 140 is owned and managed by the entity that manages the server 108. In such configurations, the remote assistance server 140 may be implemented as part of the server 108 (e.g., a computer program module, a computing device, etc.).

[0035] The remote assistance server 140 may also be configured to manage user registration. A registered user has a contact tracing account (see above) that contains the user's details. The registration process is referred to as onboarding. A user may also onboard to the remote assistance server 140 using the requester device 102.

[0036] It is not necessary to have a productivity measurement account on the remote assistance server 140 to access its features. However, there are features available to registered users. For example, there may be a graphical display of targets and potential targets in other jurisdictions. These additional features are described below.

[0037] The user onboarding process is performed by the user using one of the claimant devices 102. In one configuration, the user downloads an app (including an API for communicating with the remote assistance server 140) to the sensor 142. In another configuration, the user accesses a website on the claimant device 102 (including an API for communicating with the remote assistance server 140).

[0038] Registration details may include, for example, the user's name, the user's address, emergency contact information, or other important information, such as the sensors 142 authorized to update the remote assistance account.

[0039] Once onboarded, users will have a contact tracing account that stores all their details.

[0040] Requester device 102 The requestor device 102 is associated with a subject (or requestor) who is the party to a contact tracing request initiated on the requestor device 102. The requestor may be a member of the public of interest who assists in obtaining the data necessary to obtain a graphical representation of the network graph. The requestor device 102 may be a computing device such as a desktop computer, an Interactive Voice Response (IVR) system, a smartphone, a laptop computer, a personal digital assistant computer (PDA), a mobile computer, a tablet computer, etc.

[0041] In one configuration, the claimant device 102 is a computing device within a watch or similar wearable device, and is equipped with a wireless communication interface.

[0042] Productivity Measurement Server 108 The productivity measurement server 108 is as described in the terminology section.

[0043] The productivity measurement server 108 is configured to handle the process relating to determining the time period from the identified first movement to the identified second movement and measure productivity.

[0044] Remote support host 150 The remote assistance host 150 is a server associated with an entity (eg, a company or organization) that manages (eg, generates or manages) productivity information related to information about subjects or members of the organization.

[0045] In one configuration, the entities are organizations. Accordingly, each entity manages a remote assistance host 150 for managing resources by that entity. In one configuration, the remote assistance host 150 receives an alert signal indicating that a person of interest is moving. The remote access host 150 can then be configured to send resources to a location identified by location information included in the alert signal. For example, the host can be configured to capture relevant video or image input for processing.

[0046] Such information is useful for detecting accurate start and end timings for estimating cycle time on a factory assembly line. The present disclosure utilizes the correlation between hand position and cycle start / end timing, thereby obtaining more accurate estimated cycle timing.

[0047] Hand position is better suited to identifying when a cycle begins / ends in a factory, as objects move from left to right or vice versa on an assembly line conveyor belt, so actual position can generate a better signature for these situations.

[0048] However, because actual time-series locations differ from conventional techniques that use distance instead of location, pattern matching (given a query sequence, searching for similar sequences in the target dataset) can generate many false matches. Also, hand detection can falsely detect hand locations, which can similarly increase the number of false matches.

[0049] Therefore, to make pattern matching more accurate, the present disclosure identifies which hands have already been detected and then imputes missing data due to detection failure or occlusion (replacing missing positions with substitute values). Alternatively or additionally, the present disclosure collects sequences corresponding to ground truth in a sample dataset. For the ground truth, start and end actions that define a work cycle are predefined.

[0050] For example, a user can define the start and end actions of a work cycle (each action comprising a series of hand movements or predefined movements) by providing timestamps at which these actions occur in a video clip captured from the target's camera. In one exemplary embodiment, two predefined sets of movements are defined to cover the start and end of the cycle.

[0051] At the same time, the number of expected hands in the camera view is also specified, which directly relates to the number of workers / operators expected to be working and visible in the camera view (e.g., if there are two operators, four hands are expected, if there is one operator, two hands are expected).

[0052] Alternatively or additionally, the present disclosure generates an averaged query sequence from the collected sequences and enables searching for similar sequences indicating start and end timings within the target dataset using the query sequence as an input query.

[0053] Sensor 142 The sensor 142 is associated with a user associated with the claimant device 102. The use of the sensor is described in more detail below.

[0054] 2 illustrates a method 200 for measuring productivity according to an exemplary embodiment of the present disclosure. As shown at 202, hand detection is performed to detect hands in a given image frame, and at 206, a time series of hand positions with corresponding frame numbers is generated.

[0055] By obtaining the hand positions detected in the first process 202, imputation 214 is performed. Specifically, the method includes detecting the number of hands in a frame and comparing the number of detected hands with the number of expected hands in the frame to detect undetected hands in the frame. If an undetected hand is detected in a frame, imputation 214 is performed to imput the undetected hand in the frame, and a time series of hand positions with hand identification information (which may identify the target) and corresponding frame numbers is generated, as shown in 220.

[0056] For example, imputation looks at the expected number of hands in the camera view (specified in the ground truth) and compares it to the number of detected hands in each video frame. For example, if four hands are expected but only three hands are present in a given frame, imputation is performed to fill in the missing data, i.e., at least one missing position of the undetected hand during the undetected period, and make this data "complete" by looking at the average of the historical positions of the hands corresponding to the undetected hands.

[0057] The second output, sequence matching 224, checks which of the time series hand positions 220 matches a predefined query sequence 218 to find the start and end timings, and outputs the matched sequence 208 along with the frame number.

[0058] In a third output, cycle time estimation 216 estimates each cycle of the assembly line and outputs an estimated cycle time, as shown at 222. In various exemplary embodiments, the cycle time is the time period between an identified pair of first and second moves, where the first move corresponds to the start of the cycle, while the second move corresponds to the end of the cycle.

[0059] To provide a query sequence that is input to the third process 224, the query sequence generation 210 generates a query 218 for detecting start and end timings of the given input data based on a predetermined ground truth (or predefined movements) of start and end timings specified in the sample dataset 204. The query sequence based on the predetermined ground truth of start timings corresponds to a first series of hand positions that are start movements in a worker's task or operator's operation, while the query sequence based on the predetermined ground truth of end timings corresponds to a second series of hand positions that are end movements in the task or operation.

[0060] The use of predefined movements in ground truth averaging is illustrated in Figure 3. The start and end movements that make up a work cycle are predefined.

[0061] In various exemplary embodiments, a user defines the start and end motions of a work cycle by providing timestamps at which the start and end motions occur in a video clip captured from the target camera. Each motion comprises a continuous series of hand movements or predefined movements. Two sets of predefined movements are defined to cover the start and end of the cycle.

[0062] In one exemplary embodiment, the number of hands expected in the camera view is also specified. This value directly relates to the number of workers or operators expected to be working as visible in the camera view. For example, if there are two operators, four hands can be expected. If there is one operator, two hands can be expected.

[0063] For each video frame, imputation looks at the number of expected hands in the camera view (specified in the ground truth) and compares it to the number of detected hands in each video frame. For example, if four hands are expected but there are only three hands in a given frame, imputation is performed to fill in the missing data, i.e., at least one missing position of the undetected hand during the undetected period, and make this data "complete" by looking at the average of the historical positions of the hands corresponding to the undetected hands.

[0064] Each of 301 and 302 shows a set of expected default movements or ground truths. In 303, a movement is detected and there is an undetected hand movement 310. As shown in 304, 305, and 306, imputation is performed to fill in gaps that would have been missing data (e.g., idx:1,x:488,y:323; idx:1,x:489,y:324; idx:1,x:491,y:322). Missing data has a negative impact on time-series sequence matching by increasing the number of false matches.

[0065] The output of the ground truth averaging is a set of averaged predetermined movements representative of the action, so it can be combined with a known technique called Dynamic Time Warping to match similar first and second movements among multiple movements (detected hand positions converted into time series data) acquired from the same camera view, which may be obtained after the detected hand position data has been "pre-processed" by interpolation.

[0066] This combination allows for variations in the sequence of movements that constitute the beginning or ending movements of a work cycle. For example, the first averaged predefined movement may consist of an upward movement followed immediately by a downward movement, while the actual first movement may consist of an upward movement, a rightward movement, and a subsequent downward movement. In this case, the actual first movement will match the first averaged predefined movement despite the apparent differences. Similarly, a match may also occur if a movement is omitted.

[0067] 4 illustrates a method for receiving various ground truths according to an exemplary embodiment of the present disclosure. In FIG. 4, each of time series patterns 402, 404, 406, 408, and 410 shown in 400 is obtained from a user-defined ground truth or a predefined movement, and is an example of a sequence corresponding to the start of a cycle. To measure the productivity of a target, time series patterns for multiple subjects with similar experiences are obtained.

[0068] 5 shows how ground truth 502 is obtained by averaging various received ground truths 500. Averaging sequences 402, 404, 406 and 408 results in a final query sequence, which is used as input for sequence matching to identify similar sequences in the target dataset.

[0069] 6 illustrates major components of a method for measuring productivity. In various exemplary embodiments, a method for measuring productivity is provided. The method includes identifying a first movement based on at least one image frame, the first movement corresponding to a start motion that defines a cycle of movements (S1); identifying a second movement based on at least one image frame, the second movement corresponding to an end motion that defines the cycle (S2); and determining a period between the identified first movement and the identified second movement to measure productivity (S3).

[0070] The method further includes detecting a number of hands in the image frame, comparing the number of detected hands with an expected number of hands in the frame to detect at least one undetected hand in the frame, and, if an undetected hand in the frame is detected, performing imputation of an undetected hand movement in the frame. Thus, even if a hand in the frame is not captured, the undetected hand movement can be imputed. As a result, in such a case, the first movement and / or the second movement can be identified.

[0071] Additionally, missing positions of the undetected hand are filled in by imputing using the average of the historical positions of the hand corresponding to the undetected hand during the period when the undetected hand was not detected.

[0072] The method further includes generating a first series of hand positions (first hand position sequence) corresponding to the starting motion and generating a second series of hand positions (second hand position sequence) corresponding to the ending motion, wherein identifying the first movement includes identifying a first movement that matches the first series of hand positions, and identifying the second movement includes identifying a second movement that matches the second series of hand positions.

[0073] Furthermore, generating the first series of hand positions may include averaging a plurality of series of hand positions corresponding to start movements of a motion cycle to generate the first series of hand positions, and generating the second series of hand positions may include averaging a plurality of series of hand positions corresponding to end movements of a motion cycle to generate the second series of hand positions.

[0074] In the above-described method, identifying the first movement includes identifying whether the first movement is made by the right hand or the left hand.

[0075] Additionally, if the first movement is determined to be made by the right hand, the second movement is determined, where the right hand may be the worker / operator's dominant hand.

[0076] Alternatively, if the first movement is determined to have been made with the left hand, then the second movement is determined, where the left hand may be the worker / operator's dominant hand.

[0077] 7 illustrates the main components of an apparatus for measuring productivity according to an exemplary embodiment. The apparatus 70 includes at least one processor 71 and at least one memory 72 containing computer program code. The at least one memory 72 and the computer program code are configured to cause the apparatus, using the at least one processor 71, to perform the methods described above.

[0078] 8 illustrates an exemplary computing device 1300, hereinafter also referred to as a computing system 1300, one or more of which may be used to perform the methods described above. The exemplary computing device 1300 may be used to implement the system 100 shown in FIG. 1. The following description of the computing device 1300 is illustrative and not limiting.

[0079] 8, exemplary computing device 1300 includes a processor 1307 that executes software routines. While a single processor is shown for clarity, computing device 1300 may also include a multi-processor system. Processor 1307 is connected to a communications infrastructure 1306 for communicating with other components of computing device 1300. Communications infrastructure 1306 may include, for example, a communications bus, crossbar, or network.

[0080] The computing device 1300 further includes a main memory 1308, such as random access memory (RAM), and a secondary memory 1310. The secondary memory 1310 may include a storage drive 1312, which may be, for example, a hard disk drive, a solid-state drive, or a hybrid drive, and / or a removable storage drive 1317, which may include a magnetic tape drive, an optical disk drive, a solid-state storage drive (such as a USB flash drive, a flash memory device, a solid-state drive, a memory card, etc.). The removable storage drive 1317 reads from and / or writes to a removable storage medium 1377 in a well-known manner. The removable storage medium 1377 may include a magnetic tape, an optical disk, a non-volatile memory storage medium, etc., which may be read from and written to by the removable storage drive 1317. As will be appreciated by those skilled in the art, the removable storage medium 1377 may include a computer-readable storage medium having computer-executable program code instructions and / or data stored thereon.

[0081] In alternative embodiments, secondary memory 1310 may additionally or instead include other similar means for allowing computer programs or other instructions to be loaded into computer device 1300. Such means may include, for example, removable storage unit 1322 and interface 1314. Examples of removable storage unit 1322 and interface 1314 include program cartridges and cartridge interfaces (such as those found in video game console devices), removable memory chips (such as EPROMs or PROMs) and associated sockets, removable solid-state storage devices (such as USB flash drives, flash memory devices, solid-state drives, memory cards, etc.), and other removable storage units 1322 and interfaces 1314 that allow software and data to be transferred from removable storage unit 1322 to computer system 1300.

[0082] The computing device 1300 also includes at least one communications interface 1327. The communications interface 1327 allows software and data to be transferred between the computing device 1300 and external devices via communications path 1326. In various exemplary embodiments, the communications interface 1327 allows data to be transferred between the computing device 1300 and a data communications network, such as a public or private data communications network. The communications interface 1327 is used to exchange data between different computing devices 1300, where such computing devices 1300 form part of an interconnected computer network. Examples of communications interfaces 1327 include modems, network interfaces (e.g., Ethernet cards), communications ports (e.g., serial, parallel, printer, GPIB, IEEE 1394, RJ45, USB), antennas with associated circuitry, and the like. The communications interface 1327 may be wired or wireless. The software and data transferred via the communications interface 1327 may be in the form of electronic, electromagnetic, optical, or other signals receivable by the communications interface 1327. These signals are provided to the communications interface via communications path 1326.

[0083] As shown in FIG. 8, the computing device 1300 further includes a display interface 1302 that performs processing to render images on an associated display 1350 and an audio interface 1352 that performs processing to play audio content through an associated speaker 1357.

[0084] As used herein, the term "computer program product" may refer, in part, to removable storage medium 1377, removable storage unit 1322, a hard disk installed in storage drive 1312, or a carrier wave that carries software via communications path 1326 (wireless link or cable) to communications interface 1327. A computer-readable storage medium refers to any non-transitory, non-volatile, tangible storage medium that provides recorded instructions and / or data to computing device 1300 for execution and / or processing. Examples of such storage media include magnetic tape, CD-ROM, DVD, Blu-ray® disk, hard disk drive, ROM or integrated circuit, solid-state storage drive (such as a USB flash drive, flash memory device, solid-state drive, memory card), hybrid drive, magneto-optical disk, or computer-readable card such as a PCMCIA card, whether such device is internal or external to computing device 1300. Examples of transitory or non-tangible computer-readable transmission media that may also be involved in providing software, application programs, instructions and / or data to computer device 1300 include wireless or infrared transmission channels, as well as network connections to other computers or networked devices, email transmissions, and the Internet or intranets, including information stored on websites and the like.

[0085] Computer programs (also referred to as computer program code) are stored in main memory 1308 and / or secondary memory 1310. Computer programs may also be received via communications interface 1327. When executed, such computer programs enable computing device 1300 to perform one or more functions of the exemplary embodiments described herein. In various exemplary embodiments, when executed, the computer programs enable processor 1307 to perform the functions of the exemplary embodiments described above. Thus, such computer programs function as the controller of computing system 1300.

[0086] The software is stored on a computer program product and loaded into the computing device 1300 using the removable storage drive 1317, the storage drive 1312, or the interface 1314. The computer program product may be a non-transitory computer-readable medium. Alternatively, the computer program product may be downloaded to the computing system 1300 via communications path 1326. The software, when executed by the processor 1307, causes the computing device 1300 to perform the operations necessary to carry out the methods described above.

[0087] 8 is merely an example for purposes of illustrating the operation and structure of system 100. Accordingly, in some embodiments, one or more features of computer device 1300 may be omitted. Also, in some embodiments, one or more features of computer device 1300 may be combined. Furthermore, in some embodiments, one or more features of computer device 1300 may be divided into one or more component parts.

[0088] It will be appreciated by those skilled in the art that many changes and / or modifications may be made to the invention shown in the specific embodiments without departing from the spirit or scope of the invention as broadly described. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

[0089] This application claims the benefit of priority to Singapore Patent Application No. 10202109093T, filed on August 19, 2021, the disclosure of which is incorporated herein by reference in its entirety.

[0090] Additional notes All or part of the embodiments disclosed above can be described as the following supplementary matters. (Appendix 1) 1. A computer-implemented method for measuring productivity, comprising: identifying a first movement based on at least one image frame, the first movement corresponding to a start motion defining a movement cycle; identifying a second movement based on at least one image frame, the second movement corresponding to an end motion defining a cycle of the movement; determining a period between the identified first movement and the identified second movement to measure productivity; A method comprising: (Appendix 2) Detecting the number of hands in the image frames; comparing the number of detected hands with the number of expected hands in the image frames to detect at least one undetected hand in the image frames; if detecting the undetected hand in the image frame, performing interpolation of the movement of the undetected hand in the image frame; 2. The method of claim 1, further comprising: (Appendix 3) 3. The method of claim 2, wherein the imputation imputes missing positions of the undetected hand using an average of historical positions of the hand corresponding to the undetected hand during a period in which the undetected hand is not detected. (Appendix 4) generating a first series of hand positions corresponding to the initiation motion; generating a second series of hand positions corresponding to the terminating motion; further comprising identifying the first movement includes identifying the first movement that corresponds to the first series of hand positions; 2. The method of claim 1, wherein identifying the second movement includes identifying the second movement that matches the second series of hand positions. (Appendix 5) generating the first series of hand positions includes averaging a series of hand positions of a plurality of hands corresponding to starting movements of the movement cycle to generate the first series of hand positions; 5. The method of claim 4, wherein generating the second series of hand positions includes averaging a series of hand positions of a plurality of hands corresponding to end movements of the movement cycle to generate the second series of hand positions. (Appendix 6) 1. An apparatus for measuring productivity, comprising: at least one processor; at least one memory containing computer program code; The at least one memory and the computer program code are configured to cause the device, using at least one processor, to: identifying a first movement based on at least one image frame, the first movement corresponding to a start motion defining a movement cycle; identifying a second movement based on at least one image frame, the second movement corresponding to an end motion defining a cycle of the movement; An apparatus configured to determine a period of time between the identified first movement and the identified second movement to measure productivity. (Appendix 7) The at least one memory and the computer program code, using the at least one processor, cause the device to: detecting the number of hands in the image frames; comparing the number of detected hands with the number of expected hands in the image frames to detect at least one undetected hand in the image frames; 7. The apparatus of claim 6, configured, if detecting the undetected hand in the image frame, to cause imputation of movement of the undetected hand in the image frame. (Appendix 8) 8. The apparatus of claim 7, wherein the imputation is performed by filling in missing positions of the undetected hand using an average of historical positions of the hand corresponding to the undetected hand during a period of time when the undetected hand is not detected. (Appendix 9) The at least one memory and the computer program code, using the at least one processor, cause the device to: generating a first series of hand positions corresponding to the initiation motion; generating a second series of hand positions corresponding to the terminating motion; identifying a first movement corresponding to the first series of hand positions; 7. The apparatus of claim 6, configured to identify a second movement that corresponds to the second series of hand positions. (Appendix 10) The at least one memory and the computer program code, using the at least one processor, cause the device to: averaging a series of hand positions of a plurality of hands corresponding to initiation movements of the movement cycle to generate the first series of hand positions; 10. The apparatus of claim 9, configured to average a series of hand positions of a plurality of hands corresponding to end movements of the movement cycle to generate the second series of hand positions. (Appendix 11) A non-transitory computer-readable medium storing a program for measuring productivity, the program causing a computer to perform at least the following steps: identifying a first movement based on at least one image frame, the first movement corresponding to a start motion defining a movement cycle; identifying a second movement based on at least one image frame, the second movement corresponding to an end motion defining a cycle of the movement; A computer-readable medium for determining a period of time between an identified first movement and an identified second movement to measure productivity. [Explanation of symbols]

[0091] 70 equipment 71 processors 72 memory 100 systems 102 Requester device 108 Productivity Measurement Server 109 Database 140 Remote support server 142A~142N Sensor 150A~150N Remote support host

Claims

1. a means for acquiring the image; means for receiving a setting regarding the number of hands of the person; means for acquiring the number of hands of a person appearing in the video; a means for acquiring a history of hands according to a relationship between the set number of hands and the number of hands of a person appearing in the video; An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the history acquiring means acquires the history relating to the hands when the number of hands of the person appearing in the video is insufficient for the set number of hands.

3. The information processing apparatus according to claim 1 , further comprising: means for calculating a productivity index in response to acquisition of the hand history.

4. The information processing apparatus according to claim 3 , wherein the means for calculating the productivity index includes a process for identifying whether a predetermined motion related to work is performed with the right hand or the left hand.

5. The information processing apparatus according to claim 4 , wherein the predetermined action includes an action corresponding to a start action of a task.

6. The information processing device according to claim 4 , wherein the predetermined action includes an action corresponding to an action for finishing a task.

7. The computer Get the video and Accepts settings for the number of hands on the person, Obtaining the number of hands of the person appearing in the video; acquiring a history of hands according to a relationship between the set number of hands and the number of hands of the person appearing in the video; method.

8. For computers, acquiring a video; accepting a setting regarding the number of hands of the person; obtaining the number of hands of a person appearing in the video; acquiring a history of hands according to a relationship between the set number of hands and the number of hands of a person appearing in the video; A program that executes the following.

Images