Methods, apparatus, systems, and programs for identifying tasks from video frames.

Abstract

It is determined whether each of the first motion point sequence and the second motion point sequence contains a first motion point of a first ordinal number and a second motion point of a second ordinal number, and the first motion point sequence and the second motion point sequence contain the first and second multiple motion points of one or more body parts of a person, ordered according to the respective time points at which the first and second multiple motion points were detected from the first and second series of video frames corresponding to the detection region. A method for identifying the start and end of a motion point sequence for a person to perform the task, based on the first motion point and the second motion point, from the first motion point sequence and / or the second motion point sequence.

Description

Technical Field

[0001] The present disclosure relates to a task identification method and apparatus, and more particularly, to a method, apparatus, and system for identifying tasks performed by a person from a series of video frames.

Background Art

[0002] Currently, many tasks on factory assembly lines are still performed by humans, but defects often result from human causes. Manufacturers are very interested in controlling the quality of production on the assembly line and have a strong need to ensure that all tasks on the assembly line are correctly performed. Otherwise, defective products may be shipped and recalls may occur.

[0003] Solutions have emerged to check whether the tasks performed by humans are correctly performed using machine learning-based methods. For example, there are solutions using a human pose estimator, or solutions using a finger pose estimator and an object detector. However, in order to check whether an assembly task is correctly performed using a deep learning model, the user needs to train the model with known "correct answers" such as the start and end timings of tasks in a sample video in order to annotate and identify the tasks. Such annotation work takes an extremely long time, especially in a cell production system where the number of tasks at one station may exceed 100, and the user cannot introduce such a solution.

Summary of the Invention

Problems to be Solved by the Invention

[0004] Therefore, in order to address the above problems, there is a need to provide a method, apparatus, and system for identifying tasks performed by a person from a series of video frames. Further, other desirable features and characteristics will become apparent from the following detailed description and the appended claims, in conjunction with the accompanying drawings and the background of the present disclosure. [Means for solving the problem]

[0005] In a first aspect, the Disclosure provides a method for identifying a task performed by a person from a series of video frames. The method includes a processor that determines whether each of a first motion point sequence and a second motion point sequence contains a first motion point of a first ordinal number and a second motion point of a second ordinal number within each of the first motion point sequence and the second motion point sequence, which include the first and second motion points of one or more body parts of a person, ordered according to the respective points in time when a first and second set of motion points are detected from a first and second set of video frames corresponding to a detection region, and depending on the result of the determination, the processor identifies the start and end of a motion point sequence for the person to perform a task from the first and / or second motion point sequence based on the first and second motion points.

[0006] In a second aspect, the Disclosure provides a device for identifying a task performed by a person from a series of video frames. The device comprises at least one processor and at least one memory containing computer program code, the memory and the computer program code, using at least one processor, to cause the device to determine whether each of at least a first motion point sequence and a second motion point sequence contains a first motion point of a first order number and a second motion point of a second order number, the first motion point sequence and the second motion point sequence containing a first plurality of motion points and a second plurality of motion points of one or more body parts of a person, ordered according to the respective points in time when a first plurality of motion points and a second plurality of motion points are detected from a first sequence of video frames and a second sequence of video frames corresponding to a detection region, respectively, and depending on the result of the determination, to cause the device to identify the start and end of a motion point sequence for the person to perform a task from the first motion point sequence and / or the second motion point sequence based on the first and second motion points.

[0007] In a third aspect, the Disclosure provides a system for identifying a task performed by a person from a series of video frames, comprising an apparatus according to the second aspect and at least one video capture device configured to generate a first series of video frames and a second series of video frames.

[0008] Any additional benefits and advantages of the disclosed embodiments will become apparent from the specification and drawings. Benefits and / or advantages can be obtained individually from the various embodiments and features of the specification and drawings, and it is not necessary to provide all of them in order to obtain one or more of these benefits and / or advantages. [Brief explanation of the drawing]

[0009] The embodiments of this disclosure are for illustrative purposes only and will be better understood and readily apparent to those skilled in the art through the following description in conjunction with the drawings. [Figure 1] Figure 1 illustrates a typical process for identifying assembly tasks performed by humans and checking whether the tasks were performed correctly. [Figure 2] Figure 2 illustrates the process of training a deep learning model for identifying subsequent assembly tasks using multiple sample videos of assembly tasks performed by humans at a work station. [Figure 3] Figure 3 is a flowchart illustrating a method for identifying a task performed by a person from a series of video frames, according to various embodiments of the present disclosure. [Figure 4] Figure 4 is a block diagram showing a system, according to various embodiments of this disclosure, for identifying tasks performed by a person from a series of video frames. [Figure 5] Figure 5 shows a process for identifying tasks performed by a person from a sample video, according to an embodiment of this disclosure. [Figure 6] Figure 6 shows a sequence of tasks identified from a video of four cycles of movement, each performed by a person, according to an embodiment of the present disclosure. [Figure 7] Figure 7 is a flowchart illustrating the process of identifying a series of tasks performed by a person from four cycles of video, according to an embodiment of this disclosure. [Figure 8] Figure 8 shows the detection of a turning point according to an embodiment of the present disclosure. [Figure 9] Figure 9 shows a detection region within a video frame according to one embodiment of the present disclosure. [Figure 10] Figure 10 shows a table having four sequences of turning points detected from a video of four cycles, according to one embodiment of the present disclosure. [Figure 11]Figure 11 shows the modified table with a changed sequence of turnaround points from the table in Figure 10. [Figure 12] Figure 12 shows a table having a modified table from Figure 11 that identifies the PID set, and a diagram showing the distance between two directions of motion, according to one embodiment of the present disclosure. [Figure 13] Figure 13 shows a diagram illustrating a process for estimating the start and end of a task according to one embodiment of the present disclosure. [Figure 14] Figure 14 shows a task identification process according to another embodiment of the present disclosure. [Figure 15] Figure 15 shows a different set of motion points identified from a video of a cycle within a detection region, according to yet another embodiment of the present disclosure. [Figure 16] Figure 16 shows a sequence of motion patterns detected from a video of a cycle, according to yet another embodiment of the present disclosure. [Figure 17] Figure 17 shows an interface for reviewing annotation data after an automated task identification has been performed, according to one embodiment of the present disclosure. [Figure 18] Figure 18 is a schematic diagram of an exemplary computing device suitable for use in carrying out the method in Figure 3 and implementing the apparatus in Figure 4. [Modes for carrying out the invention]

[0010] Embodiments of this disclosure are for illustrative purposes only and are described with reference to the drawings. Similar reference numerals and letters in the drawings refer to similar elements or equivalents.

[0011] Some portions of the following description are presented from the perspective of algorithms and functional or symbolic representations of operations on data within a computer memory, either explicitly or implicitly. These algorithmic descriptions and functional or symbolic representations are means used by those skilled in the data processing arts and are used to most effectively convey the essence of their work to other such skilled persons. An algorithm is considered to be a self-consistent sequence of steps that brings about a desired result. These steps require physical operations of physical quantities, such as electrical, magnetic, or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated.

[0012] Unless otherwise specified, and as will be apparent from the following, throughout this specification, discussions using terms such as "receive", "calculate", "determine", "update", "generate", "initialize", "output", "receive", "search", "identify", "distribute", "authenticate", etc. refer to the operations and processes of a computer system or similar electronic device, and manipulate and transform data represented as physical quantities within the computer system into other data represented as physical quantities within the computer system or other information storage device, transmission device, or display device.

[0013] This specification also discloses an apparatus for performing the operations of a method. Such an apparatus may be specially constructed for the required purpose or may include a computer or other device selectively activated or reconfigured by a computer program stored in the computer. The algorithms and displays presented in this specification are not inherently related to any particular computer or other device. A variety of machines may be used with a program in accordance with the teachings of this specification. Alternatively, a more specialized configuration of the apparatus for performing the required method steps may be appropriate. The structure of the computer will become apparent from the following description.

[0014] Furthermore, this specification implicitly discloses a computer program in that it will be apparent to those skilled in the art that the individual steps of the methods described herein can be executed by computer code. The computer program is not intended to be limited to any particular programming language and its implementation. It will be understood that various programming languages and their codings can be used to implement the teachings of the disclosure contained herein. Furthermore, the computer program is not intended to be limited to any particular control flow. There are many other variations of computer programs that can use different control flows without departing from the spirit or scope of the present disclosure.

[0015] Furthermore, one or more steps of the computer program may be executed in parallel rather than sequentially. Such a computer program can be stored on any computer-readable medium. The computer-readable medium can include a storage device such as a magnetic disk or optical disk, a memory chip, or other storage devices suitable for interfacing with a computer. The computer-readable medium can also include a wired medium as exemplified by an Internet system or a wireless medium as exemplified by a GSM mobile phone system. Loading and executing such a computer program on a computer effectively results in an apparatus for performing the steps of the preferred method.

[0016] Various embodiments of the present disclosure relate to a method and apparatus for identifying a task performed by a person from a series of video frames generated by at least one video capture device. It will be understood by those skilled in the art that such an apparatus and at least one video capture device can be realized as part of a system that provides the same technical effect.

[0017] Figure 1 shows a typical process for identifying assembly tasks performed by a person and checking whether the tasks were performed correctly. Typically, a video of the assembly tasks is processed by a machine learning-based method to generate tabular results. The results include a set of assembly tasks identified from the video by the machine learning-based method, such as placing the lid, picking up screws, tightening screws, and checking the lid, with the time required to complete each assembly task. These results are then compared against work procedures with standard times to complete each assembly task to determine whether the tasks performed by a person were performed correctly. In this example, the tasks of placing the lid, picking up screws, and tightening screws are identified as performed correctly based on the time, but the task of checking the lid is not detected and is therefore identified as not performed correctly.

[0018] Figure 2 shows the process of training a deep learning model for identifying subsequent assembly tasks using multiple sample videos of assembly tasks performed by a person at a work station (A123). As mentioned earlier, the deep learning model needs to be trained with "correct" results in order for it to identify and check whether the subsequent assembly tasks were performed correctly. Generally, the user needs to manually annotate each task or subtask from each sample video to generate a task list for the work station. The tabular results of the annotation work from the sample videos are shown in Table 208. If the number of tasks at a single station exceeds 100, such annotation work, annotating each task from each sample video, can take an enormous amount of time to complete, thus preventing the user from introducing a deep learning model for identifying assembly tasks.

[0019] Therefore, the objective is to provide a method, apparatus, and system for identifying tasks performed by a person from a series of video frames, thereby addressing the above challenges by automatically generating the start and end timings for each task in a series of tasks within a sample video and eliminating such annotation workload.

[0020] According to this disclosure, methods, apparatus, and systems can generate training data using movement patterns of human body parts detected from a video of an assembly task and discover the task by looking at stationary points, turning points, and short paths that are commonly observed over sample cycles. In the various embodiments described below, hand movement points or patterns detected from a video of an assembly task within a detection area such as a work station are used to identify the assembly task. This is because, in typical assembly scenes, hands are commonly visible and used, and pre-trained models can also be used. Advantageously, such methods, apparatus, and systems provide solutions that minimize manual annotation work, minimize the need to refer to work instructions, and lower the hurdles to adopting deep learning-based solutions for identifying such assembly tasks.

[0021] Figure 3 shows a flowchart 300 illustrating a method for identifying a task performed by a person from a series of video frames according to various embodiments of the present disclosure. Step 302 involves determining whether each of the first motion point sequence and the second motion point sequence contains first motion points of a first ordinal number and second motion points of a second ordinal number within each of the first motion point sequence and the second motion point sequence, wherein the first motion point sequence and the second motion point sequence contain first and second multiple motion points of one or more body parts of a person, ordered according to the respective points in time when the first and second multiple motion points were detected from the first and second series of video frames corresponding to the detection region. Step 304 involves identifying the start and end of a motion point sequence for a person to perform a task from the first and / or second motion point sequence based on the first and second motion points.

[0022] Figure 4 is a block diagram showing a system 400 for identifying a task to be performed by a person from a series of video frames, according to various embodiments of the present disclosure.

[0023] Image or video input management is performed by at least one video capture device 402 and apparatus 404. For simplicity, only one video capture device 402 is shown. System 400 includes the video capture device 402 communicating with apparatus 404. In embodiments, apparatus 404 may generally be described as a physical device including at least one processor 406 and at least one memory 408 containing computer program code. The at least one memory 408 and computer program code are configured to cause the physical device to perform the operations described in Figure 3 using at least one processor 406. Processor 406 is configured to receive one or more input videos from the video capture device 402 or to retrieve one or more videos from a database. Alternatively or additionally, one or more videos captured by the video capture device 402 are stored in a database 410, and processor 406 is configured to retrieve one or more videos from the database 410.

[0024] The video capture device 402 may be a device such as a closed-circuit television (CCTV) that provides various data, such as data relating to the appearance and / or movement of one or more body parts of a person, in order to identify a task performed by that person. In embodiments, appearance data obtained from the video capture device 402 may be stored in the memory 408 of the device 404 or in a database 410 accessible by the device 404. The data may include (i) facial feature data such as the relative position, size, shape and / or contour of the eyes, nose, cheekbones, jaw and chin, as well as iris pattern, skin color, hair color or combination thereof; (ii) height, body size, body proportions, limb length, hair color, skin color, clothing, possessions, other similar features or combinations; and (iii) behavioral feature data such as body movement, limb position, direction of movement, difference in direction of movement, speed of movement, frequency, pattern of movement, how or for how long a person or their body parts are stationary or moving, other similar features or combinations.

[0025] In the embodiment, camera data such as position and resolution, and / or time data including a timestamp identifying one or more people, may also be obtained from the video capture device 402. The camera data and / or time data may be stored in the memory 408 of the device 404 or in a database 410 accessible by the device 404, and the processor 406 is configured to identify and retrieve data or video based on the time data. It should be understood that the database 410 may be part of the device 404.

[0026] The device 404 may be configured to communicate with a video capture device 402 and a database 410. For example, the device 404 may receive as input multiple videos, each having a series of video frames related to a detection area on an assembly task workstation (corresponding to the field of view of the video capture device 402), from either the video capture device 402 or the database 410, and after processing by a processor 406 within the device 404, generate an output relating to the identification of a task or series of tasks performed by a person from one or more videos. Such an output may then be used to train a deep learning model to identify a task or series of tasks performed by a person from the videos.

[0027] According to this disclosure, after receiving a first series of video frames or a second series of video frames from a video capture device 402, which can be obtained from a single video file or separate video files, or after retrieving the first and second series of video frames from a database 410, the memory 408 and the computer program code stored therein are configured to cause the device 404 to determine, using the processor 406, whether each of the first and second motion point sequences contains a first motion point of a first ordinal number and a second motion point of a second ordinal number within each of the first and second motion point sequences.

[0028] The first motion point sequence and the second motion point sequence each include a first and second set of motion points on one or more body parts of a person, ordered according to the time points at which the first and second sets of motion points were detected from a first and second set of video frames corresponding to the detection region.

[0029] The memory 408 and the computer program code stored therein may be configured, using the processor 406, to enable the device 404 to determine, based on the result of a determination, the start and end of a motion point sequence for a person to perform a task, from the first motion point sequence and / or the second motion point sequence.

[0030] In one embodiment, if the first motion point sequence includes a first motion point at a third ordinal number which is an ordinal number count that is separate from the first ordinal number in the first motion point sequence, the memory 408 and the computer program code stored therein are configured to cause the processor 406 to increment the third ordinal number of the first motion point sequence and the ordinal numbers of subsequent motion points in the first motion point sequence by an ordinal number count, shifting the first motion point to the first ordinal number of the first motion point sequence (the subsequent motion point becomes the ordinal number of the first motion point sequence following the first ordinal number), and to determine whether the first motion point sequence and the second motion point sequence each include a first motion point at the first ordinal number and a second motion point at the second ordinal number in the first and second motion point sequences, respectively, after the increment / shift.

[0031] In one embodiment, if the first motion point sequence includes a third motion point in a first ordinal number, the memory 408 and the computer program code stored therein are configured to use the processor 406 to cause the device 404 to determine whether the third motion point is within a threshold distance from the first motion point, and to cause the first motion point sequence and the second motion point sequence to determine whether each of them includes a first motion point or a third motion point in a first ordinal number, and a second motion point in a second ordinal number.

[0032] In another embodiment, the memory 408 and the computer program code stored therein are configured to use the processor 406 to cause the device 404 to detect one or more body parts of a person in a portion of the detection area within each video frame of the first and second series of video frames, and to assign motion points corresponding to the portions of the detection area within each video frame of the first and second series of video frames. Furthermore, in such an embodiment, the memory 408 and the computer program code stored therein are configured to use the processor 406 to cause the device 404 to determine whether a first portion of the detection area within the first video frame of either the first or second series of video frames and a second portion of the detection area within the second video frame of either the first or second series of video frames are both located within one of a plurality of smaller detection areas within the detection area, and to assign a single motion point corresponding to the first and second portions of the detection area.

[0033] In yet another embodiment, the memory 408 and the computer program code stored therein are configured to use the processor 406 to cause the device 404 to detect a change in the direction of movement of one or more body parts of a person in a portion of the detection area within each video frame of the first and second series of videos (hereinafter sometimes referred to as a change in direction). In this other embodiment, the memory 408 and the computer program code stored therein are configured to use the processor 406 to cause the device 404 to determine whether the angle between the preceding direction of movement of one or more body parts of a person detected at a time prior to the time when one or more body parts of a person are detected in a portion of the detection area and the subsequent direction of movement of one or more body parts of a person detected at a time later than the time when one or more body parts of a person are detected in a portion of the detection area is greater than a threshold angle, and to detect a change in the direction of movement of one or more body parts of a person based on the result of this angle determination.

[0034] In this embodiment, the memory 408 and the computer program code stored therein are configured to use the processor 406 to cause the device 404 to determine whether one or more parts of a person's body are stationary or moving within a given time in a portion of the detection area within each video frame of the first and second series of video frames, and to assign a motion point corresponding to the portion of the detection area based on the determination result of one or more parts of a person's body.

[0035] In another embodiment, the memory 408 and the computer program code stored therein use the processor 406 to cause the device 404 to determine whether each of the first and second motion point sequences contains a first motion pattern and a second motion pattern, and, based on the result of the determination, to identify the start and end of the motion point sequence for a person to perform a task, based on the first and second motion patterns.

[0036] In yet another embodiment, the memory 408 and the computer program code stored therein are configured to use the processor 406 to cause the device 404 to extract first data and second data relating to one or more body parts of a person performing a motion point sequence from the first sequence of motion points and the second sequence of motion points, respectively, and to display at least some of the first data that is not present in the second data and some of the second data that is not present in the first data in different colors across the detection area.

[0037] Figure 5 shows Figure 500 illustrating a process for identifying tasks performed by a person from a sample video, according to one embodiment of the present disclosure. This process can be divided into a setup phase 502 in which the sample video is input and a deep learning model is trained to identify tasks or sets of tasks performed by a person, and an operation phase 504 in which the trained model is then used to identify tasks or sets of tasks, or similar tasks or sets of similar tasks, performed by that person or another person.

[0038] In setup phase 502, a sample video 506 containing 5 to 10 cycles of movement of one or more people is processed by performing hand detection. A series of tasks (e.g., Task A, Task B, Task C) are identified for each cycle of movement based on the hand detection results. Annotation data for each task on the sample video is also automatically generated. This annotation data is used to train a deep learning model. Then, in operation phase 504, the video (or a series of video frames) is acquired from camera 508 and processed by the trained deep learning model to check and identify the same or a series of tasks, or similar or a series of similar tasks, performed by the person or another person. If the deep learning model identifies that a series of tasks are not performed correctly, it can generate a warning to alert the user.

[0039] Figure 6 shows Figure 600, which illustrates a set of tasks identified from a video of four cycles of movement (Cycle 1, Cycle 2, Cycle 3, Cycle 4) each performed by a person, according to an embodiment of the present disclosure. Figure 7 shows a flowchart 700, which illustrates a process for identifying a set of tasks performed by a person from a video of four cycles, according to an embodiment of the present disclosure. Note that two or more cycles of movement may be obtained from a single sample video (or a series of video frames).

[0040] Four cycles of video, each having a series of video frames covering the same detection region 712, are used, and the following steps are performed for each cycle of video (series of video frames) to identify a series of tasks performed by a person. Step 702 is performed using an object detector to detect the position of a hand in a portion (x and y coordinates) of the detection region 712 within the video frame. Step 704 is performed to generate a hand trajectory that tracks the movement of a person's hand, once various hand positions have been detected over time. An example of Table 714 is shown, which shows the hand trajectory (e.g., x and y coordinates) of a person's hand (e.g., right hand) detected from the Cycle 1 video. Step 706 is performed to detect motion points. In this embodiment, as shown in the example of Table 716, a hand turning motion is identified as a motion point. From each cycle of video, various motion (turning) points of a person's hand within the detection region within the video frame are detected. The motion points of a person's hand detected from each cycle of video are ordered according to the time of the video frame in which the motion point was detected to form a motion point sequence.

[0041] In the following embodiments, a motion point identifier (e.g., a position / point identifier (PID)) may be assigned to each specific motion point detected within the detection area, specifying the specific x and y coordinates within the detection area in the video frame. The same motion point identifier may be assigned to the same motion point within the same detection area in different sets of video frames in different motion cycles. For simplicity of explanation, motion points will be represented using motion point identifiers below, without specifying their correspondence to the x and y coordinates within the detection area.

[0042] Table 1 shows four sequences of motion points (in this case, turning points) detected from four different motion cycles (Cycle 1, Cycle 2, Cycle 3, and Cycle 4). Each sequence is formed by rearranging the motion points detected from the motion cycle (a series of video frames) according to the time of the detected video frames, or the order in which a person performed actions to form the sequence. Specifically, the first motion point detected from the motion cycle becomes the first term of the sequence (ordinal number "1" or "1st"), and the motion point detected immediately after the first detected motion point becomes the second term of the sequence (ordinal number "2" or "2nd"). In this embodiment, all four sequences have 10 terms, from the 1st to the 10th.

[0043] [Table 1]

[0044] According to this disclosure, the motion points of a certain ordinal number in all sequences (cycles) are compared to determine whether they all share the same motion point at the same ordinal number in the sequence. For example, all four sequences are determined and identified to have motion point IDs of "6" in the 3rd term, "8" in the 5th term, "11" in the 7th term, "11" in the 8th term, and "13" in the 10th term. Furthermore, if two or more motion points are close to each other, that is, within a threshold distance, it is determined that they form a motion point set and may be treated as a single motion point in subsequent processing. For example, if the motion point sets "12" and "14" of the 1st term are determined to be within a threshold distance from each other, the same applies to the motion point sets "3" and "4" of the 9th term. This identifies that all four sequences have the same motion point in the 1st and 9th terms, respectively.

[0045] Next, in step 708, the step of determining the task boundaries is performed. Specifically, each term that has the same movement point ID in all four sequences is identified as a task boundary and assigned a boundary ID. In this case, terms with ordinal numbers 1, 3, 5, 7, 8, 9, and 10 are assigned boundary IDs "B1", "B2", "B3", "B4", "B4", "B5", and "B6", respectively. In step 710, the step of estimating the start and end of the task is performed. Specifically, the start and end of the task are defined using the boundaries and a time or duration. For example, movement points from boundary IDs "B1" and "B2" (items 1-3) are identified in Task 1, movement points from boundary IDs "B2" and "B3" (items 3-5) are identified in Task 2, movement points from boundary IDs "B3" and "B4" (items 5-7) are identified in Task 3, movement points from the two boundary IDs "B4" and "B4" (items 7-8) are identified in Task 4, movement points from boundary IDs "B4" and "B5" (items 8-9) are identified in Task 5, and movement points from boundary IDs "B5" and "B6" (items 9-10) are identified in Task 6.

[0046] Figure 8 is a figure 800 illustrating the detection of a turning point according to an embodiment of the present disclosure. When processing a video covering a detection region, the position of the hand within the detection region (e.g., x and y coordinates) is acquired at each point in the video (e.g., T1 to T6). The direction of the hand movement (angle relative to the x-axis) at each point in time can be calculated for each point in time by acquiring the hand position at that point and the previous point in time. For example, the direction of the hand movement from the hand position at T1 to the hand position at T2 is 180°. From the hand position at T2 to the hand position at T3 it is 176.53°. From the hand position at T3 to the hand position at T4 it is 177.4°. From the hand position at T4 to the hand position at T5 it is 19.54°. From the hand position at T5 to the hand position at T6 it is 19.36°.

[0047] According to the present invention, a turning point is detected when there is a switch in the direction of motion, that is, when there is a change in the direction of motion. For example, a change in direction is detected when the difference in angle with respect to the x-axis between the direction of motion at Ti and the previous direction of motion at Ti-1 is greater than a threshold angle (e.g., greater than the direction of motion at Ti or 90°). For example, the difference in the direction of motion between T3 and T4 is 157.86°, which is greater than the direction of motion at T4, so a change in direction is detected at T4.

[0048] Figure 9 shows Figure 900, which illustrates a detection region within a video frame according to one embodiment of the present disclosure. The detection region within the video frame is subdivided into a plurality of smaller detection regions that form a grid, each smaller detection region occupying different x and y coordinates within the detection region. If multiple motion points, in this case turnaround points, are determined within the same smaller detection region 902, a single turnaround point 904 is created and assigned to represent all of them.

[0049] Figure 10 shows Table 1000, which includes four sequences of reversal points detected from four cycles of video, according to one embodiment of the present disclosure. Motion points (reversal points) that occur and are detected within a detection area in the video are indicated using PIDs. Figure 11 shows modified Tables 1102 and 1104, which include modified sequences of reversal points from the table in Figure 10. In this embodiment, it is identified that PID "6" is not aligned across all four sequences. In particular, PID "6" is in the third term or cell (ordinal number 3) of the sequence obtained from the cycle 2 video, while it is in the second term or cell (ordinal number 2) of the sequences obtained from the cycles 1, 3, and 4 videos. There is a one-term count difference (or ordinal number difference) between PID "6" in the cycle 2 video sequence and PID "6" in the cycles 1, 3, and 4 video sequences. In the sequence of cycles 1, 3, and 4 videos, PID "6" is shifted and moved back by the same number of ordinal counts so that the same PID "6" is aligned (present in the same column of the table) across all sequences. Therefore, the sequence length (i.e., the total number of moving points) of the sequence of cycles 1, 3, and 4 videos on which the shift is performed increases by the same number of ordinal counts.

[0050] A similar shift operation is performed for each sequence, aligning PIDs "8", "11", "11", and "13" to the 3rd, 5th, 7th, 8th, and 10th terms, respectively, across all sequences. For each term (or ordinal number) with the same PID, the PID is added to the bounding list.

[0051] Figure 12 shows Table 1200, which identifies a set of PIDs from the modified table of Figure 11, and Figure 1202, which shows the distance between two PIDs, according to an embodiment of the present disclosure. If different PIDs exist in the same term of different sequences, for example, if the first term of cycle 1 and 4 videos is PID "12" and cycle 2 and 3 videos are PID "14", the distance between the two PIDs, for example, the difference in xy coordinates within the detection region, is calculated, and it is determined whether the calculated distances are close to each other and within the threshold distance. The xy coordinates within the detection region corresponding to the PIDs are used to calculate the distance between the two PIDs, for example, using equation (1) below.

number

[0052] For example, we determine that the first term of the cycle 1 and 4 videos with PID "12" is different from the first term of the cycle 2 and 3 videos with PID "14". The distance between PID "12" and PID "14" is calculated using equation (1). In this case, the x and y coordinates of PID "12" are (575, 624) and the x and y coordinates of PID "14" are (512, 666). The distance between the two PIDs is 75.7, which is within the threshold distance. Therefore, the set of PIDs {12, 14} is also added to the boundary list. The same applies to the ninth term of the sequence of PIDs "2", "3", and "4". Since these are all within the threshold distance of each other, they form a set of PIDs and are added to the boundary list.

[0053] Figure 13 shows Figure 1300 illustrating a process for estimating the start and end of a task according to one embodiment of the present disclosure. After a boundary list is formed, each PID or set of PIDs in the boundary list is assigned a boundary ID, as shown in Table 1302. Next, it is checked whether the PIDs in each sequence of PIDs exist in the boundary list. Then, the start and end of the task are identified and annotated using the video time points in which the PIDs were detected.

[0054] For example, the PIDs in the sequence obtained from the video of Cycle 1 are checked against the boundary list. It is confirmed that the PIDs "12", "6", "8", "11", "11", "4", and "13" exist in the sequence and match the PIDs in the boundary list of boundary IDs "B5", "B1", "B2", "B3", "B3", "B6", and "B4", respectively. The start and end of the task are then determined using the time when matching PIDs in the boundary list are detected. Specifically, the start and end of the first task, Task 1, are set using the first time point T400 when matching PID "12" is detected and the time point T629 immediately before the time point T630 when the next matching PID "6" is detected. Similarly, the start and end of the next task, Task 2, are set using the second time point T630 when the second matching PID "6" is detected and the time point T791 immediately before the time point T840 when the next matching PID "8" is detected. The start and end times for the next task, Task 3, are set using the third time point T840, when the third matching PID "8" is detected, and the time point T976, immediately preceding the time point T977, when the next matching PID "11" is detected. Similarly, the start and end times for each task in Cycle 1 are automatically estimated to form a series of tasks with six tasks, as shown in Table 1308.

[0055] In alternative embodiments, moving points may collectively form areas or regions within the detection area based on their proximity to other moving points and other characteristics such as the type of movement (e.g., reversal movement, stillness, slight movement) and the duration of stillness (e.g., immediate reversal without a period of stillness, reversal after a period of stillness). Thus, unlike a grid, different regions may be separate and discontinuous from one another. In such embodiments, detection of hand movement points is based on transitions and movements from one region to another.

[0056] Figure 14 shows Figure 1400 illustrating a task identification process according to another embodiment of the present disclosure. The detection area can be divided into five distinct areas, areas A to E. Areas A, B, C, and E each include turnaround points that are close to each other, except area D, which includes only a stationary point where it is detected that the hand remains at that point for a predetermined amount or period of time. Hand movements and sequences of movements between areas are recorded as shown in sequence 1402. For example, a task is identified when there is movement between stationary point area D and a turnaround point area (e.g., area C), and another task is identified when there is a switch in movement between stationary point area D and another turnaround point area (e.g., areas B, A, or E). Specifically, the period 1412 in which movement occurred between areas D and C is identified as the start and end of task 1, the subsequent period 1414 in which movement occurred between area D and B is identified as the start and end of task 2, and the subsequent period 1416 in which movement occurred between area D and A is identified as the start and end of task 3. The subsequent period 1418 during which movement occurred between regions A and E is identified as the start and end of Task 4.

[0057] In yet another embodiment, motion points detected from a cycle video are further classified under the same set of motion points based on a certain type of movement and period of stillness, and the detection of hand motion points is based on transitions and movements from one set of motion points to another. Figure 15 shows Figure 1500, which illustrates different sets of motion points identified from a cycle video within a detection area, according to yet another embodiment of the present disclosure. From the cycle video, four different sets (types) of motion points are identified: a still point where the hand remains motionless, such as the palm touching the workbench; a still point 2 where the hand remains motionless, such as tightening a screw; a turning point 1 where the hand returns after a short period of stillness, such as picking up a small screw; and a turning point 2 where the hand returns immediately with almost no period of stillness, such as grasping a large part or pressing a button.

[0058] In yet another embodiment, motion patterns may be detected based on a plurality of detected motion points, and tasks may be identified based on a series of motion patterns. Figure 16 shows Figure 1600, which illustrates a series of motion patterns detected from a video of a cycle according to yet another embodiment of the present disclosure. A series of motion patterns a, B, C, C, C, C, D are detected. As an example, a task may be identified and created based on one motion pattern, and another task may be identified and folded when a different motion pattern is detected. In this case, the period from when motion pattern A is detected to when motion pattern B is detected is the start and end of task 1. The period from when motion pattern B is detected to when motion pattern C is detected is the start and end of task 2. The period from when the motion pattern is first detected to when a different motion pattern D is detected is the start and end of task 3. The remaining time from when motion pattern D is detected until the end of the video of the cycle is the start and end of task 4.

[0059] Figure 17 is a 1700 interface for reviewing annotation data after an automated task identification according to one embodiment of the present disclosure has been performed. A series of 18 tasks (Tasks 1 to 18) are identified from multiple sample videos having multiple motion cycles. The user can select a task of interest, for example, Task 5, and the video clips specific to the selected task are displayed on the playback screen. In one example, video clips of all sample videos annotated under Task 5 are displayed on the playback screen, and data on the worker's silhouette from different video clips is extracted and overlaid. If it is detected that the silhouette from one sample video is significantly different from the silhouette from other sample videos in which the worker is performing the same task, for example, if the data on the worker's silhouette in one video clip is not present in the data on the worker's silhouette in other video clips, and the amount of data in the video clip that is not present in other video clips is greater than a threshold amount, it will be displayed in a different color (e.g., red). The user can remove certain video clips that appear to be misestimated from the training sample videos in order to improve the annotation and task identification results.

[0060] Figure 18 shows a schematic diagram of an exemplary computing device 1800 (hereinafter also referred to as computer system 1800). One or more such computing devices 1800 may be used, or may be suitable for use, to perform the method of Figure 3 and implement the apparatus of Figure 4. The following description of computing device 1800 is provided as an example only and is not intended to limit it.

[0061] As shown in Figure 18, the exemplary computing device 1800 includes a processor 1804 for executing software routines. Although a single processor is shown for clarity, the computing device 1800 may include a multiprocessor system. The processor 1804 is connected to a communication infrastructure 1806 for communicating with other components of the computing device 1800. The communication infrastructure 1806 may include, for example, a communication bus, a crossbar, or a network.

[0062] The computing device 1800 further includes a main memory 1808, such as random access memory (RAM), and a secondary memory 1810. The secondary memory 1810 may include a storage drive 1812, which may be, for example, a hard disk drive, a solid-state drive, or a hybrid drive, and / or a removable storage drive 1814, which may include a magnetic tape drive, an optical disc drive, a solid-state storage drive (such as a USB flash drive, a flash memory device, a solid-state drive, or a memory card). The removable storage drive 1814 reads from and / or writes to the removable storage medium 1818 in a well-known manner. The removable storage medium 1818 may include a magnetic tape, an optical disc, a non-volatile memory storage medium, etc., which is read from and written to by the removable storage drive 1814. As will be understood by those skilled in the art, the removable storage medium 1818 includes a computer-readable storage medium storing computer executable program code instructions and / or data.

[0063] In alternative embodiments, the secondary memory 1810 may additionally or alternatively include other similar means that enable loading computer programs or other instructions into the computing device 1800. Such means may include, for example, a removable storage unit 1822 and interface 1820. Examples of removable storage units 1822 and interface 1820 include program cartridges and cartridge interfaces (e.g., those found in video game consoles), removable memory chips (e.g., EPROM or PROM) and associated sockets, removable solid-state storage drives (e.g., USB flash drives, flash memory devices, solid-state drives, memory cards), and other removable storage units 1822 and interface 1820 capable of transferring software and data from the removable storage unit 1822 to the computer system 1800.

[0064] The computing device 1800 also includes at least one communication interface 1824. The communication interface 1824 enables software and data to be transferred between the computing device 1800 and an external device via a communication path 1826. In various embodiments of this disclosure, the communication interface 1824 enables data transfer between the computing device 1800 and a data communication network, such as a public data or private data communication network. The communication interface 1824 can be used to exchange data between different computing devices 1800, such computing devices 1800 forming part of an interconnected computer network. Examples of the communication interface 1824 include a modem, a network interface (such as an Ethernet card), a communication port (such as serial, parallel, printer, GPIB, IEEE 1394, RJ45, USB), and an antenna with associated circuitry. The communication interface 1824 may be wired or wireless. The software and data transferred via the communication interface 1824 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals receivable by the communication interface 1824. These signals are provided to the communication interface via the communication path 1826.

[0065] As shown in Figure 18, the computing device 1800 further includes a display interface 1802 that performs operations to render images to an associated display 1830, and an audio interface 1832 that performs operations to play audio content through one or more associated speakers 1834.

[0066] As used herein, the term “computer program product” may partially refer to a removable storage medium 1818, a removable storage unit 1822, a hard disk installed in a storage drive 1812, or a carrier wave that carries software to a communication interface 1824 via a communication path 1826 (wireless link or cable). Computer-readable storage medium refers to any non-temporary, non-volatile tangible storage medium that is provided to the computing device 1800 for executing and / or processing recorded instructions and / or data. Examples of such storage media include magnetic tape, CD-ROM, DVD, Blu-ray disc, hard disk drive, ROM or integrated circuit, solid-state storage drive (such as USB flash drive, flash memory device, solid-state drive, or memory card), hybrid drive, magneto-optical disk, or computer-readable card such as a PCMCIA card, whether these devices are inside or outside the computing device 1800. Examples of temporary or intangible computer-readable transmission media that may be used to provide software, application programs, instructions and / or data to the computing device 1800 include wireless or infrared transmission channels, as well as network connections to other computers or networked devices, and the Internet or intranet, including information recorded on websites and the like, including email transmissions.

[0067] The computer program (also called computer program code) is stored in the main memory 1808 and / or the secondary memory 1810. The computer program can also be received via the communication interface 1824. When such a computer program is executed, it enables the computing device 1800 to perform one or more functions of the embodiments described herein. In various embodiments, when the computer program is executed, it enables the processor 1804 to perform the functions of the embodiments described above. Thus, such a computer program represents the controller of the computer device 1800.

[0068] The software may be stored in a computer program product and loaded onto the computing device 1800 using a removable storage drive 1814, a storage drive 1812, or interface 1820. The computer program product may be a non-temporary computer-readable medium. Alternatively, the computer program product may be downloaded to the computer system 1800 via a communication path 1826. Once executed by the processor 1804, the software causes the computing device 1800 to perform the operations necessary to implement the apparatus shown in Figure 4, in the manner shown in Figure 3.

[0069] It should be understood that the embodiment shown in Figure 18 is presented merely as an example to illustrate the operation and structure of the device 400. Therefore, in some embodiments, one or more features of the computing device 1800 may be omitted. Also, in some embodiments, one or more features of the computing device 1800 may be combined together. Furthermore, in some embodiments, one or more features of the computing device 1800 may be divided into one or more components.

[0070] Those skilled in the art will understand that numerous variations and / or modifications can be made to this disclosure, as shown in the particular embodiments, without departing from the spirit or scope of the broadly described disclosure. Therefore, these embodiments are considered illustrative and non-restrictive in all respects.

[0071] This application claims priority over Singapore Patent Application No. 10202301201Q, filed on 28 April 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0072] For example, the exemplary embodiments disclosed above, in whole or in part, may be described as follows, but are not limited to these. (Note 1) The processor determines whether each of the first motion point sequence and the second motion point sequence contains a first motion point of a first ordinal number and a second motion point of a second ordinal number, and the first motion point sequence and the second motion point sequence contain the first and second motion points of one or more body parts of a person, ordered according to the respective points in time when the first and second motion points were detected from a first and second series of video frames corresponding to the detection region. Depending on the result of the determination, the processor identifies the start and end of the motion point sequence for the person to perform the task from the first motion point sequence and / or the second motion point sequence, based on the first motion point and the second motion point. A method for identifying tasks performed by a person from a series of video frames. (Note 2) The first motion point sequence includes the first motion point in the first motion point sequence at a third ordinal number that is separated from the first ordinal number by an ordinal number count, and the method is The third order number of the first movement point and subsequent movement points in the first movement point sequence are incremented by the order number count so that the first movement point becomes the first order number of the first movement point sequence, and after the increment, the first movement point sequence and the second movement point sequence are determined respectively. The method described in Appendix 1. (Note 3) The increase in the third ordinal number of the first movement point and the ordinal number of subsequent movement points includes increasing the number of the plurality of movement points in the first movement point sequence by the ordinal number count. The method described in Appendix 2. (Note 4) The first motion point sequence includes the third motion point in the first ordinal number, and the determination of the first motion point sequence and the second motion point sequence is as follows: Determining whether the third movement point is within a threshold distance from the first movement point, The method includes determining whether each of the first motion point sequence and the second motion point sequence contains, within each of the first motion point sequence and the second motion point sequence, the first motion point in the first ordinal number or the third motion point and the second motion point in the second ordinal number. The method described in any one of the appendices 1 to 3. (Note 5) Detecting one or more body parts of the person in a portion of the detection area in each video frame of each of the first and second series of video frames, A method further comprising assigning motion points corresponding to a portion of the detection area in each video frame of each of the first and second series of video frames. The method described in any one of the appendices 1 to 4. (Note 6) It is determined whether a first portion of the detection region in one of the first and second video frames and a second portion of the detection region in one of the first and second video frames are within one of a plurality of smaller detection regions of the detection region, and each of the plurality of smaller detection regions occupies different x and y coordinates within the detection region. The further includes assigning a single motion point corresponding to the first and second portions of the detection region, The method described in Appendix 5. (Note 7) In each of the first and second series of video frames, detecting one or more body parts of the person in a part of the detection area is: This includes detecting a change in the direction of movement of one or more body parts of the person in a portion of the detection area in each of the first and second series of video frames, The method described in Appendix 5 or 6. (Note 8) In a portion of the detection area, it is determined whether the angle between the previous direction of movement of one or more body parts of the person detected at a time prior to the detection of one or more body parts of the person, and the subsequent direction of movement of one or more body parts detected at a time later than the detection of one or more body parts of the person, is greater than a threshold angle. The detection of a change in the direction of movement of one or more body parts of the person is based on the angle determination result. The method described in Appendix 7. (Note 9) It is determined whether one or more body parts of the person are stationary or moving within a portion of the detection area in each video frame of each of the first and second series of video frames, and the assignment of the motion points corresponding to the portion of the detection area is based on the determination result of one or more body parts of the person. The method described in any one of the appendices 5 to 8. (Note 10) One of the first motion point and the second motion point is one of two motion points identified as the start and end of another motion point sequence sequence for the person to perform another task, from a third motion point sequence including a third plurality of motion points of the one or more body parts of the person detected from a first series of video frames, and a fourth motion point sequence including a fourth plurality of motion points of the one or more body parts of the person detected from a second series of video frames, wherein the task and the other task are two of a series of tasks ordered according to the respective time points at which the motion points of the motion point and the other motion point sequence were detected. The method described in any one of the appendices 1 to 9. (Note 11) The determination of the first motion point sequence and the second motion point sequence is as follows: It is determined whether each of the first motion point sequence and the second motion point sequence contains a first motion pattern and a second motion pattern within each of the first motion point sequence and the second motion point sequence. The determination of the start and end of the motion point sequence for the person to perform the task, in accordance with the result of the determination, is based on the first motion pattern and the second motion pattern. The method described in any one of the appendices 1 to 10. (Note 12) The first set of video frames and the second set of video frames are from two different videos. The method described in any one of the appendices 1 to 11. (Note 13) From the first motion point sequence and the second motion point sequence, respectively, first data and second data relating to one or more body parts of the person executing the motion point sequence are extracted. To determine whether the amount of the first data that is not present in the second data is greater than a threshold amount, On the detection area, at least a portion of the first data that is not present in the second data is displayed in a different color from the other portions of the first data and the second data. including, The method described in any one of the appendices 1 to 12. (Note 14) At least one processor, It comprises at least one memory containing computer program code, At least one memory and computer program code are used to send at least one processor to the device, The first motion point sequence and the second motion point sequence are to be determined as to whether each of the first motion point sequence and the second motion point sequence contains a first motion point of a first ordinal number and a second motion point of a second ordinal number, and the first motion point sequence and the second motion point sequence contain a first plurality of motion points and a second plurality of motion points of one or more body parts of a person, which are ordered according to the respective time points at which a first plurality of motion points and a second plurality of motion points are detected from a first sequence of video frames and a second sequence of video frames corresponding to the detection region. Depending on the result of the determination, based on the first and second movement points, the start and end of the movement point sequence for the person to perform the task are identified from the first movement point sequence and / or the second movement point sequence. It is configured in such a way. A device for identifying tasks performed by a person from a series of video frames. (Note 15) The first motion point sequence includes the first motion point in a third ordinal number that is separated from the first ordinal number by an ordinal number count within the first motion point sequence. The at least one memory and the computer program code are used by the at least one processor to provide the device with at least, The third order number of the first movement point, and subsequent movement points in the first movement point sequence, are incremented by the order number count so that the first movement point becomes the first order number of the first movement point sequence. The system determines whether each of the first motion point sequence and the second motion point sequence, after the increase, includes the first motion in the first ordinal number and the second motion point in the second ordinal number. It is configured in such a way. The apparatus described in Appendix 14. (Note 16) The at least one memory and the computer program code are used by the at least one processor to provide the device with at least, As the third ordinal number of the first movement point and the ordinal number of subsequent movement points increase, the number of the multiple movement points in the first movement point sequence is increased by the ordinal number count. The apparatus described in Appendix 15, configured as follows. (Note 17) The first motion point sequence includes a third motion point in the first ordinal number, The at least one memory and the computer program code are used by the at least one processor to provide the device with at least, Determining whether the third movement point is within a threshold distance from the first movement point, Determining whether each of the first motion point sequence and the second motion point sequence includes, within each of the first motion point sequence and the second motion point sequence, the first motion point in the first ordinal number or the third motion point and the second motion point in the second ordinal number, It is configured to perform the following: The apparatus described in any one of the appendices 14 to 16. (Note 18) The at least one memory and the computer program code are used by the at least one processor to provide the device with at least, Detecting one or more body parts of the person in a portion of the detection area in each video frame of each of the first and second series of video frames, Assigning motion points corresponding to a portion of the detection area in each video frame of each of the first and second series of video frames, It is configured to perform the following: The apparatus described in any one of the appendices 14 to 17. (Note 19) The at least one memory and the computer program code are used by the at least one processor to provide the device with at least, It is determined whether a first portion of the detection region in one of the first and second video frames and a second portion of the detection region in one of the first and second video frames are within one of a plurality of smaller detection regions of the detection region, and each of the plurality of smaller detection regions occupies different x and y coordinates within the detection region. Assigning a single motion point corresponding to the first and second portions of the detection region, It is configured to perform the following: The apparatus described in Appendix 18. (Note 20) The at least one memory and computer program code are used by the device to operate using at least one processor. The system detects a change in the direction of movement of one or more body parts of the person in a portion of the detection area in each video frame of each of the first and second series of video frames, and detects one or more body parts of the person in a portion of the detection area in each video frame of each of the first and second series of video frames. It is configured in such a way. The apparatus described in Appendix 18 or 19. (Note 21) The at least one memory and computer program code are used by the device to operate using at least one processor. In a portion of the detection area, it is determined whether the angle between the previous direction of movement of one or more body parts of the person detected at a time prior to the detection of one or more body parts of the person, and the subsequent direction of movement of one or more body parts detected at a time later than the detection of one or more body parts of the person, is greater than a threshold angle, and based on the angle determination result, a change in the direction of movement of one or more body parts of the person is detected. It is configured in such a way. The apparatus described in Appendix 20. (Note 22) The at least one memory and computer program code are used by the device to operate using at least one processor. It is determined whether one or more body parts of the person are stationary or moving within a portion of the detection area in each video frame of each of the first and second series of video frames. Based on the determination result of one or more body parts of the person, the motion point corresponding to a part of the detection area is assigned. It is configured in such a way. The apparatus described in any one of the items 18 to 21 of the appendix. (Note 23) One of the first motion point and the second motion point is one of two motion points identified as the start and end of another motion point sequence sequence for the person to perform another task, from a third motion point sequence including a third plurality of motion points of the one or more body parts of the person detected from a first series of video frames, and a fourth motion point sequence including a fourth plurality of motion points of the one or more body parts of the person detected from a second series of video frames, wherein the task and the other task are two of a series of tasks ordered according to the respective time points at which the motion points of the motion point and the other motion point sequence were detected. The apparatus described in any one of the appendices 14 to 22. (Note 24) The at least one memory and computer program code are used by the device to operate using at least one processor. It is determined whether each of the first motion point sequence and the second motion point sequence contains a first motion pattern and a second motion pattern within each of the first motion point sequence and the second motion point sequence. Based on the result of the determination, the start and end of the motion point sequence for the person to perform the task are identified based on the first motion pattern and the second motion pattern. It is configured in such a way. The apparatus described in any one of the appendices 14 to 23. (Note 25) The at least one memory and computer program code are used by the device to operate using at least one processor. Obtain the first set of video frames and the second set of video frames from two different videos. The apparatus described in any one of the items 14 to 24 of the appendix. (Note 26) The at least one memory and computer program code are transmitted to the device using at least one processor, It is configured to make it happen. From the first motion point sequence and the second motion point sequence, respectively, first data and second data relating to one or more body parts of the person executing the motion point sequence are extracted. To determine whether the amount of the first data that is not present in the second data is greater than a threshold amount, On the detection area, at least a portion of the first data that is not present in the second data is displayed in a different color from the other portions of the first data and the second data. It is configured to perform the following: The apparatus described in any one of the appendices 14 to 25. (Note 27) A system for identifying a task performed by a person from a series of video frames, comprising the apparatus described in any one of Appendix 14 to 26, and at least one video capture device configured to generate the first series of video frames and the second series of video frames. [Explanation of symbols]

[0073] 400 System 402 Video Capture Devices 404 Equipment 406 Processors 408 memory 410 Databases 502 Setup Phase 504 Operation Phase 506 Sample Video 508 Camera 1800 Computing Devices 1802 Display Interface 1804 Processor 1806 Communication Infrastructure 1808 Main Memory 1810 Secondary Memory 1812 Storage Drive 1814 Removable Storage Drive 1818 Removable Storage Media 1820 interface 1822 Removable Storage Unit 1824 Communication Interface 1826 Communication Pass 1830 Display 1832 Audio Interface 1834 Speaker

Claims

1. The processor determines whether each of the first motion point sequence and the second motion point sequence contains a first motion point of a first ordinal number and a second motion point of a second ordinal number, and the first motion point sequence and the second motion point sequence contain the first and second motion points of one or more body parts of a person, which are ordered according to the respective points in time when the first and second motion points are detected from a first and second series of video frames corresponding to the detection region. Depending on the result of the determination, the processor identifies the start and end of the movement point sequence for the person to perform the task from the first movement point sequence and / or the second movement point sequence, based on the first movement point and the second movement point. A method for identifying tasks performed by a person from a series of video frames.

2. The first motion point sequence includes the first motion point in the first motion point sequence at a third ordinal number that is separated from the first ordinal number by an ordinal number count, and the method is The third order number of the first movement point and the subsequent movement points in the first movement point sequence are incremented by the order number count so that the first movement point becomes the first order number of the first movement point sequence, and after the increment, the first movement point sequence and the second movement point sequence are determined respectively. The method according to claim 1.

3. The increase in the third ordinal number of the first movement point and the ordinal number of subsequent movement points includes increasing the number of the plurality of movement points in the first movement point sequence by the ordinal number count. The method according to claim 2.

4. The first motion point sequence includes a third motion point in the first ordinal number, and the determination of the first motion point sequence and the second motion point sequence is as follows: Determining whether the third movement point is within a threshold distance from the first movement point, The method includes determining whether each of the first motion point sequence and the second motion point sequence contains, within each of the first motion point sequence and the second motion point sequence, the first motion point in the first ordinal number or the third motion point and the second motion point in the second ordinal number. The method according to any one of claims 1 to 3.

5. Detecting one or more body parts of the person in a portion of the detection area in each video frame of each of the first and second series of video frames, A method further comprising assigning motion points corresponding to a portion of the detection area in each of the first and second sets of video frames. The method according to any one of claims 1 to 4.

6. It is determined whether a first portion of the detection area in one of the first and second video frames and a second portion of the detection area in one of the first and second video frames are within one of a plurality of smaller detection areas of the detection area, and each of the plurality of smaller detection areas occupies different x and y coordinates within the detection area. The further includes assigning a single motion point corresponding to the first and second portions of the detection region, The method according to claim 5.

7. In each of the first and second series of video frames, detecting one or more body parts of the person in a part of the detection area is: This includes detecting a change in the direction of movement of one or more body parts of the person in a portion of the detection area in each of the first and second series of video frames, The method according to claim 5 or 6.

8. In a portion of the detection area, it is determined whether the angle between the previous direction of movement of one or more body parts of the person detected at a time prior to the detection of one or more body parts of the person, and the subsequent direction of movement of one or more body parts detected at a time later than the detection of one or more body parts of the person, is greater than a threshold angle. The detection of a change in the direction of movement of one or more body parts of the person is based on the angle determination result. The method according to claim 7.

9. It is determined whether one or more body parts of the person are stationary or moving within a portion of the detection area in each video frame of each of the first and second series of video frames, and the assignment of the motion points corresponding to the portion of the detection area is based on the determination result of one or more body parts of the person. The method according to any one of claims 5 to 8.

10. One of the first motion point and the second motion point is one of two motion points identified as the start and end of another motion point sequence sequence for the person to perform another task, from a third motion point sequence including a third plurality of motion points of the one or more body parts of the person detected from a first series of video frames, and a fourth motion point sequence including a fourth plurality of motion points of the one or more body parts of the person detected from a second series of video frames, wherein the task and the other task are two of a series of tasks ordered according to the respective time points at which the motion points of the motion point and the other motion point sequence were detected. The method according to any one of claims 1 to 9.

11. The determination of the first motion point sequence and the second motion point sequence is as follows: It is determined whether each of the first motion point sequence and the second motion point sequence contains a first motion pattern and a second motion pattern within each of the first motion point sequence and the second motion point sequence. The identification of the start and end of the motion point sequence for the person to perform the task, in accordance with the result of the determination, is based on the first motion pattern and the second motion pattern. The method according to any one of claims 1 to 10.

12. The first series of video frames and the second series of video frames are from two different videos. The method according to any one of claims 1 to 11.

13. From the first motion point sequence and the second motion point sequence, respectively, first data and second data relating to one or more body parts of the person executing the motion point sequence are extracted. Determining whether the amount of the first data that is not present in the second data is greater than a threshold amount, On the detection area, at least a portion of the first data that is not present in the second data is displayed in a different color from the other portions of the first data and the second data. including, The method according to any one of claims 1 to 12.

14. At least one processor, It comprises at least one memory containing computer program code, At least one memory and computer program code are used to provide the device with at least one processor, The first motion point sequence and the second motion point sequence are to be determined as to whether each of the first motion point sequence and the second motion point sequence contains a first motion point of a first ordinal number and a second motion point of a second ordinal number, and the first motion point sequence and the second motion point sequence contain a first plurality of motion points and a second plurality of motion points of one or more body parts of a person, which are ordered according to the respective time points at which a first plurality of motion points and a second plurality of motion points are detected from a first sequence of video frames and a second sequence of video frames corresponding to the detection region. Depending on the result of the determination, based on the first and second movement points, the start and end of the movement point sequence for the person to perform the task are identified from the first movement point sequence and / or the second movement point sequence. It is configured in such a way. A device for identifying tasks performed by a person from a series of video frames.

15. The first motion point sequence includes the first motion point in a third ordinal number that is separated from the first ordinal number by an ordinal number count within the first motion point sequence. The at least one memory and the computer program code are used by the at least one processor to provide the device with at least, The third order number of the first movement point and subsequent movement points in the first movement point sequence are incremented by the order number count so that the first movement point becomes the first order number of the first movement point sequence. The system determines whether each of the first motion point sequence and the second motion point sequence, after the increase, includes the first motion in the first ordinal number and the second motion point in the second ordinal number. It is configured in such a way. The apparatus according to claim 14.

16. The at least one memory and the computer program code are used by the at least one processor to provide the device with at least, As the third ordinal number of the first movement point and the ordinal number of subsequent movement points increase, the number of the plurality of movement points in the first movement point sequence is increased by the ordinal number count. The apparatus according to claim 15, configured as follows.

17. The first motion point sequence includes a third motion point in the first ordinal number, The at least one memory and the computer program code are used by the at least one processor to provide the device with at least, Determining whether the third movement point is within a threshold distance from the first movement point, Determining whether each of the first motion point sequence and the second motion point sequence includes, within each of the first motion point sequence and the second motion point sequence, the first motion point in the first ordinal number or the third motion point and the second motion point in the second ordinal number, It is configured to perform the following: The apparatus according to any one of claims 14 to 16.

18. The at least one memory and the computer program code are used by the at least one processor to provide the device with at least, Detecting one or more body parts of the person in a portion of the detection area in each video frame of each of the first and second series of video frames, Assigning motion points corresponding to a portion of the detection area in each video frame of each of the first and second series of video frames, It is configured to perform the following: The apparatus according to any one of claims 14 to 17.

19. The at least one memory and the computer program code are used by the at least one processor to provide the device with at least, It is determined whether a first portion of the detection area in one of the first and second video frames and a second portion of the detection area in one of the first and second video frames are within one of a plurality of smaller detection areas of the detection area, and each of the plurality of smaller detection areas occupies different x and y coordinates within the detection area. Assigning a single motion point corresponding to the first and second portions of the detection region, It is configured to perform the following: The apparatus according to claim 18.

20. The at least one memory and computer program code are used by the device to operate using at least one processor. The system detects a change in the direction of movement of one or more body parts of the person in a portion of the detection area in each video frame of each of the first and second series of video frames, and detects one or more body parts of the person in a portion of the detection area in each video frame of each of the first and second series of video frames. It is configured in such a way. The apparatus according to claim 18 or 19.

21. The at least one memory and computer program code are used by the device to operate using at least one processor. In a portion of the detection area, it is determined whether the angle between the previous direction of movement of one or more body parts of the person detected at a time prior to the detection of one or more body parts of the person, and the subsequent direction of movement of one or more body parts detected at a time later than the detection of one or more body parts of the person, is greater than a threshold angle, and based on the angle determination result, a change in the direction of movement of one or more body parts of the person is detected. It is configured in such a way. The apparatus according to claim 20.

22. The at least one memory and computer program code are used by the device to operate using at least one processor. It is determined whether one or more body parts of the person are stationary or moving within a portion of the detection area in each video frame of each of the first and second series of video frames. Based on the determination result of one or more body parts of the person, the motion point corresponding to a part of the detection area is assigned. It is configured in such a way. The apparatus according to any one of claims 18 to 21.

23. One of the first motion point and the second motion point is one of two motion points identified as the start and end of another motion point sequence sequence for the person to perform another task, from a third motion point sequence including a third plurality of motion points of the one or more body parts of the person detected from a first series of video frames, and a fourth motion point sequence including a fourth plurality of motion points of the one or more body parts of the person detected from a second series of video frames, wherein the task and the other task are two of a series of tasks ordered according to the respective time points at which the motion points of the motion point and the other motion point sequence were detected. The apparatus according to any one of claims 14 to 22.

24. The at least one memory and computer program code are used by the device to operate using at least one processor. The determination of the first motion point sequence and the second motion point sequence is as follows: It is determined whether each of the first motion point sequence and the second motion point sequence contains a first motion pattern and a second motion pattern within each of the first motion point sequence and the second motion point sequence. In accordance with the result of the determination, the start and end of the motion point sequence for the person to perform the task are identified. It is configured in such a way. The apparatus according to any one of claims 14 to 23.

25. The at least one memory and computer program code are used by the device to operate using at least one processor. Obtain the first set of video frames and the second set of video frames from two different videos. The apparatus according to any one of claims 14 to 24.

26. The at least one memory and computer program code are transmitted to the device using at least one processor, It is configured to make it happen. From the first motion point sequence and the second motion point sequence, respectively, first data and second data relating to one or more body parts of the person executing the motion point sequence are extracted. Determining whether the amount of the first data that is not present in the second data is greater than a threshold amount, On the detection area, at least a portion of the first data that is not present in the second data is displayed in a different color from the other portions of the first data and the second data. It is configured to perform the following: The apparatus according to any one of claims 14 to 25.

27. A system for identifying a task performed by a person from a series of video frames, comprising the apparatus according to any one of claims 14 to 26, and at least one video capture device configured to generate the first series of video frames and the second series of video frames.

Images