Periodic work recognition program, method, and device

Abstract

This periodic work recognition device associates each frame included in a video image, which is obtained by imaging a person who executes periodic work including a plurality of steps, with any of a prescribed number of anchor frames serving as a reference; assigns a character for identifying the associated anchor frame to each of the frames to thereby convert each of the frames into a first character string; summarizes, on the basis of the periodicity of changes of characters included in the first character string, a plurality of second character strings into which the first character string has been divided to generate third character strings; estimates a fourth character string indicating the workflow on the basis of a character string appearing in common in each of the third character strings; and recognizes, in the video image, a section from the start to the end of one cycle of the periodic work and breaks between the plurality of steps on the basis of association between the fourth character string and the first character string.

Description

Periodic operation recognition program, method, and apparatus

[0001] The disclosed technology relates to a periodic operation recognition program, a periodic operation recognition method, and a periodic operation recognition apparatus.

[0002] For example, videos of people being photographed are analyzed to recognize the movements and actions of the people. For example, videos of workers performing tasks in factories, warehouses, construction sites, etc. are analyzed for work management, etc. In the operations of factories, warehouses, construction sites, etc., there are operations that periodically repeat a plurality of processes as regular operations. Analyzing the workflow of such periodic operations is effective in improving work efficiency.

[0003] As technologies related to the recognition of human movements and actions, for example, technologies for detecting each interval of an operation executed repeatedly on a video have been proposed. Also, for example, technologies for recognizing sections corresponding to each process in a video of a workflow including a plurality of processes have been proposed. Also, for example, technologies for extracting a workflow from a text event log related to periodic time-series regular operations have been proposed.

[0004] International Publication No. 2022 / 162780

[0005] C. Panagiotakis, G. Karvounas and A. Argyros, "Unsupervised Detection of Periodic Segments in Videos," 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece, 2018, pp. 923-927.Fan Yang, Shigeyuki Odashima, Shoichi Masui, Shan Jiang, "Is Weakly-supervised Action Segmentation Ready For Human-Robot Interaction? No, Let's Improve It With Action-union Learning," IROS 2023.Wil Van der Aalst, "Process Mining: Data Science in Action," January 2016.

[0006] However, techniques for detecting repetitive actions cannot recognize workflows that involve multiple steps. Furthermore, techniques for recognizing intervals between each step in a workflow use trimmed video from the start to the end of the workflow, making it impossible to detect periodic work intervals from untrimmed video. Additionally, these conventional techniques require training data for each step in order to recognize each step from the video. Moreover, techniques for extracting workflows from text event logs use discrete-time data to detect workflows, making them difficult to apply to techniques for recognizing actions that change over continuous time.

[0007] One aspect of the disclosure technology is the ability to recognize each segment of a periodic work and workflow from untrimmed video footage of the work segments.

[0008] In one embodiment, the disclosed technology associates each frame in a video of a person performing a periodic task involving multiple steps with one of a predetermined number of reference anchor frames. The disclosed technology then converts each of the frames into a first string by assigning a character to each of the frames that identifies the associated anchor frame. The disclosed technology also generates each of the third strings by summarizing each of the multiple second strings obtained by dividing the first string based on the periodicity of the changes in the characters contained in the first string. The disclosed technology also estimates a fourth string representing the workflow based on characters that appear in common in each of the third strings. The disclosed technology then recognizes the interval in the video from the start to the end of one cycle of the periodic task, and the boundaries of each of the multiple steps, based on the association between the fourth string and the first string.

[0009] One aspect of this approach is that it allows for the recognition of each section of a cyclical work process and the workflow from untrimmed footage of the work process.

[0010] This is a functional block diagram of a periodic work recognition device. This diagram illustrates anchor extraction. This diagram illustrates the raw transcript and dominant period. This diagram illustrates online processing. This diagram illustrates the splitting of the raw transcript and the generation of a summarized transcript. This diagram illustrates workflow estimation. This diagram illustrates the recognition of periodic work. This diagram shows an example of the recognition result. This block diagram outlines the configuration of a computer functioning as a periodic work recognition device. This flowchart shows an example of offline processing. This flowchart shows an example of online processing.

[0011] An example of an embodiment relating to the disclosed technology will be described below with reference to the drawings.

[0012] As shown in Figure 1, the periodic work recognition device 10 functionally includes a conversion unit 12, a division unit 14, an estimation unit 16, and a recognition unit 18. In addition, an anchor DB (Database) 22 and a workflow DB 24 are stored in a predetermined storage area of ​​the periodic work recognition device 10.

[0013] The periodic work recognition device 10 receives offline video during offline work and online video during online work. The offline video and online video are images of a person performing periodic work that includes multiple processes. The offline video includes footage from the start to the end of recording, while the online video is footage in which frames from the start of recording are input sequentially without waiting for the end of recording.

[0014] During offline processing, the conversion unit 12 extracts a predetermined number of frames (hereinafter referred to as "anchors") from the frames included in the offline video, as shown in Figure 2, which serve as a reference for recognizing individual processes included in a periodic operation. An anchor is an example of an "anchor frame" in the disclosed technology. The predetermined number may be, for example, 8 to 12 frames (9 frames in the example in Figure 2), depending on the number of types of processes included in the periodic operation.

[0015] Specifically, the conversion unit 12 classifies each frame contained in the offline video into a predetermined number of clusters based on image features, and extracts the frame corresponding to the centroid of each cluster as an anchor. The image features may be, for example, skeletal information of a person recognized from each frame, and pose information of a person identified based on that skeletal information. Alternatively, the conversion unit 12 may accept anchor selection from the user and extract a predetermined number of frames selected by the user as anchors.

[0016] The conversion unit 12 assigns a character to each of the extracted anchors to identify them. In the example in Figure 2, lowercase letters a to i are assigned to each anchor. Note that the characters assigned to the anchors are not limited to lowercase letters; uppercase letters, numbers, symbols, etc., may also be used.

[0017] The conversion unit 12 stores the extracted anchors in the anchor DB 22. The conversion unit 12 also stores in the anchor DB 22 information for identifying anchors that can be merged in the processing of the recognition unit 18, which will be described later. Mergeable anchors are anchors that exhibit behavior that can be considered identical (details will be described later). For example, the conversion unit 12 may use information of anchor groups whose similarity of image features is greater than or equal to a predetermined value as information for identifying mergeable anchors. The image features here may be the posture information of the person mentioned above, or the position of the person within the frame, etc. The conversion unit 12 may vectorize this information and calculate the cosine similarity between the vectors, etc., as the similarity of the image features. Alternatively, the conversion unit 12 may calculate the reciprocal of the sum of the differences in pixel values ​​of corresponding pixels between anchors as the similarity.

[0018] The conversion unit 12 associates each frame contained in the offline video with one of a predetermined number of anchors. Specifically, the conversion unit 12 associates each frame contained in the offline video with the anchor that is most similar to the predetermined number of anchors. Then, as shown in Figure 3A, the conversion unit 12 converts each frame contained in the offline video into a transcript by assigning a character to each frame that identifies the associated anchor. The transcript converted here will be referred to as the "raw transcript" below. The raw transcript is an example of the "first string of characters" of the disclosed technology.

[0019] Furthermore, during online processing, as shown in Figure 4A, the conversion unit 12 refers to the anchors stored in the anchor DB 22 and sequentially associates each frame included in the online video with one of the anchors, converting it into a raw transcript. The conversion unit 12 then sequentially passes the converted raw transcript portions to the recognition unit 18.

[0020] The splitting unit 14, during offline processing, splits the raw transcript based on the periodicity of character changes contained in the raw transcript. Specifically, as shown in Figure 3B, the splitting unit 14 sets the characters contained in the raw transcript as different values ​​on the vertical axis and the frame number on the horizontal axis, representing the raw transcript as a waveform. The splitting unit 14 then performs frequency analysis, such as a Fourier transform, on the waveform to identify the dominant frequency. The dominant frequency may be, for example, the frequency with the maximum intensity in the results of the frequency analysis. The splitting unit 14 calculates the dominant period T from the reciprocal of the dominant frequency.

[0021] As shown in Figure 5A, the splitting unit 14 divides the raw transcript by sliding a window of period T × α (where α is, for example, 1.5) by a predetermined width, and extracting the strings of the raw transcript contained in period T × α. Hereinafter, the divided transcript will be referred to as the "divided transcript". The divided transcript is an example of the "second string" of the disclosed technology. The dominant period T is estimated as the period corresponding to one interval of periodic work, but a margin is added to take into account variations in work time, etc.

[0022] As shown in Figure 5B, the splitting unit 14 generates each of the transcripts that summarize each of the multiple split transcripts. Hereinafter, the summarized transcript will be referred to as the "summary transcript". The summary transcript is an example of the "third string" of the disclosed technology. Specifically, the splitting unit 14 generates the summary transcript by treating consecutive identical characters in the split transcript as a single character. For example, the splitting unit 14 generates the summary transcript "abc" from the split transcript "aabbbbccc".

[0023] The estimation unit 16 estimates a string representing a workflow that corresponds to a sequence of steps included in a periodic task, based on strings that appear in common in each of the summary transcripts. Hereinafter, the string representing the workflow will also simply be referred to as "workflow." The workflow is an example of the "fourth string" of the disclosed technology. The workflow can be described as a sequence of characters that indicate the order of steps included in a periodic task, with each step being an anchor. Specifically, as shown in Figure 6A, the estimation unit 16 extracts the longest common string in each of the summary transcripts, allowing for some non-matching characters.

[0024] More specifically, the estimation unit 16 searches for matching strings in each summary transcript (in the example in Figure 6, "fbedaeibgc"), replaces non-matching characters with predetermined symbols (in the example in Figure 6, "_ (underscore)"), and further extends the matching strings. As shown in Figure 6B, the estimation unit 16 extracts the string that has been searched up to the point where the matching string can no longer be extended as the longest common string in each summary transcript.

[0025] The estimation unit 16 identifies the string portion that appears repeatedly within the common longest string (hereinafter referred to as the "repeating string") (in the example in Figure 6, "fbe"). The estimation unit 16 estimates the first character of the repeating string that appears for the first time within the common longest string as the beginning of the workflow. The estimation unit 16 also estimates the last character that appears within the common longest string, excluding the repeating string that appears for the second time or later and symbols indicating non-matching characters, as the end of the workflow. As a result, the estimation unit 16 estimates the workflow as shown in Figure 6C. Figure 6D represents the workflow in graph form, including nodes (circles in Figure 6D) and edges (arrows in Figure 6D). In the example in Figure 6D, the nodes show a combination of the character corresponding to that node and the order of the process corresponding to that node (0, 1, 2, ...). The estimation unit 16 stores the estimated workflow in the workflow DB 24.

[0026] During offline processing, the recognition unit 18 uses the estimated workflow to segment the raw transcript into sections corresponding to one cycle of periodic work. Specifically, as shown in Figure 7, the recognition unit 18 recognizes the sections in the offline video from the start to the end of one cycle of periodic work, and the boundaries of each of the multiple processes included in the periodic work, based on the correspondence between the workflow and the raw transcript.

[0027] More specifically, the recognition unit 18 recognizes the portion of the raw transcript that matches the workflow when consecutively identical characters are merged and represented as a single character, as an interval from the start to the end of one cycle of periodic work. Characters are considered identical when the characters are exactly the same, or when the similarity between the anchors corresponding to the characters is greater than or equal to a predetermined value. The recognition unit 18 determines whether the similarity between the anchors is greater than or equal to a predetermined value based on information stored in the anchor DB 22 for identifying mergeable anchors.

[0028] For example, let's consider the case of associating the workflow shown in Figure 7 with the raw transcript. The recognition unit 18 recognizes the frame corresponding to the first character of the raw transcript as one of the frame groups indicating the first step of the workflow, because the first character "f" of the raw transcript matches the first character "f" of the workflow. The recognition unit 18 also determines whether it is possible to merge "e" into "f" because the second character "e" of the raw transcript does not match the first character "f" of the workflow. Here, it is assumed that it is possible to merge "e" into "f". In this case, the recognition unit 18 also recognizes the frame corresponding to the second character of the raw transcript as one of the frame groups indicating the first step of the workflow. Similarly, the recognition unit 18 recognizes the third character of the raw transcript as one of the frame groups indicating the first step of the workflow.

[0029] Next, regarding the fourth character of the raw transcript, if it is determined that "b" cannot be merged with "f", the recognition unit 18 changes the target to be associated with the raw transcript to the second character of the workflow and continues processing. That is, the recognition unit 18 recognizes the frame corresponding to the fourth character of the raw transcript as one of the frame groups indicating the second step of the workflow, because the fourth character "b" of the raw transcript matches the second character "b" of the workflow. The recognition unit 18 repeats the above process and recognizes the portion of the raw transcript up to the point where the association with the last character of the workflow is completed as a single section (in the example of Figure 7, "Period 1"). The recognition unit 18 returns to the first character of the workflow and repeats the above process from the character following Period 1 of the raw transcript. As a result, as shown in Figure 7, Period 2, Period 3, ... are recognized, and the boundaries of each step are recognized in the raw transcript based on the parts corresponding to each character included in the workflow.

[0030] The recognition unit 18 creates recognition results as shown in Figure 8. In the example in Figure 8, the recognition results include the number of each section (e.g., Period 1), the frame numbers of the first and last frames of that section, and indicators showing the divisions of processes within each section. The indicators showing the divisions may be, for example, a plot of the indicator showing the process on the frame group corresponding to each process included in each section. In the example in Figure 8, a different color is assigned to each character indicating an anchor, i.e., each process included in the workflow ("Action Unit" in Figure 8). The display plots the colors assigned to the recognized processes for each frame within a band-shaped figure where each horizontal position corresponds to a frame number. In the example in Figure 8, the difference in color is represented by differences in hatching or halftone dot types. The recognition unit 18 outputs the created recognition results.

[0031] Furthermore, during online processing, the recognition unit 18 associates the workflow stored in the workflow DB 24 with the raw transcripts sequentially received from the conversion unit 12, as shown in Figure 4B. This allows the recognition unit 18 to sequentially recognize the start point of one cycle of the periodic work and the boundaries of each of the multiple processes, recognizing the end point of the last process as the end point of one cycle of the periodic work. Additionally, as shown in Figure 4C, the recognition unit 18 sequentially creates and outputs the recognition results from the recognized portions.

[0032] The periodic work recognition device 10 may be implemented, for example, by a computer 40 as shown in Figure 9. The computer 40 includes a CPU (Central Processing Unit) 41, a GPU (Graphics Processing Unit) 42, a memory 43 as a temporary storage area, and a non-volatile storage device 44. The computer 40 also includes input / output devices 45 such as input devices and display devices, and an R / W (Read / Write) device 46 that controls the reading and writing of data to and from the storage medium 49. The computer 40 also includes a communication interface 47 that connects to a network such as the Internet. The CPU 41, GPU 42, memory 43, storage device 44, input / output devices 45, R / W device 46, and communication interface 47 are connected to each other via a bus 48.

[0033] The storage device 44 is, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or flash memory. The storage device 44 stores a periodic work recognition program 50 that causes the computer 40 to function as a periodic work recognition device 10. The periodic work recognition program 50 includes a conversion process control command 52, a division process control command 54, an estimation process control command 56, and a recognition process control command 58. The storage device 44 also has an information storage area 60 in which information constituting the anchor DB 22 and the workflow DB 24 is stored.

[0034] The CPU 41 reads the periodic work recognition program 50 from the storage device 44, loads it into memory 43, and sequentially executes the control instructions contained in the periodic work recognition program 50. The CPU 41 operates as the conversion unit 12 shown in Figure 1 by executing the conversion process control instruction 52. The CPU 41 also operates as the division unit 14 shown in Figure 1 by executing the division process control instruction 54. The CPU 41 also operates as the estimation unit 16 shown in Figure 1 by executing the estimation process control instruction 56. The CPU 41 also operates as the recognition unit 18 shown in Figure 1 by executing the recognition process control instruction 58. The CPU 41 also reads information from the information storage area 60 and loads the anchor DB 22 and workflow DB 24 into memory 43. As a result, the computer 40 that executed the periodic work recognition program 50 functions as a periodic work recognition device 10. The CPU 41 that executes the program is hardware. Part of the program may be executed by the GPU 42.

[0035] The functions realized by the periodic work recognition program 50 may be implemented, for example, by a semiconductor integrated circuit, more specifically by an ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), etc.

[0036] Next, the operation of the periodic work recognition device 10 according to this embodiment will be described. When offline video is input to the periodic work recognition device 10, the offline processing shown in Figure 10 is executed in the periodic work recognition device 10. When online video is input to the periodic work recognition device 10, the online processing shown in Figure 11 is executed in the periodic work recognition device 10. Note that the offline processing and online processing are examples of the periodic work recognition method of the disclosed technology.

[0037] First, please refer to Figure 10 to explain the offline process.

[0038] In step S10, the conversion unit 12 acquires the offline video input to the periodic work recognition device 10. Next, in step S12, the conversion unit 12 extracts a predetermined number of anchors from the frames included in the offline video. The conversion unit 12 then assigns a character to each of the extracted anchors to identify them, and stores them in the anchor DB 22 along with information for identifying mergeable anchors.

[0039] Next, in step S14, the conversion unit 12 associates each frame included in the offline video with the most similar anchor from a predetermined number of anchors. Then, the conversion unit 12 assigns a character to each frame included in the offline video to identify the associated anchor, thereby converting each frame included in the offline video into a raw transcript.

[0040] Next, in step S16, the splitting unit 14 performs frequency analysis on the waveform representing the raw transcript to identify the dominant frequency and calculates the dominant period T from the reciprocal of the dominant frequency. Next, in step S18, the splitting unit 14 divides the raw transcript into multiple split transcripts by sliding a window of period T × α by a predetermined width and extracting the string of the raw transcript contained in period T × α. Next, in step S20, the splitting unit 14 summarizes each of the split transcripts by treating consecutive identical characters in the split transcript as a single character, thereby generating each of the summarized transcripts.

[0041] Next, in step S22, the estimation unit 16 extracts the longest common string from each of the summarized transcripts, allowing for the inclusion of some non-matching characters. The estimation unit 16 then estimates the beginning of the workflow from the first character of the repeating string that appears within the longest common string. Furthermore, the estimation unit 16 estimates the workflow by excluding the repeating string that appears for the second time or later, and symbols indicating non-matching characters, from the last character that appears within the longest common string, thereby estimating the workflow.

[0042] Next, in step S24, the recognition unit 18 recognizes, based on the association between the workflow and the raw transcript, the section from the start to the end of one cycle of the periodic work and the delimiters of each of the plurality of steps included in the periodic work in the offline video. Next, in step S26, the recognition unit 18 creates a recognition result indicating the recognized section and the delimiters of the steps, outputs the created recognition result, and the offline process ends.

[0043] Next, referring to FIG. 11, the online process will be described.

[0044] In step S30, the conversion unit 12 acquires frames of a predetermined number of minutes of the online video sequentially input to the periodic work recognition device 10. Next, in step S32, the conversion unit 12 refers to the anchors stored in the anchor DB 22, associates each of the frames included in the online video with the most similar anchor sequentially, converts it into a raw transcript, and passes it to the recognition unit 18.

[0045] Next, in step S34, the recognition unit 18 associates the workflow stored in the workflow DB 24 with the raw transcript sequentially passed from the conversion unit 12, and recognizes the section of the periodic work and the delimiters of the steps. Next, in step S36, the recognition unit 18 creates recognition results sequentially from the portions where the section and the delimiters of the steps are recognized, and outputs the created recognition results.

[0046] Next, in step S38, the conversion unit 12 determines whether or not the final frame of the online video has been acquired. If subsequent frames are continuously input to the periodic work recognition device 10, the process returns to step S30. If the final frame has been acquired, the online process ends.

[0047] As described above, the periodic work recognition device according to the present embodiment associates each frame included in a video of a person performing a periodic work including a plurality of steps with any one of a predetermined number of anchors serving as a reference. The periodic work recognition device converts each frame of the video into a raw transcript by assigning a character for identifying the associated anchor to each frame. Further, the periodic work recognition device divides the raw transcript into a plurality of divided transcripts based on the periodicity of changes in the characters included in the raw transcript, and generates each of the summary transcripts by summarizing each of the divided transcripts. Then, the periodic work recognition device estimates the workflow based on a character string that commonly appears in each of the summary transcripts. Further, the periodic work recognition device recognizes, in the video, a section from the start to the end of one cycle of the periodic work and the boundary of each of the plurality of steps based on the association between the workflow and the raw transcript. Thereby, each section of the periodic work and the workflow can be recognized from a video that has not been trimmed in the work section. Thus, the technology of the present disclosure improves the functions of a computer.

[0048] The recognition result by the periodic work recognition device according to the present embodiment can be used, for example, for the analysis of periodic routine work in factories, warehouses, construction sites, etc. For example, the working time for each step can be aggregated, or the working time for each step can be compared between a new worker and a skilled worker, improving the efficiency of work management.

[0049] In the above embodiment, the periodic work recognition program is stored (installed) in the storage device in advance, but the present invention is not limited to this. The program according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

[0050] 10 Periodic work recognition device 12 Conversion unit 14 Splitting unit 16 Estimation unit 18 Recognition unit 22 Anchor DB 24 Workflow DB 40 Computer 41 CPU 42 GPU 43 Memory 44 Storage device 45 Input / output device 46 R / W device 47 Communication I / F 48 Bus 49 Storage medium 50 Periodic work recognition program 52 Conversion process control command 54 Splitting process control command 56 Estimation process control command 58 Recognition process control command 60 Information storage area

Claims

1. A periodic work recognition program for causing a computer to perform a process that includes: associating each frame in a video of a person performing a periodic work consisting of multiple steps with one of a predetermined number of reference anchor frames, assigning a character to each frame to identify the associated anchor frame, thereby converting each frame into a first string; generating each of a third string by summarizing each of a plurality of second strings obtained by dividing the first string based on the periodicity of the changes in the characters contained in the first string; estimating a fourth string indicating the workflow based on a character that appears in common in each of the third strings; and recognizing the interval in the video from the start to the end of one cycle of the periodic work, and the boundaries of each of the plurality of steps, based on the association between the fourth string and the first string.

2. A periodic work recognition program according to claim 1, which stores a predetermined number of anchor frames and the fourth string, sequentially associates each frame included in the input online video with one of the anchor frames and converts it into the first string, sequentially associates the converted first string with the fourth string to sequentially recognize the start point of one cycle of the periodic work and the boundaries of each of the multiple processes in the video, and recognizes the end point of the last process as the end point of one cycle of the periodic work.

3. The periodic work recognition program according to claim 1 or 2, wherein the process of converting each of the frames into a first string includes associating each of the frames with the most similar anchor frame among the predetermined number of anchor frames.

4. The periodic work recognition program according to claim 1 or claim 2, wherein the anchor frame is the frame corresponding to the centroid of each of the predetermined number of clusters obtained by classifying each frame included in the video based on image features, or the predetermined number of frames selected by the user.

5. The periodic work recognition program according to claim 1 or claim 2, wherein the second string is obtained by representing the first string as a waveform, assigning a different value to each character in the first string, and dividing the first string at a period that takes into account a margin in addition to the dominant period obtained by frequency analysis of the waveform.

6. The periodic work recognition program according to claim 1 or claim 2, wherein the third string is generated by making a single character the portion of the second string in which the same character is repeated.

7. The periodic work recognition program according to claim 1 or 2, wherein the process for identifying the fourth string includes, in each of the third strings, allowing for the inclusion of some non-matching characters, extracting the longest common string, estimating the first character of a repeatedly appearing string portion within the extracted string as the beginning of the fourth string, and estimating the last character of the fourth string, excluding subsequent occurrences of the string portion and the non-matching characters.

8. A periodic work recognition program according to claim 1 or 2, which recognizes a portion of the first string of characters that matches the fourth string of characters when consecutively appearing characters that can be considered identical are represented by a single character, as an interval from the start to the end of one period of the periodic work.

9. The periodic work recognition program according to claim 8, wherein the characters that can be considered identical are those in which the characters are exactly the same, and those in which the similarity between the anchor frames corresponding to the characters is greater than or equal to a predetermined value.

10. The periodic work recognition program according to claim 1 or 2, wherein each of the plurality of processes is recognized based on the portion of the first string that corresponds to each of the characters contained in the fourth string.

11. A method for recognizing a periodic task, in which a computer performs a process that includes: associating each frame in a video of a person performing a periodic task involving multiple steps with one of a predetermined number of reference anchor frames, assigning a character to each frame to identify the associated anchor frame, thereby converting each of the frames into a first string; generating each of a third string by summarizing each of a plurality of second strings obtained by dividing the first string based on the periodicity of the changes in the characters contained in the first string; estimating a fourth string indicating a workflow based on a character that appears in common in each of the third strings; and recognizing the interval in the video from the start to the end of one cycle of the periodic task, and the boundaries of each of the plurality of steps, based on the association between the fourth string and the first string.

12. A method for recognizing a periodic work according to claim 11, comprising: storing a predetermined number of anchor frames and the fourth string; sequentially associating each frame included in the input online video with one of the anchor frames and converting it into the first string; sequentially associating the converted first string with the fourth string to sequentially recognize the starting point of one cycle of the periodic work and the boundaries of each of the multiple processes in the video; and recognizing the ending point of the last process as the ending point of one cycle of the periodic work.

13. The periodic work recognition method according to claim 11 or claim 12, wherein the process of converting each of the frames into a first string includes associating each of the frames with the most similar anchor frame among the predetermined number of anchor frames.

14. The periodic work recognition method according to claim 11 or claim 12, wherein the anchor frame is the frame corresponding to the centroid of each of the predetermined number of clusters obtained by classifying each frame included in the video based on image features, or the predetermined number of frames selected by the user.

15. The periodic work recognition method according to claim 11 or claim 12, wherein the second string is obtained by representing the first string as a waveform, assigning a different value to each character in the first string, and dividing the first string at a period that takes into account a margin to the dominant period obtained by frequency analysis of the waveform.

16. The periodic work recognition method according to claim 11 or claim 12, wherein the third string is generated by making a single character the portion of the second string in which the same character is repeated.

17. The periodic work recognition method according to claim 11 or claim 12, wherein the process for identifying the fourth string includes, in each of the third strings, allowing for the inclusion of some non-matching characters, extracting the longest common string, estimating the first character of a repeatedly appearing string portion within the extracted string as the beginning of the fourth string, and estimating the last character of the fourth string, excluding subsequent occurrences of the string portion and the non-matching characters.

18. A method for recognizing periodic work according to claim 11 or 12, wherein, in the first string of characters, when consecutively appearing characters that can be considered identical are represented by a single character, the portion that matches the fourth string of characters is recognized as the interval from the start to the end of one period of the periodic work.

19. The periodic work recognition method according to claim 18, wherein the characters that can be considered identical are those in which the characters are exactly the same, and those in which the similarity between the anchor frames corresponding to the characters is greater than or equal to a predetermined value.

20. A periodic work recognition device, comprising a processor, which performs the following processes: associating each frame in a video of a person performing a periodic work consisting of multiple steps with one of a predetermined number of reference anchor frames, assigning a character to each frame that identifies the associated anchor frame, thereby converting each of the frames into a first string; generating each of a third string by summarizing each of a plurality of second strings obtained by dividing the first string based on the periodicity of the changes in the characters contained in the first string; estimating a fourth string indicating a workflow based on a character that appears in common in each of the third strings; and recognizing the interval in the video from the start to the end of one cycle of the periodic work, and the boundaries of each of the plurality of steps, based on the association between the fourth string and the first string.

Images