Work analysis method, information processing system, and program

Abstract

To provide a work analysis method, an information processing system, and a program capable of more accurately specifying a work period of repetitive work.SOLUTION: The work analysis method includes the following steps. (1) A setting step of setting in advance a captured image of a motion of a worker at a start of predetermined work as a reference start image, and setting in advance a captured image of a motion of the worker at an end of the work as a reference end image. (2) An acquisition step of acquiring imaging data of predetermined work repeated a plurality of times. (3) An extraction step of extracting, from the acquired photographing data, a first candidate of an action at a start time based on a first similarity with a reference start image, and extracting a second candidate of an action at an end time based on a second similarity with a reference end image. (4) A specifying step of specifying an operation period by evaluating validity of a third candidate for the operation period defined by a combination of the extracted first candidate and second candidate.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a work analysis method, an information processing system, and a program. [Background technology]

[0002] Increasing work efficiency is also important in workplaces where products are assembled manually. Patent Document 1 discloses a technology that extracts two types of recurring movements from video data of a worker's work and estimates the time required for activities other than work movements based on the time interval between the two types of movements. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-3491 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when extracting an arbitrary movement from image data, it is difficult to deal with the case where the arbitrary movement is erroneously extracted, which poses a problem that it is difficult to improve accuracy.

[0005] An object of the present invention is to provide a task analysis method, an information processing system, and a program that can more accurately identify the task period of a repetitive task. [Means for solving the problem]

[0006] In order to achieve the above object, one aspect of the present invention is a setting step of setting in advance a photographed image of the worker's movement at the start of a predetermined task as a reference start image, and setting in advance an image of the worker's movement at the end of the task as a reference end image; an acquisition step of acquiring photographic data of the predetermined work repeated a plurality of times; an extraction step of extracting a first candidate for the action at the start from the acquired photographic data based on a first similarity with the reference start image, and extracting a second candidate for the action at the end based on a second similarity with the reference end image; a specifying step of evaluating the validity of a third candidate for the work period defined by a combination of the extracted first candidate and the extracted second candidate, and specifying the work period; This is a work analysis method that includes the following.

[0007] Another aspect of the present invention is A photographed image of the worker's movement at the start of a predetermined task is set in advance as a reference start image, and a photographed image of the worker's movement at the end of the task is set in advance as a reference end image, Acquire photographic data of the predetermined work repeated multiple times; extracting a first candidate for the action at the start based on a first similarity with the reference start image from the acquired photographic data, and extracting a second candidate for the action at the end based on a second similarity with the reference end image; identifying a work period by evaluating the validity of a third candidate for the work period defined by a combination of the extracted first candidate and the extracted second candidate; The information processing system includes a control unit.

[0008] Another aspect of the present invention is Computer a setting means for setting in advance a photographed image of a worker's movement at the start of a predetermined task as a reference start image, and setting in advance an image of a worker's movement at the end of the task as a reference end image; an acquisition means for acquiring photographic data of the predetermined work repeated multiple times; an extraction means for extracting a first candidate for the action at the start from the acquired photographic data based on a first similarity with the reference start image, and extracting a second candidate for the action at the end based on a second similarity with the reference end image; an identification means for evaluating the validity of a third candidate for the work period defined by a combination of the extracted first candidate and the extracted second candidate, and identifying the work period; It is a program that functions as a [Effects of the Invention]

[0009] According to the present invention, it is possible to more accurately identify the work period of a repetitive work. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 is a block diagram showing a functional configuration of the information processing device. [Figure 2] FIG. 10 is a diagram illustrating a procedure of a work detection operation. [Figure 3] 10A and 10B are diagrams illustrating an example of a change in the horizontal position of the right wrist. [Figure 4] FIG. 1 is a diagram illustrating dynamic programming. [Figure 5] 10A and 10B are diagrams illustrating examples of extracted candidates for work start and work end, and examples of identified work start and work end timings. [Figure 6] 10 is a graph showing the maximum evaluation value for the number of provisional tasks. [Figure 7] 10 is a flowchart showing a control procedure of a work analysis control process. [Figure 8] 10 is a flowchart showing another example of the work analysis control process. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the functional configuration of an information processing device 1 in the information processing system of this embodiment.

[0012] The information processing device 1 may be a computer (electronic calculator) such as a normal PC (Personal Computer). The information processing device 1 includes a control unit 11, a RAM 12, a storage unit 13, a display unit 14, an operation reception unit 15, a communication unit 16, etc. The control unit 11 has a hardware processor that performs arithmetic processing and controls the overall operation of the information processing device 1. The control unit 11 is configured to be included in a computer at a minimum. The hardware processor may be a general-purpose CPU (Central Processing Unit). Alternatively, the hardware processor may operate multiple CPUs in parallel, or multiple CPUs may perform arithmetic processing independently depending on the application. Furthermore, some or all of the hardware processor may be designed to be suitable for a specific application, such as image processing.

[0013] The RAM 12 provides a working memory space for the control unit 11 and stores temporary data. The RAM 12 may be, for example, a DRAM.

[0014] The storage unit 13 has a nonvolatile memory that stores the program 131, setting data, etc. The nonvolatile memory may be, for example, a flash memory. The nonvolatile memory may also include an HDD (Hard Disk Drive). Part or all of the storage unit 13 may be an auxiliary storage device that is externally attached to the information processing device 1 as a peripheral device. Alternatively, part or all of the storage unit 13 may be a network drive on a network, a cloud server, or the like. The storage unit 13 also stores captured image data to be analyzed.

[0015] The program 131 detects the start and end of a specific task from captured images of the repetition of the task, and identifies the period of the task for a set number of times. The program 131 includes a skeleton recognition process 1311. The skeleton recognition process 1311 identifies specific points related to the skeleton of the subject from the captured images, such as the eyes, nose, both ends of the mouth, neck, shoulders, elbows, wrists, and waist. If the work is standing work and the lower body is also captured, the knees, ankles, toes, and the like may also be identified in addition to the above.

[0016] The setting data may include task information 132 of the subject, who is a worker performing a predetermined task. The task information 132 may include information on the number of times the subject repeated a task during the shooting period.

[0017] The display unit 14 may have a digital display screen that digitally displays various information. The digital display screen may be, for example, a liquid crystal display screen or an organic EL (Electro-Luminescent) display screen. The display unit 14 may be a peripheral device externally attached to the information processing device 1.

[0018] The operation reception unit 15 receives an input operation from an external device such as a user, and outputs an electrical signal corresponding to the input operation to the control unit 11. The operation reception unit 15 may include a keyboard, a pointing device such as a mouse, and / or a touch panel. A part or all of the operation reception unit 15 may be a peripheral device externally attached to the information processing device 1.

[0019] The communication unit 16 controls communication connections with external devices via a LAN (Local Area Network), a wireless LAN, or the Internet. The communication unit 16 may include a network card or the like. The communication unit 16 may also be directly connectable to external devices or peripheral devices via a USB (Universal Serial Bus) or short-range wireless communication. The external devices connected in this manner may include the image capture device 9 or another information processing device that receives image capture data from the image capture device 9. Alternatively, the communication unit 16 may have a terminal for connecting a portable recording medium, such as an SD card, on which image capture data is recorded.

[0020] The image data acquired from the image capture device 9 and subjected to analysis is video. However, as long as characteristic postures and movements can be identified as described below, the number of captured images per unit time, i.e., the number of frames, may be small enough that the movement does not appear smooth when viewed visually. For example, the image data may be images of about a few frames per second. Furthermore, the captured images do not need to be data in a video format. The captured images may also be continuous still images.

[0021] Next, the task detection operation according to the task analysis method of this embodiment will be described. The information processing device 1 of this embodiment detects the start and end of a specific task repeatedly performed by a worker from captured images. This identifies each worker's work time and non-work periods between work periods. The specific task here may primarily be work performed in a factory, such as product assembly, adjustment, or inspection. Non-work periods include breaks and waiting, including absences. Breaks may include scheduled breaks, necessary breaks, and slacking off. Waiting may include the time from the end of a task until the completion of an upstream task in the work process and the worker can begin the next task. The task detection operation of this embodiment does not need to be performed in real time. The captured images of a certain time range are later collected and used as task detection targets.

[0022] FIG. 2 is a diagram showing the procedure of the work detection operation. The information processing device 1 acquires data of photographed images of work performed by the person being analyzed from the photographing device 9 directly or via another information processing device (P1; acquisition step, acquisition means). The photographed images show work that has been repeated multiple times, and it is preferable that the number of times of work is large enough that statistical variation does not need to be taken into account. Specifically, the number of times of work may be 10 or more, or may be 100 or more. The work time of the person being analyzed is identified from the acquired photographed image data.

[0023] Information on the number of tasks and information on the reference time (standard time span), which is the average task time, may be input separately from information such as the number of products obtained by the task (P2). The reference time does not have to be limited to the task time of the person being analyzed. If there are multiple workers performing the same task, it may be determined based on the task time of those multiple workers. The reference time does not have to be determined based on the actual task. The time assumed by the manager as the reference time may be simply input.

[0024] The movement of the person being analyzed may be identified by the positions, relative positional relationships, and changes thereof of a predetermined number of skeleton points identified by the skeleton recognition process 1311 in the program 131. The skeleton recognition process 1311 may be performed by inputting a target image into a trained model. The trained model may be any algorithm, as long as it is an algorithm related to image recognition. As the trained model, a general-purpose model that has already been trained may be acquired externally. Alternatively, a person in charge may generate correct answer data for skeleton positions for previously acquired photographed images of work and associate the correct answer data with the photographed images to generate training data (P3). An untrained machine learning model may be trained using this training data to obtain a trained model related to the skeleton recognition process 1311 that is specialized for images of work (P4). Note that even when generating a trained model, the photographed images included in the training data do not have to be limited to photographed images of work. The photographed image data for machine learning may be captured and acquired separately from the photographed images to be analyzed.

[0025] The behavior to be analyzed may be, for example, items that an administrator or other person has visually noticed, etc. That is, it does not have to be a feature that is difficult for humans to understand and is expressed in multiple dimensions (e.g., 50 or more dimensions) using a machine learning model or the like.

[0026] By inputting each frame of captured image data into the trained model, time series data of the skeleton position in each frame can be obtained (P7).

[0027] FIG. 3 is a diagram showing an example of a change in the horizontal position of the right wrist. Tasks involving body movements such as those described above involve postures specific to the task, i.e., the relative position of the skeleton and its changes. Here, the position of the right wrist is identified as the skeleton position, and the change in the X-coordinate, which is the horizontal component, in the captured image is shown. The horizontal orientation of the captured image may be parallel to the horizontal direction. Similar movements are observed repeatedly in the horizontal position of the right wrist, as seen in the ranges W1 to W7. This represents the right wrist movements associated with multiple repetitions of the task. The information processing device 1 further detects characteristic skeletal positions and movements at the start and end of such tasks to extract candidates for the start and end of the task. The start and end of the task here refer to a period that spans a predetermined time from the start of the task and a predetermined time before the end of the task. The predetermined time should be set within a range that ensures uniqueness of the relative position and movement of the skeleton relative to other sections, is not too high in relation to the duration of a single task, and is unlikely to be affected by variations in the movements of each session. For example, for a task that typically requires one minute, the predetermined time may be set to approximately five seconds. Alternatively, this predetermined time does not have to be set in advance. Multiple candidates for the predetermined time may be set, and subsequent processes may be executed, with the candidate time that ultimately yields the best results being set as the result for the task being analyzed. Furthermore, the predetermined time may be changed for each task based on the uniqueness of characteristic actions corresponding to the task. The predetermined time for the start period and the predetermined time for the end period may be different from each other.

[0028] The start and end periods of the above-mentioned predetermined task to be detected are selected and set by a person in charge or the like as reference start images and reference end images, respectively, by selecting typical movement portions from the captured image data for machine learning. These reference start images and reference end images are collectively referred to as a reference image group. In addition, the predetermined time range is set by input operations or the like by the person in charge (P5; setting step, setting means). The reference image group within the set range is input into the generated trained model, and the skeleton positions in the reference image group are obtained (P6).

[0029] The skeletal positions in each of the reference images obtained in P6 are compared with the skeletal positions in each image to be analyzed obtained in P7 to determine the degree of similarity (P8). The degree of similarity relative to the reference start image is defined as the first similarity, and the degree of similarity relative to the reference end image is defined as the second similarity. The degree of similarity may be quantitatively evaluated by vector calculation using feature vectors representing the features of each image. For example, for each frame, the sum or square sum of the Euclidean distances between each skeletal position in the captured image and the reference images may be calculated. In this case, the coordinates on the image may be used directly as a position vector from the origin, and the difference vector may be used to calculate the Euclidean distance. Depending on the action, the skeletal position relative to a specific origin of the subject, such as the intersection of a line connecting both shoulders and a vertical line passing through the neck, may also be expressed as a feature vector. Furthermore, the relative positional relationship between multiple skeletal positions in each image may be expressed as a feature vector, and the Euclidean distance between the feature vectors may be considered. For example, the orientation of the right forearm, represented by a vector from the elbow to the wrist, may be used to evaluate similarity. In this way, the degree of similarity can be easily quantified by numerically expressing the magnitude of mismatch. Note that when there are multiple items to be compared and matched, the magnitudes of mismatch for the multiple items may be weighted and added together.

[0030] Furthermore, the first and second similarity values ​​may be obtained by adding the similarity values ​​obtained for each frame for all frames in a predetermined period. Alternatively, the cosine similarity of the feature vectors to be compared may be used instead of the Euclidean distance to calculate the similarity. The shooting direction and magnification of the image capture device 9 may be fixed, and the distance may simply be a two-dimensional position based on a reference point in the image. Furthermore, dynamic time warping (DTW) or the like may be applied to take into account the time shift. In this case, the minimum and maximum time widths that can be considered in DTW may be set in advance. Note that if some of the calculated skeleton positions are not related to the start or end of the task, those parts of the skeleton positions may not be used to calculate the Euclidean distance or cosine similarity.

[0031] If the obtained degree of similarity satisfies a reference value, the predetermined period having the degree of similarity that satisfies the reference value is extracted as a first candidate for the work start period or a second candidate for the work end period (P9). Steps P8 and P9 correspond to the extraction step and extraction means of this embodiment. At this stage, it is sufficient that each candidate is extracted independently; that is, the first candidate may occur multiple times in succession, and the second candidate may occur multiple times in succession.

[0032] In this way, by targeting the start and end of work as the extraction targets, the duration of repeated work periods and the duration of non-work periods, such as breaks between work periods, can be easily identified. However, depending on the work content, the skeleton positions and their changes at the start and end of a work may not necessarily be highly unique compared to other skeleton positions and their changes that appear within the work. In this case, a work portion that is not actually the start or end of a work may be mistakenly identified as the start or end. Therefore, in the work analysis method of this embodiment, the extracted first and second candidates are appropriately combined, and inappropriate first and second candidates are excluded.

[0033] A third candidate for the work period is generated by combining the obtained first candidate for the work start period with a subsequent second candidate for the work end period to the extent possible (P10). For example, the first candidate for the work start period detected first in the captured images of 100 work tasks may be paired with all of the second candidates for the work end period detected after the first candidate, up to the last 100 candidates. Note that overlaps in the number of tasks and work periods may not be considered at this stage. An upper limit may also be set for the interval between the combined work start and end times. An evaluation value indicating the degree of validity is calculated for each generated pair of work periods. The evaluation value may be calculated by combining the degree of similarity between the combined first candidate and second candidate, as well as the degree of deviation between the period from the start to the end of the work, i.e., the work time, and a reference time (standard time range). The degree of deviation between the work time and the reference time may be calculated as a function of the deviation. The function may simply calculate an absolute value, or may be a higher-order function of quadratic or higher order, such as calculating the square of the deviation. While even skilled workers may have some variation in speed, workers who are somewhat accustomed to the task are unlikely to experience significant differences in work time. Therefore, the function may be such that the degree of deviation does not change much when the deviation is small, but changes significantly and diverges when the deviation becomes larger than a certain level.

[0034] The combination of the above three components in calculating the evaluation value may be, for example, a simple weighted average. Alternatively, the degree of similarity of the first candidate and the degree of similarity of the second candidate may be weighted and added using a higher-order function such as an exponential function, and then further weighted and added with a function of the deviation. Alternatively, conversely, the evaluation of the degree of similarity of each candidate may be given relatively less weight than the function of the deviation. The evaluation value may be higher the larger, or may be higher the smaller. In the following explanation, it is assumed that the smaller the evaluation value, the higher the evaluation, and the larger the evaluation value, the lower the evaluation, i.e., the lower the degree of similarity or the larger the deviation.

[0035] Based on the thus obtained validity evaluation value, a set of operation times is selected such that their periods are exclusive (P11). The steps of P10 and P11 correspond to the specific steps and specific means of this embodiment. For example, the start and end of the operations may be determined such that the maximum value among the evaluation values of each selected set is minimized. More simply, a greedy algorithm may be applied to the evaluation value to select a set of operation times. That is, when the number of operations is known, each generated set may be selected in ascending order of the evaluation value until the same number as the number of operation times is selected in order within a range where the periods do not overlap.

[0036] Alternatively, the selection of the set may be performed by dynamic programming. In dynamic programming, when selecting in order from the second candidates included in the third candidate along the elapsed time so as to obtain a set of known operation times, a path with the best evaluation value is searched. In this case, the evaluation value may be directly treated as a cost. That is, the path with the minimum total evaluation value of the selected sets is the best path. Let the evaluation value (cost) of the combination of the h-th first candidate (h < i) combinable with the i-th second candidate at the elapsed time be c(h, i).

[0037] FIG. 4 is a diagram for explaining dynamic programming. In the dynamic programming of this embodiment, for the first candidate S and the second candidate E included in the third candidate, a three-dimensional matrix represented by the time sequence i, the number of selections j, and the selection presence / absence k is used. (i, j) represents that a set of the second candidate E with the number of selections j and the corresponding first candidate S has been selected up to the time sequence i. k = 0 represents that the candidate at the time sequence i is not selected as the j-th end timing. k = 1 represents that the candidate at the time sequence i is selected as the j-th end timing. That is, (i, j, k) means that j (sets) are selected from among those up to the time sequence i, and depending on k, the candidate at the time sequence i is selected or not selected as the j-th set. The time sequence i is 0 ≦ i < I based on the total number I of the first candidate and the second candidate included in the generated third candidate.

[0038] In dynamic programming, as the time sequence i increases, the path branches into two paths: one that keeps the number of selected items j constant and one that increases the number of selected items j by one, depending on whether or not a selection is made (k). Among the paths from i=j=k=0 to the state represented by (i, j, k), the minimum sum of the evaluation values ​​c(h, i) of the j selected sets is represented as the minimum cost p[i][j][k]. The minimum-cost path to (i, j) passes through either the minimum-cost paths to (i-1, j) or (i-1, j-1) that can reach (i, j). If the number of selected items j does not increase from time sequence (i-1) to time sequence i, p[i] is k=0, and the cost of the movement is zero. For time sequence (i-1), k can be either 0 or 1. The minimum-cost path to time sequence i must also be the minimum-cost path to time sequence (i-1). Therefore, p[i][j][0] = min(p[i-1][j][0], p[i-1][j][1]). If the i-th array element is included in the first candidate S, then (i, j, k) always has k = 0. There may be cases where the time sequence i is included in the second candidate E but is not selected.

[0039] When a time order i is selected during a move that increases the time order i by one, and the number of selections j increases by one, the cost c(h, i) corresponding to the first candidate corresponding to the selected second candidate is generated. If the i-th candidate is included in the second candidates E, it may be selected and (i, j, k) may become k=1. When k=1, p[i][j][1]=min(p[h][j-1][0]+c(h, i)) is satisfied. As mentioned above, p[h][j-1][0] is the minimum cost value among the paths that select (j-1) candidates by the time order (h-1) and reach the time order h. Therefore, when there are multiple time orders h of the first candidate S that can be selected corresponding to the i-th second candidate E, the minimum value among the sum of their minimum costs p[h] and the costs c(h, i) combined with the minimum costs p[h] may be selected as the minimum cost p[i][j][1].

[0040] The number of selections j satisfies max(0, ie - I - J + 1) ≤ j ≤ min(ie, J) based on the number of operations J. The second candidate order ie represents the chronological order within the second candidates. max(0, ie - I - J + 1) represents the larger value between 0 and "ie - I - J + 1". min(ie, J) represents the smaller value between ie and J. That is, (i, j) (0 ≤ i < I, 0 ≤ j ≤ J) may include elements that cannot be included in the path. By setting a large value such as infinity for the minimum cost other than p[0][0][0] which is the initial position in advance, impossible paths are not selected. The chronological order i and the number of selections j do not change in the decreasing direction.

[0041] Furthermore, since the (j - 1)-th selection is limited to being made before the element at the h-th position in chronological order, the overlap of the working periods is excluded. Therefore, the selectable paths are limited to the paths that can finally make J selections determined in advance while excluding duplicates. In other words, for the second candidate E at the i-th chronological order, which is the target for determining the selection presence k, there is a path directly connecting from the first candidate S at the h-th chronological order. For example, in FIG. 4, for the second candidate E with i = 2, it can be connected from either of the first candidates S with i = 0 and 1. Also, even within the above range of the number of selections j, there may be no path that continuously selects the second candidate E multiple times. For example, in FIG. 4, the second candidate E with i = I - 4 and the second candidate E with i = I - 3 cannot be selected simultaneously. By obtaining the value of the minimum cost in order from i = 0, the path of the minimum cost to i = I - 1, j = J, that is, the J selected pairs are obtained.

[0042] The start timing and end timing of each operation, as well as the working period and non-working period between them, are specified (P12) by the second candidates of the number of operations J selected as described above from the extracted I ones and the corresponding first candidates.

[0043] FIG. 5 shows examples of extracted candidates for task start and task end, and examples of the timing of identified task start and task end. As shown in the upper part (A), in the extraction stage, task start and task end may be detected consecutively, and the intervals between extracted candidates may also be non-uniform. From these, the number of tasks, i.e., the number of pairs to be selected, is fixed, and selection is performed to align task times. As a result, as shown in the lower part (B), task start, indicated by a solid line, and task end, indicated by a dashed line, alternate. Note that in some parts, task end and task start occur almost simultaneously, and the two lines overlap.

[0044] If the number of tasks is unknown, the number of tasks is assumed while changing, and the same process as above is performed for each assumed number of tasks. This provisionally identifies the same number of start and end combinations as the provisional number of tasks. If the assumed number of tasks is greater than the actual number of tasks, incorrect pairs will inevitably be mixed in. In this case, the evaluation value of the incorrect pair is likely to be significantly larger than the evaluation value of the correct pair. Therefore, if the provisional number of tasks is associated with the maximum value among the evaluation values ​​of the identified pairs of provisional number of tasks and the trend of change is tracked, the maximum value is likely to be significantly larger if the evaluation values ​​of incorrect pairs are mixed in. In other words, if the provisional number of tasks exceeds the actual number of tasks, it is expected that the maximum (worst) evaluation value will change significantly as the provisional number of tasks increases. The provisional number of tasks at which the trend of change in the worst evaluation value changes is identified as the actual number of tasks.

[0045] FIG. 6 is a graph showing the maximum evaluation value versus the number of provisional tasks. As the number of provisional work operations increases, the maximum value of the evaluation value gradually increases up to 52 operations. When the number of provisional work operations exceeds 53 operations, the maximum value of the evaluation value increases sharply. Therefore, the number of work operations is identified as 52 operations. That is, the slope (difference) of the maximum value of the evaluation value relative to the provisional work operation number is calculated, and the provisional work operation number at which this slope changes significantly can be identified as the number of work operations. Alternatively, the difference (second derivative) of the differences of the maximum values ​​of the evaluation values ​​can be calculated, and the number of work operations can be identified from the maximum point.

[0046] FIG. 7 is a flowchart showing the control procedure of the task analysis control process executed by the information processing device 1 of this embodiment. This task analysis control process is a process when the number of tasks is known. For example, it may be started when a person who uses or provides the analysis data to a user performs a predetermined input operation on the operation reception unit 15. The input operation or setting data may specify a captured image to be analyzed. Alternatively, a captured image placed in a specific folder may be automatically targeted for analysis.

[0047] The control unit 11 acquires a photographed image of the work (S1). The control unit 11 may acquire a file at a specified location. The photographed image may be acquired from a portable recording medium or an external device connected to the communication unit 16.

[0048] The control unit 11 acquires the number of tasks in the captured image (S2). The number of tasks may be acquired separately from the captured image, for example, based on the number of output products in the work process.

[0049] The control unit 11 identifies the skeletal position of the subject in each frame data of the acquired photographed image (S3). The skeletal position of the identified subject may be only the upper body, as described above.

[0050] The control unit 11 acquires reference data indicating the skeletal position at the start and end of a task at predetermined times, and a reference time for each task (S4). The control unit 11 compares each frame of the captured image data, starting from the first frame, with each frame of the reference data at the start of the task to calculate the degree of similarity (S5). The control unit 11 extracts portions of the captured image whose degree of similarity over the entire predetermined time is equal to or exceeds the reference value as task start candidates (S6).

[0051] The control unit 11 compares the captured image data with each frame of the reference data at the end, starting from the first frame, and calculates the degree of similarity (S7). The control unit 11 extracts a portion of the captured image for a predetermined time period whose overall degree of similarity is equal to or exceeds the standard as a task end candidate (S8).

[0052] The control unit 11 groups a task start candidate and a subsequent task end candidate, and calculates an evaluation value for each group (S9). When task sets with non-overlapping periods are selected for the task counts, the control unit 11 identifies the selection that results in the smallest total evaluation value for the selected task sets (S10). This allows the control unit 11 to identify task start and task end sets for the task counts. Based on this, the control unit 11 may identify the task time for each task and the interval between tasks. The control unit 11 then terminates the task analysis control process.

[0053] Figure 8 is a flowchart showing another example of the work analysis control process. This work analysis control process is a process when the number of tasks is unknown. In this work analysis control process, steps S2 and S10 in the work analysis control process shown in Figure 3 above are replaced with steps S2a and S10a, respectively, and steps S11 to S14 are added. The other processes are the same, and the same process contents are assigned the same reference numerals and detailed explanations will be omitted.

[0054] When the process proceeds to step S2a, the control unit 11 sets an initial value for the tentative number of tasks (S2a). As described above, the number of tasks is always 2 or more, so the initial value is also an appropriate value equal to or greater than 2. Thereafter, the process of the control unit 11 proceeds to step S3.

[0055] After the process of step S9, the control unit 11 selects the same number of pairs as the tentative number of work operations (S10a). At this time, the control unit 11 may select the pairs so that the sum of the evaluation values ​​of the selected work pairs is the smallest. The control unit 11 stores the tentative number of work operations and the maximum evaluation value in association with each other (S11).

[0056] The control unit 11 determines whether the tentative task count is the upper limit (S12). If it is determined that the tentative task count is not the upper limit (S12; N), the control unit 11 adds 1 to the tentative task count (S13). Then, the processing of the control unit 11 returns to step S10a.

[0057] If it is determined that the provisional task count is the upper limit (S12; Y), the control unit 11 identifies the task count based on the trend of change in the maximum evaluation value relative to the provisional task count. As described above, the control unit 11 identifies the provisional task count immediately before the maximum evaluation value begins to rise sharply based on the difference value of the maximum evaluation value or the difference value between the difference values, and identifies this as the actual task count. The control unit 11 identifies the start and end timings of each task according to the group selected corresponding to the identified task count (S14). The control unit 11 then terminates the task analysis control process.

[0058] As described above, the task analysis method of this embodiment includes the following steps: (1) a setting step of setting in advance a captured image of a worker's actions at the start of a predetermined task as a reference start image, and setting in advance a captured image of the worker's actions at the end of the task as a reference end image; (2) an acquisition step of acquiring captured data of a predetermined task that is repeated multiple times; (3) an extraction step of extracting, from the acquired captured data, a first candidate for the action at the start based on a first similarity with the reference start image, and extracting a second candidate for the action at the end based on a second similarity with the reference end image; and (4) an identification step of evaluating the validity of a third candidate for the task period defined by a combination of the extracted first and second candidates, and identifying the task period. In this way, when identifying a task period, by identifying its start and end actions, the task period can be identified reliably. This also clearly separates the task period from the non-task period. On the other hand, even if the start and end actions cannot be completely identified depending on their characteristics, misidentification can be eliminated by evaluating the combination. Therefore, this task analysis method can identify the task time of repetitive tasks with greater accuracy.

[0059] Furthermore, if the number of times a predetermined task has been repeated in the photographic data is known, the task period for that number of tasks may be identified in the identification step. If the number of tasks is known in advance, it is sufficient to identify the task period for that number of tasks, thereby reducing the possibility of overlooking a task period or leaving a misidentified task period.

[0060] In addition, the number of times a specific task is repeated in the captured data may be unknown. In this case, in the identification step, the task period for each task may be tentatively identified while changing the number of tasks, and the worst validity value among the tentatively identified task periods may be associated with the number of tasks. The number of tasks may be identified based on the trend in changes in the validity value relative to the number of tasks, with reference to the correspondence thus obtained. The task period provisionally identified for the identified number of tasks is identified as the final task period. In this way, the task analysis method of this embodiment can identify the number of tasks even if the number of tasks is unknown, thereby enabling the task period to be identified with high accuracy.

[0061] The validity may be evaluated based on a value indicating the first similarity, a value indicating the second similarity, and a value indicating the degree of deviation between the duration of the third candidate and the standard duration. By appropriately combining the similarity between the actions in the captured images and the actions in the reference images and the validity of the duration of the work, it is possible to exclude inappropriate combinations of the first and second candidates and determine the work period with greater accuracy.

[0062] In addition, in the setting step, a predetermined skeletal position of the worker performing the predetermined task may be identified in each of the reference start image and the reference end image. In the extraction step, a predetermined skeletal position of the worker may be identified in the captured images at each timing in the imaging data. The first similarity and the second similarity may be calculated using the predetermined skeletal position. By comparing only the movements of parts characteristic of the worker's actions rather than comparing the images themselves, the first and second candidates can be extracted more easily and accurately.

[0063] The first similarity and the second similarity may be calculated based on feature vectors that characterize the worker at each timing in the reference start image and the reference end image, respectively, and feature vectors that characterize the worker in the images captured at each timing in the imaging data. In other words, the validity of the first and second candidates can be objectively evaluated, making it possible to more accurately identify the work period.

[0064] The first and second similarities may be calculated by the sum of Euclidean distances or the sum of cosine similarities of the differences between the feature vectors to be compared, or by dynamic time warping. In this way, the task analysis method disclosed herein can quantitatively evaluate similarities using simple vector calculations, thereby reducing the possibility of misidentifying task durations.

[0065] The deviation may also be calculated by the absolute value of the difference between the time width of the third candidate and the standard time width, or by the square of the difference. Among the many first and second candidates, there are usually correct combinations. Therefore, even if the deviation is expressed by a simple function like this, it is possible to easily exclude inappropriate combinations of the first and second candidates based on the relative magnitude of the deviation.

[0066] In addition, in the identification step, the task duration with the best validity may be identified using dynamic programming. This task analysis method can efficiently obtain the optimal solution because the best task duration is obtained by combining the optimal third candidates.

[0067] The information processing device 1, which is an information processing system according to this embodiment, also includes a control unit 11. The control unit 11 pre-sets a captured image of a worker's movements at the start of a predetermined task as a reference start image and pre-sets a captured image of the worker's movements at the end of the task as a reference end image. The control unit 11 acquires captured data of a predetermined task that is repeated multiple times. From the acquired captured data, the control unit 11 extracts a first candidate for the movement at the start based on a first similarity to the reference start image and a second candidate for the movement at the end based on a second similarity to the reference end image. The control unit 11 evaluates the validity of a third candidate for the work period, which is defined by a combination of the extracted first and second candidates, to identify the work period. According to this information processing device 1, by identifying the start and end movements of a work period, the work period can be reliably identified. This also clearly separates work periods from non-work periods. On the other hand, even if the start and end movements cannot be completely identified due to their characteristics, misidentification can be eliminated by evaluating the combination. Therefore, according to this information processing device 1, it is possible to more accurately identify the task time of a repetitive task.

[0068] Furthermore, by installing and executing the program 131 relating to the above-described task analysis method on a computer, the task period of a repetitive task can be easily and accurately identified without requiring special hardware or the like.

[0069] The present invention is not limited to the above-described embodiment, and various modifications are possible. For example, in the above, the first candidate and the second candidate are detected completely independently, but this is not limited to this. For example, if the range of the first candidate and the range of the second candidate are detected to overlap, only the one with the highest degree of similarity may be selected.

[0070] Furthermore, when the number of operations is unknown, the number of operations may be identified based on the number of times the presence or absence of the assembly target parts shown in the captured image changes.

[0071] Furthermore, when using the worst evaluation value according to the provisional number of operations to identify the number of operations, it is not necessary to identify the worst evaluation value and operation period for all provisional number of operations. For example, it is possible to identify the worst evaluation value and operation period at appropriate intervals, and identify the worst evaluation value and operation period for all provisional number of operations only near the provisional number of operations where the trend of change in the worst evaluation value is estimated to change.

[0072] In the above example, the skeletal position is used to extract the characteristics of the movement of the subject, who is a worker, but this is not limiting. For example, the outline of the worker performing a specific movement may be extracted and the similarity thereof may be determined.

[0073] In the above example, the captured images of the worker whose work is to be analyzed are used as the reference image group, but this is not the only option. For example, in cases where individual characteristics are difficult to discern in the movements of a worker during work or where non-individual characteristics are prominent, the reference image group may be used to analyze the work of multiple workers.

[0074] In the above example, the standard time width and the third candidate time width are simply compared, but this is not limiting. In particular, when the calculation of the similarity degree includes fluctuations in the time axis direction, such as in the case of DTW, the degree of deviation may be adjusted depending on the corresponding situation.

[0075] In the above, the task analysis is performed by a single information processing device 1, but this is not limiting. The task analysis may be performed by a plurality of information processing devices in a distributed manner. The information processing device 1 may also be part of a task management system.

[0076] In the above explanation, the storage unit 13, which is composed of a nonvolatile memory such as an HDD or flash memory, has been used as an example of a computer-readable medium for storing the program 131 related to the work analysis control of the present invention, but this is not limited to this. Other computer-readable media that can be used include other nonvolatile memories such as MRAM, and portable recording media such as CD-ROMs and DVD discs. Furthermore, a carrier wave can also be used as a medium for providing the program data related to the present invention via a communication line. In addition, the specific configurations, contents and procedures of the processing operations, etc. shown in the above embodiments can be modified as appropriate without departing from the spirit of the present invention. The scope of the present invention includes the scope of the invention described in the claims and its equivalents. [Explanation of symbols]

[0077] 1. Information processing equipment 9. Imaging equipment 11 Control section 12 RAM 13 Storage section 131 Programs 1311 Skeleton Recognition Processing 132 Work Information 14 Display section 15 Operation reception section 16 Communications Department

Claims

1. a setting step of setting in advance a photographed image of the worker's movement at the start of a predetermined task as a reference start image, and setting in advance an image of the worker's movement at the end of the task as a reference end image; an acquisition step of acquiring photographic data of the predetermined work repeated a plurality of times; an extraction step of extracting a first candidate for the action at the start from the acquired photographic data based on a first similarity with the reference start image, and extracting a second candidate for the action at the end based on a second similarity with the reference end image; a specifying step of evaluating the validity of a third candidate for the work period defined by a combination of the extracted first candidate and the extracted second candidate, and specifying the work period; Work analysis methods, including:

2. The number of times the predetermined task has been repeated in the photographed data is known, In the specifying step, the work period of the number of work operations is specified. The work analysis method according to claim 1.

3. The number of times the predetermined task has been repeated in the photographed data is unknown; In the identifying step, tentatively specifying the work period for each of the work counts while changing the work counts, and associating the worst validity value among the tentatively specified work periods with the work counts; Identifying the number of times of work based on a change trend of the validity value with respect to the number of times of work; Identifying the provisionally specified work period based on the specified number of work operations. The work analysis method according to claim 1.

4. 2. The work analysis method according to claim 1, wherein the evaluation of the validity is performed using an evaluation value based on a value indicating the first similarity, a value indicating the second similarity, and a value indicating a degree of deviation between the time width of the third candidate and a standard time width.

5. In the setting step, a predetermined skeletal position of the worker is identified in each of the reference start image and the reference end image; In the extraction step, the predetermined skeletal position of the worker is identified in the captured image at each timing in the imaging data; the first similarity and the second similarity are each determined using the predetermined skeleton position; The work analysis method according to claim 4.

6. The work analysis method of claim 4, wherein the first similarity and the second similarity are calculated based on feature vectors that characterize the worker at each timing in the reference start image and the reference end image, respectively, and feature vectors that characterize the worker in the images captured at each timing in the shooting data.

7. 7. The task analysis method according to claim 6, wherein the first similarity and the second similarity are calculated by either a sum of Euclidean distances or a sum of cosine similarities of differences between the feature vectors to be compared, or by a dynamic time warping method.

8. 5. The task analysis method according to claim 4, wherein the degree of deviation is calculated by taking the absolute value of a difference between the third candidate duration and the standard duration, or by taking the square of the difference.

9. The task analysis method according to claim 1 , wherein the task period that maximizes the validity is identified in the identifying step by using dynamic programming.

10. A photographed image of the worker's movement at the start of a predetermined task is set in advance as a reference start image, and a photographed image of the worker's movement at the end of the task is set in advance as a reference end image, Acquire photographic data of the predetermined work repeated multiple times; extracting a first candidate for the action at the start based on a first similarity with the reference start image from the acquired photographic data, and extracting a second candidate for the action at the end based on a second similarity with the reference end image; identifying a work period by evaluating the validity of a third candidate for the work period defined by a combination of the extracted first candidate and the extracted second candidate; An information processing system comprising a control unit.

11. Computer a setting means for setting in advance a photographed image of a worker's movement at the start of a predetermined task as a reference start image, and setting in advance an image of a worker's movement at the end of the task as a reference end image; an acquisition means for acquiring photographic data of the predetermined work repeated multiple times; an extraction means for extracting a first candidate for the action at the start from the acquired photographic data based on a first similarity with the reference start image, and extracting a second candidate for the action at the end based on a second similarity with the reference end image; an identification means for evaluating the validity of a third candidate for the work period defined by a combination of the extracted first candidate and the extracted second candidate, and identifying the work period; A program that functions as a

Images