Labeling support method, labeling support apparatus, and labeling support program

Abstract

To reduce a burden of labeling by a user.SOLUTION: A labeling support device includes a reference cycle label input part 1 in which a user manually performs labeling to a reference cycle S, and a processing object cycle label input part 2 in which the user manually performs labeling with accuracy lower than the reference cycle S, and extracts a feature of a label boundary of the reference cycle S. The label correction device includes an image feature extraction unit 3 that extracts a feature near the label in the processing target cycle T, a similarity determination unit 4 that determines similarity between the feature of the label boundary and the feature near the label, and a label correction unit 5 that corrects the label boundary attached by the user in the processing target cycle T by using an image frame having a feature most similar to the feature of the label boundary in the reference cycle S among a plurality of image frames having the feature near the label in the reference cycle S.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a labeling support method, a labeling support device, and a labeling support program that support a user in labeling time-series image data that is obtained by repeating a cycle of a plurality of unit tasks performed in series by a worker multiple times, the cycle indicating the boundaries of each unit task. [Background technology]

[0002] In the labeling support method described in Patent Document 1, the similarity between waveform data of an arbitrary section in time-series data to be labeled and waveform data of training data input by the user is calculated, and the user then determines a threshold for the similarity. Then, a computer labels sections of the time-series data where the similarity is equal to or greater than the threshold as anomalous data, and the labeled data is provided to the user. Furthermore, in addition to the labeled data, statistics that appear to be highly correlated with the anomalous data are provided to the user as auxiliary data. The user then determines whether the labeling of the time-series data is correct by referring to the labeled data and statistics. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-122487 Summary of the Invention [Problem to be solved by the invention]

[0004] In the labeling support method of Patent Document 1, the user determines the threshold value of the similarity that serves as the criterion for labeling and judges whether the labeling is correct or incorrect, which tends to increase the burden on the user.

[0005] Furthermore, when labeling time-series image data that captures multiple unit tasks performed in succession by workers on a factory line, for example, it becomes more difficult for a user to adjust the similarity threshold. Furthermore, when labeling such time-series image data, there is a risk that the labeling performed manually by a user to create training data will vary greatly.

[0006] The present invention was devised in view of the current situation, and one of its objects is to provide a labeling support method, a labeling support device, and a labeling support program that can reduce the burden of labeling on users. [Means for solving the problem]

[0007] The present invention relates to a labeling support method, in which each label of a reference cycle labeled with a predetermined accuracy is prepared, each label of a target cycle labeled by a user with lower accuracy than the reference cycle is prepared, features of an image frame that defines the label boundary between two unit operations that are adjacent in the chronological order of the reference cycle are extracted as features of the label boundary, features of the image frame that defines the label boundary of the target cycle corresponding to the two unit operations of the reference cycle and features of a plurality of image frames before and after this image frame are extracted as features in the label vicinity, the similarity between the features of the label boundary of the reference cycle and the features in the label vicinity of the target cycle is determined, and the label boundary assigned by the user in the target cycle is corrected using the image frame that has features most similar to the features of the label boundary of the reference cycle from among the plurality of image frames having features in the label vicinity of the target cycle. [Effects of the Invention]

[0008] All the user has to do is perform strict labeling on the reference cycle and rough labeling on the target cycle, and the computer will automatically correct the user's rough labeling using the image frame that has features most similar to the features of the label boundary of the reference cycle from among multiple image frames that have features near the label of the target cycle.

[0009] According to the present invention, the burden of labeling on the user can be reduced. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a system block diagram of a labeling assistance device according to a first embodiment. [Figure 2] FIG. 1 is an explanatory diagram illustrating a labeling assistance method using the labeling assistance device of the first embodiment. [Figure 3] 1 is a flowchart showing the flow of a labeling assistance method according to a first embodiment. [Figure 4] FIG. 10 is a system block diagram of a labeling assistance device according to a second embodiment. [Figure 5] 10 is a flowchart showing the flow of a labeling assistance method according to a second embodiment. [Figure 6] FIG. 10 is a system block diagram of a labeling assistance device according to a third embodiment. [Figure 7] 10 is a flowchart showing the flow of a labeling assistance method according to a third embodiment. [Figure 8] FIG. 10 is a system block diagram of a labeling assistance device according to a fourth embodiment. [Figure 9] The following shows an example of searching for a maximum value of similarity, where (a) shows a case where there is a maximum value of similarity, (b) shows an example where there is no maximum value of similarity, and (c) shows another example where there is no maximum value of similarity. [Figure 10] 10 is a flowchart showing the flow of a labeling assistance method according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of a labeling assisting device according to the present invention will be described with reference to the accompanying drawings.

[0012] The labeling assistance device is a device that assists a user in labeling time-series image data, which is obtained by repeating a cycle of a plurality of unit tasks performed in series by a worker, to indicate the boundaries of each unit task. The labeling assistance device is used, for example, to compare the work of a beginner and an expert for a plurality of unit tasks performed in series or for individual unit tasks in order to help a beginner master the task in a workplace such as a factory. The labeling assistance device may also be used, for example, to monitor whether each worker's work is being performed appropriately within a predetermined time, thereby achieving efficient operation of a workplace such as a factory.

[0013] As shown in FIG. 1, the labeling support device is configured by a so-called computer and includes a reference cycle label input unit 1, a processing target cycle label input unit 2, an image feature extraction unit 3, a similarity determination unit 4, and a label correction unit 5.

[0014] The reference cycle label input unit 1 is an input unit that allows a user to manually label any one cycle of the time-series image data as a reference cycle S. Here, the time-series image data is image data of repetitive work, such as factory line work. In this embodiment, the time-series image data is data of a cycle shown in FIG. 2 , which is composed of four unit tasks: a screw-removing task W1, a screw-fastening task W2, a walking task W3, and a screw-fastening task W4, repeated multiple times in a specific section of the factory line. In this embodiment, the series of tasks W1, W2, W3, and W4 takes approximately three minutes. Note that in this embodiment, the reference cycle S is composed of four unit tasks as an example, but the number of unit tasks is not limited to four and may be any number other than four. Furthermore, the image data representing each of tasks W1, W2, W3, and W4 includes multiple image frames arranged in chronological order.

[0015] Labeling is the process of assigning correct labels to time-series image data in order to train an AI model that solves tasks such as classifying tasks. The labeling in the reference cycle S serves as the basis for subsequent cycles T to be processed, so the user performs labeling on an image frame-by-image frame basis with as high accuracy as possible (a predetermined high accuracy). In this embodiment, labeling is performed on an image frame-by-image frame basis with high accuracy in 0.1-second increments. Furthermore, in this embodiment, labeling is performed such that 0 is assigned to the task W1 of unscrewing, 1 is assigned to the task W2 of loosening screws, 2 is assigned to the task W3 of walking, and 3 is assigned to the task W4 of fully tightening screws. With this labeling, the first label of 1 when the label switches from 0 to 1 becomes the first label boundary B1, the first label of 2 when the label switches from 1 to 2 becomes the second label boundary B2, the first label of 3 when the label switches from 2 to 3 becomes the third label boundary B3, and the label next to the last label of 3 becomes the fourth label boundary B4. By labeling in this manner, as shown in "(1) Labeling of the Reference Cycle" in Figure 2, the first label boundary B1 is defined between operations W1 and W2, the second label boundary B2 is defined between operations W2 and W3, the third label boundary B3 is defined between operations W3 and W4, and the fourth label boundary B4 is defined as the end point of operation W4. Note that labeling may also be performed to define a label boundary at the start point of operation W1.

[0016] Furthermore, since the order of a series of unit tasks performed by a worker is predetermined, labeling may be performed only on the first to fourth label boundaries B1, B2, B3, and B4, rather than labeling on an image frame basis.

[0017] The processing target cycle label input unit 2 is an input unit that allows the user to manually label a processing target cycle T that is located chronologically later than the reference cycle S in the time-series image data. The processing target cycle T includes a first processing target cycle T1, a second processing target cycle (not shown), a third processing target cycle (not shown), ..., and an Nth processing target cycle (not shown), arranged in chronological order. Labeling of the processing target cycle T is performed with lower precision than labeling of the reference cycle S. In other words, labeling of the processing target cycle T does not need to be performed accurately on an image frame-by-frame basis, but is performed roughly in one-second increments. Furthermore, in this embodiment, because repetitive work such as factory line work is performed, the same type of label as the label input by the user in the reference cycle label input unit 1 is assigned. As a result of this labeling, as shown in "(2) Labeling of process target cycles" in FIG. 2, for the first process target cycle T1, a fifth label boundary B5 is defined between tasks W1 and W2, a sixth label boundary B6 is defined between tasks W2 and W3, a seventh label boundary B7 is defined between tasks W3 and W4, and an eighth label boundary B8 is defined as the end point of task W4. In this way, the fifth to eighth label boundaries B5, B6, B7, and B8 are roughly labeled less precisely than the first to fourth label boundaries B1, B2, B3, and B4 of the reference cycle S. Then, labeling similar to that for the first process target cycle T1 is performed for the second process target cycle and thereafter (not shown).

[0018] The labeling in the process target cycle T may be performed by roughly labeling each cycle and then adjusting the labeling by the computer. Alternatively, the labeling in the process target cycle T may be performed by roughly labeling multiple cycles collectively and then adjusting the labeling by the computer.

[0019] In this embodiment, the first cycle to be processed T1 is arranged as the cycle immediately following the reference cycle S in the time-series image data, but the first cycle to be processed T1 does not necessarily have to be immediately following the reference cycle S. In addition, the cycle to be processed T may be arranged in time-series video data other than the time-series video data including the reference cycle S.

[0020] Labeling in the reference cycle label input unit 1 and the target cycle label input unit 2 is not performed in real time, but rather occurs some time after the time-series image data to be labeled is acquired. Furthermore, the labels input into the reference cycle label input unit 1 and the target cycle label input unit 2 are stored in a storage unit (not shown) which is a database built into the computer.

[0021] The image feature extraction unit 3 extracts, as label boundary features, image frame features that define the label boundary between two unit tasks adjacent in the time series direction of the reference cycle S. In this embodiment, the image feature extraction unit 3 extracts the image frame features between tasks W1 and W2 as the features of the first label boundary B1, the image frame features between tasks W2 and W3 as the features of the second label boundary B2, the image frame features between tasks W3 and W4 as the features of the third label boundary B3, and the image frame features corresponding to the end point of task W4 as the features of the fourth label boundary B4. Furthermore, when extracting the features of the first to fourth label boundaries B1, B2, B3, and B4, appropriate features of the first to fourth label boundaries B1, B2, B3, and B4 are extracted based on edge information detected (extracted) using well-known edge detection techniques, such as a Sobel filter, Laplacian filter, or Canny filter, as well as features of a dataset, such as basic information about the image, such as average brightness and saturation.

[0022] Furthermore, the image feature extraction unit 3 extracts, as features near the label, the features of the image frame that defines the label boundary of the processing target cycle T corresponding to the two unit tasks of the reference cycle S, and the features of multiple image frames in the vicinity before and after this image frame. The number of multiple image frames in the vicinity before and after the image frame is predetermined, and for example, in a task where labeling is expected to be off by about 2 seconds, if the video is 30 fps (frames per second), there will be a total of 60 image frames, consisting of 30 image frames before and after the image frame that defines the boundary. Furthermore, in this embodiment, as shown in the upper part of “(3) Determination of similarity between features of label boundaries and features in the vicinity of labels” in FIG. 2, the image feature extraction unit 3 extracts features in the vicinity of the image frame that defines the fifth label boundary B5 as features of the first label vicinity C1, features in the vicinity of the image frame that defines the sixth label boundary B6 as features of the second label vicinity C2, features in the vicinity of the image frame that defines the seventh label boundary B7 as features of the third label vicinity C3, and features in the vicinity of the image frame that defines the eighth label boundary B8 as features of the fourth label vicinity C4.

[0023] The similarity determination unit 4 determines the similarity between the features of the label boundaries of the reference cycle S and the features in the label neighborhood of the process target cycle T. In this embodiment, the similarity determination unit 4 determines the similarity, more specifically the degree of similarity, between the features of the first label boundary B1, the second label boundary B2, the third label boundary B3, and the fourth label boundary B4 of the reference cycle S and the features of the first label neighborhood C1, the second label neighborhood C2, the third label neighborhood C3, and the fourth label neighborhood C4 of the first process target cycle T1.

[0024] Here, as an example, a description will be given of determining the similarity between features of the first label boundary B1 of the reference cycle S and features of the first label neighborhood C1 of the first process cycle T1. For example, when determining the similarity based on image edge information, the degree of match between the edge images of the features of the first label boundary B1 and the features of the first label neighborhood C1, that is, the greater the number of matching pixels between the two, the higher the similarity is determined. Furthermore, when determining the similarity based on the difference in average brightness, the smaller the difference between the average brightness values ​​of the features of the first label boundary B1 and the features of the first label neighborhood C1, the higher the similarity is determined.

[0025] After determining the similarity, the similarity determination unit 4 selects an image frame having a feature most similar to the feature of the label boundary of the reference cycle S from among the multiple image frames having features near the label of the target cycle T. For example, as shown in the third frame from the left in a frame G at the bottom of FIG. 2 , the similarity determination unit 4 selects a first image frame F1 from among the multiple image frames having features near the first label C1 of the first target cycle T1, which has a feature most similar to the feature of the first label boundary B1 of the reference cycle S. Similarly, the similarity determination unit 4 selects a second image frame F2 from among the multiple image frames having features near the second label C2 of the first target cycle T1, which has a feature most similar to the feature of the second label boundary B2 of the reference cycle S. Similarly, the similarity determination unit 4 selects a third image frame F3 from among the multiple image frames having features near the third label C3 of the first target cycle T1, which has a feature most similar to the feature of the third label boundary B3 of the reference cycle S. Furthermore, the similarity determination unit 4 selects a fourth image frame F4 from among a plurality of image frames having features of the fourth label vicinity C4 of the first processing target cycle T1, the fourth image frame F4 having features most similar to the features of the fourth label boundary B4 of the reference cycle S.

[0026] The label correction unit 5 corrects the rough fifth to eighth label boundaries B5, B6, B7, and B8 assigned by the user in the processing target cycle T, in this embodiment the first processing target cycle T1, using the first to fourth image frames F1, F2, F3, and F4 selected by the similarity determination unit 4. As a result, the rough fifth to eighth label boundaries B5, B6, B7, and B8 are corrected to the positions of the first to fourth image frames F1, F2, F3, and F4.

[0027] Next, the flow of a labeling support method performed using the labeling support device of the first embodiment will be described with reference to Fig. 3. This labeling support method is implemented by a computer executing a labeling support program (not shown).

[0028] First, in step S1, a user labels the reference cycle S by manually inputting the label into the reference cycle label input unit 1. In this way, each label of the reference cycle S labeled with a predetermined accuracy is prepared.

[0029] Next, in step S2, the user labels the cycle to be processed T by manually inputting the label into the cycle to be processed label input unit 2. As a result, labels for the cycle to be processed T with lower accuracy than the reference cycle S are prepared.

[0030] Then, in step S3, the image feature extracting unit 3 extracts the features of the label boundaries of the reference cycle S, in this embodiment, the features of the first to fourth label boundaries B1, B2, B3, and B4.

[0031] Next, in step S4, the image feature extraction unit 3 extracts features near the labels of the cycle T to be processed, in this embodiment, features near the first to fourth labels C1, C2, C3, and C4 of the first cycle T1 to be processed. The extraction of features near the labels is also performed for the second and subsequent cycles to be processed.

[0032] Next, in step S5, the similarity determination unit 4 selects an image frame having features most similar to the features of the label boundary from among a plurality of image frames having features near the label. In this embodiment, the first to fourth image frames F1, F2, F3, and F4 are selected.

[0033] Then, in step S6, the label correction unit 5 corrects the label boundaries of the processing target cycle T using the image frame having the most similar features. In this embodiment, the first to fourth image frames F1, F2, F3, and F4 are used to correct the rough fifth to eighth label boundaries B5, B6, B7, and B8 assigned by the user in the first processing target cycle T1.

[0034] As described above, in the first embodiment, the image feature extraction unit 3 extracts features of the first to fourth label boundaries B1, B2, B3, and B4 that have been labeled with high accuracy by the user, and features of the first to fourth label neighborhoods C1, C2, C3, and C4 that are associated with the fifth to eighth label boundaries B5, B6, B7, and B8 that have been roughly labeled by the user. After determining the similarity between the features of the first to fourth label boundaries B1, B2, B3, and B4 and the first to fourth label neighborhoods C1, C2, C3, and C4, the similarity determination unit 4 selects the first to fourth image frames F1, F2, F3, and F4 that have features most similar to the features of the first to fourth label boundaries B1, B2, B3, and B4 of the reference cycle S from among the multiple image frames that have the features of the first to fourth label neighborhoods C1, C2, C3, and C4 of the first process target cycle T1. Then, the label correction unit 5 uses the first to fourth image frames F1, F2, F3, and F4 to correct the fifth to eighth label boundaries B5, B6, B7, and B8 assigned by the user in the first process cycle T1. Therefore, as long as the user performs strict labeling in the reference cycle S and rough labeling in the process cycle T, the label correction unit 5 automatically corrects the rough labeling in the process cycle T, thereby reducing the labeling burden on the user.

[0035] FIG. 4 is a system block diagram of a labeling assistance device according to the second embodiment.

[0036] The labeling assistance device of the second embodiment is configured by replacing the image feature extraction unit 3 of the labeling assistance device of the first embodiment with a skeleton estimation unit 6.

[0037] The skeleton estimation unit 6 acquires the features of the first to fourth label boundaries B1, B2, B3, and B4 by estimating the skeleton of the worker in each image frame having the features of each of the first to fourth label boundaries B1, B2, B3, and B4 of the reference cycle S. When estimating the worker's skeleton, each image frame having the features of each of the first to fourth label boundaries B1, B2, B3, and B4 is used as input, and the skeleton of the worker in each image frame is estimated using a well-known skeleton estimation technique, such as OpenPose or MMPose, and this estimated skeleton is used as the features of the first to fourth label boundaries B1, B2, B3, and B4.

[0038] Furthermore, the skeleton estimation unit 6 acquires features near the labels by estimating the worker's skeleton in multiple image frames having features near the labels of the processing target cycle T. In this embodiment, the skeleton estimation unit 6 estimates the worker's skeleton in multiple image frames having features near the first label C1, multiple image frames having features near the second label C2, multiple image frames having features near the third label C3, and multiple image frames having features near the fourth label C4, thereby acquiring features near the first to fourth labels C1, C2, C3, and C4. The worker's skeleton is estimated using the same well-known skeleton estimation technique as that used for each image frame of the first to fourth label boundaries B1, B2, B3, and B4.

[0039] The similarity determination unit 4 determines the similarity between the features of the first to fourth label boundaries B1, B2, B3, B4 based on the skeleton information estimated by the skeleton estimation unit 6 and the features of the first to fourth label vicinities C1, C2, C3, C4 based on the skeleton information estimated by the skeleton estimation unit 6. Here, the similarity means the similarity between the skeleton information related to the features of the first to fourth label boundaries B1, B2, B3, B4 and the skeleton information related to the features of the first to fourth label vicinities C1, C2, C3, C4. In this embodiment, this similarity is obtained by the root mean square error (RMSE). For example, for the boundary image frames of the reference cycle S (corresponding to the first to fourth label boundaries B1, B2, B3, B4), the position coordinates of the joints x = {x|0 < x < n|n is the number of joints that can be estimated in one image frame} obtained by estimating the skeleton are y 1x , and for the boundary image frames of the first processing target cycle T1 (corresponding to the fifth to eighth label boundaries B5, B6, B7, B8), the position coordinates of the joints x obtained by estimating the skeleton are y 2xt = {y 2xt |t is the frame number of the boundary frame}, and for the vicinity image frames of the boundary image frames of the first processing target cycle T1 (corresponding to the first to fourth label vicinities C1, C2, C3, C4), the position coordinates obtained by estimating the skeleton are y 2xi = {y 2xt |i = ±a|a is an arbitrary number of image frames regarded as the vicinity image frame}, then the RMSE in the vicinity image frame i is defined by the following formula.

[0040]

Equation

[0041] And the vicinity image frame i with the smallest value obtained by this formula is selected as the image frame with the highest similarity.

[0042] Next, while referring to FIG. 5, the flow of the labeling support method performed using the labeling support device of the second embodiment will be described.

[0043] First, in step S1, the user manually inputs a label into the reference cycle label input unit 1 to label the reference cycle S, and in step S2, the user manually inputs a label into the process target cycle label input unit 2 to label the process target cycle T.

[0044] Next, in step S7, the skeleton estimation unit 6 estimates the skeleton of the worker in the image frames of the label boundaries of the reference cycle S. More specifically, the skeleton of the worker is estimated in each image frame having the characteristics of each of the first to fourth label boundaries B1, B2, B3, and B4 of the reference cycle S.

[0045] Then, in step S8, the skeleton estimation unit 6 also performs a process to estimate the worker's skeleton in image frames near the labels of the target cycle T. More specifically, for the first target cycle T1, the skeleton is estimated in a plurality of image frames having a feature near the first label C1, a plurality of image frames having a feature near the second label C2, a plurality of image frames having a feature near the third label C3, and a plurality of image frames having a feature near the fourth label C4. This estimation of the worker's skeleton is also performed for the second target cycle and thereafter.

[0046] Next, in step S9, the image frame that is most similar to the skeleton estimated in step S7 from among the skeletons estimated in step S8 is selected.

[0047] Then, in step S6, the label correcting unit 5 corrects the label boundary of the cycle T to be processed using the image frame having the most similar features.

[0048] As described above, in the second embodiment, the skeleton estimation unit 6 estimates the worker's skeleton in each image frame having the features of the first to fourth label boundaries B1, B2, B3, and B4 of the reference cycle S, thereby acquiring the features of the first to fourth label boundaries B1, B2, B3, and B4. Furthermore, the skeleton estimation unit 6 estimates the worker's skeleton in multiple image frames having the feature of the first label neighborhood C1, multiple image frames having the feature of the second label neighborhood C2, multiple image frames having the feature of the third label neighborhood C3, and multiple image frames having the feature of the fourth label neighborhood C4, thereby acquiring the features of the first to fourth label neighborhoods C1, C2, C3, and C4. In this way, the features of the first to fourth label boundaries B1, B2, B3, and B4 and the features of the first to fourth label neighborhoods C1, C2, C3, and C4 can be obtained by estimating the worker's skeleton, thereby enabling efficient classification of the worker's tasks on an image frame-by-image basis, and more accurate labeling.

[0049] FIG. 6 is a system block diagram of a labeling assistance device according to the third embodiment.

[0050] The labeling assistance device of the third embodiment is configured by adding a label prediction unit 7 to the labeling assistance device of the first embodiment.

[0051] The label prediction unit 7 learns a classification model (AI model) that predicts label boundaries based on the features of the label boundaries of the reference cycle S, the features near the labels of the cycle to be processed T, and the label boundaries corrected by the label correction unit 5, and uses this classification model to predict the label boundaries of the next cycle to be processed. The next cycle to be processed refers to one or more cycles to be processed after the first cycle to be processed T1, but does not refer to all cycles to be processed from the first cycle to the Nth cycle to be processed as in the first embodiment. The label prediction unit 7 predicts the label boundaries each time labeling for one cycle of the next cycle to be processed is completed.

[0052] The classification model is trained using well-known learning methods, such as deep learning methods such as CNN, or machine learning methods such as SVM or Random Forest. When training the classification model, the system may be trained to label each image frame as task W1, task W2, task W3, and task W4. When training the classification model, the system may be trained to label the boundary between tasks W1 and W2, the boundary between tasks W2 and W3, and the end point of task W4. As the number of cycles in the next cycle to be processed increases, the system may be trained to label whether or not the boundary is a boundary, such as the boundary between tasks W1 and W2.

[0053] If the label boundary predicted by the label prediction unit 7 matches or is similar to the label boundary corrected by the label correction unit 5, the user's labeling for the next cycle to be processed is stopped, and labeling is automatically performed using a classification model. Here, the predicted label boundary and the corrected label boundary are considered similar if, for example, the predicted value (predicted label boundary) and the corrected value (corrected label boundary) for the label boundary between work W1 and work W2 differ by 20 image frames, and the predicted value and the corrected value for the label boundary between work W2 and work W3 differ by 50 image frames. The total image frame misalignment is 70 image frames, and the average misalignment per label boundary is 35 image frames. If the number of such image frames, 35, is smaller than a predetermined threshold, the predicted label boundary and the corrected label boundary are considered similar.

[0054] Next, with reference to FIG. 7, the flow of the labeling assistance method performed using the labeling assistance device of the third embodiment will be described.

[0055] First, in step S1, the user manually inputs a label into the reference cycle label input unit 1, thereby labeling the reference cycle S.

[0056] Next, in step S10, the user manually inputs a label into the target cycle label input unit 2 to label the next target cycle. During this labeling, for example, the first target cycle T1 and the second target cycle T2 are roughly labeled with lower precision than the reference cycle S. This rough labeling is preferably performed for one or several cycles. This is because even rough labeling can lead to significant variability in labeling if repeated by a human over a long period of time, and data containing such variability is not desirable for training the classification model.

[0057] Then, in step S3, features of the label boundary of the reference cycle S are extracted, and in step S4, features near the label of the cycle to be processed T are extracted. In step S5, an image frame having features most similar to the features of the label boundary is selected from multiple image frames having features near the label. Furthermore, in step S6, the label boundary of the cycle to be processed T is corrected using the image frame having the most similar feature.

[0058] Next, in step S11, it is determined whether the predicted label boundary matches or is similar to the corrected label boundary. If they do not match or are similar, the process proceeds to step S12, where a classification model for predicting the label boundary is trained based on the features of the label boundary of the reference cycle S, the features near the label of the target cycle T, and the label boundary corrected by the label correction unit 5.

[0059] Then, in step S13, the label boundary of the next cycle to be processed is predicted using the classification model. In step S10, the predicted label boundary is compared with the label boundary of the next cycle to be processed whose label boundary correction has been completed (corrected label boundary).

[0060] Furthermore, if it is determined in step S11 that the predicted label boundary matches or is similar to the corrected label boundary, the process proceeds to step S14, where the user is stopped from manually labeling using the processing target cycle label input unit 2, and labeling is performed automatically using a classification model.

[0061] As described above, in the third embodiment, a classification model for predicting label boundaries is trained based on the features of the label boundaries, the features near the labels, and the corrected label boundaries, and this classification model is used to predict the label boundaries for the next cycle to be processed. If the predicted label boundaries match or are similar to the corrected label boundaries, labeling by the user is stopped, and labeling is performed automatically using the classification model. This reduces the burden of labeling on the user. In other words, once the user has performed labeling up to a certain point, the computer can perform the subsequent labeling automatically.

[0062] FIG. 8 is a system block diagram of a labeling assistance device according to the fourth embodiment.

[0063] The labeling assistance device of the fourth embodiment is configured by adding a label correction control unit 8 to the labeling assistance device of the first embodiment.

[0064] If there is no image frame among the plurality of image frames having features near the label of the target cycle T that has features most similar to the features of the label boundary of the reference cycle S (in this embodiment, an image frame having a maximum value of similarity between the features of the label boundary and the features near the label), the label correction control unit 8 increases the number of the plurality of image frames and searches for an image frame having a maximum value of similarity.Then, from the increased number of image frames, the image frame having features most similar to the features of the label boundary of the reference cycle S is extracted, and this image frame is used to correct the label boundary assigned by the user in the target cycle T.

[0065] Now, with reference to Figures 9(a) to 9(c), we will explain the cases where there is and is not an image frame among multiple image frames having features of the first label vicinity C1 between tasks W1 and W2 of the first process target cycle T1 that has features most similar to the features of the first label boundary B1 of the reference cycle S, in this embodiment, having a maximum value of similarity.

[0066] First, as shown in FIG. 9(a), if we define the first image frame F1, which is the boundary between tasks W1 and W2 in the first process cycle T1, as corresponding to image frame number "0" on the horizontal axis of FIG. 9(a), then the multiple image frames having the characteristics of the first label neighborhood C1 related to the first image frame F1 are in the range of 10 image frames around image frame number "0," i.e., the image frame numbers range from "-10" to "10." In FIG. 9(a), the maximum value of similarity exists in the range of image frame numbers "-10" to "10." In other words, the maximum value of similarity (indicated by a black star) exists when the image frame number is "5."

[0067] Even if there is a maximum value of similarity, for example, if the similarity fluctuates around 0 and the maximum value is 1, the similarity is relatively low and the data reliability is low, so in such cases the maximum value is not used. Also, even if there is no maximum value of similarity, if there is an image frame with a relatively high similarity (an image frame with high similarity), the image frame with high similarity may be used instead of the maximum value.

[0068] FIG. 9(b) shows an example in which there is no maximum value of similarity in the range of image frame numbers "-10" to "10." In FIG. 9(b), the similarity increases steadily to the right in the range before image frame number "10," as indicated by the arrow X. In such a case, it is expected that the similarity will continue to increase even after image frame number "10." Therefore, the number of image frames on the side where the similarity is increasing is increased to search for the maximum value of similarity. In this embodiment, the number of image frames on the image frame number "10" side is increased by five to create a new range of image frame numbers "10" to "15," and the maximum value of similarity, indicated by the dashed star, is searched for.

[0069] FIG. 9(c) shows another example in which there is no maximum value of similarity in the range of image frame numbers "-10" to "10." In FIG. 9(c), the similarity increases gradually to the left in the range of image frame numbers just before "-10," as indicated by arrow Y. In such a case, it is expected that the similarity will continue to increase even after the image frame number "-10." Therefore, the number of image frames on the side where the similarity is increasing is increased to search for the maximum value of similarity. In this embodiment, the number of image frames on the "-10" side is increased by five to create a new range of image frame numbers from "-10" to "-15," and the maximum value of similarity, indicated by the dashed star, is searched for.

[0070] Furthermore, if the number of image frames is increased and a maximum value of similarity is obtained from a newly established range, and if such a maximum value is obtained a predetermined number of times in succession, the user is notified that the labeling tendency has changed and is given information on the labeling before and after the change. This notification can be made by any method, such as displaying a pop-up window on a display (not shown) of the labeling assistance device or by audio notification.

[0071] The labeling assistance device is also configured to receive feedback as to whether the rough labeling performed by the user in the target cycle T is significantly correct. If it is determined that the user's labeling is appropriate, the image frames defining the label boundary between two chronologically adjacent unit operations are not selected from the image frames defining the first to fourth label boundaries B1, B2, B3, and B4 of the reference cycle S, but rather the number of image frames in the target cycle T most recently labeled is increased and image frames having maximum values ​​obtained in each of newly established ranges are extracted. The average or median of the features of the extracted image frames is then taken and used as a new reference for the label boundary between two adjacent unit operations.

[0072] Next, with reference to FIG. 10, the flow of the labeling assistance method performed using the labeling assistance device of the fourth embodiment will be described.

[0073] First, in step S1, the user manually inputs a label into a reference cycle label input unit 1 to label the reference cycle S, and in step S2, the user manually inputs a label into a target cycle label input unit 2 to label the target cycle T. Then, in step S3, features of the label boundary of the reference cycle S are extracted, in step S4, features near the label of the target cycle T are extracted, and in step S5, an image frame having features most similar to the features of the label boundary is selected from multiple image frames having features near the label.

[0074] Then, in step S15, it is determined whether or not there is an image frame among the plurality of image frames having features near the label that has features most similar to the features of the label boundary. That is, it is determined whether or not there is an image frame among the plurality of image frames having features near the label that has a maximum value of similarity between the features of the label boundary and the features near the label. If there is an image frame with a maximum value of similarity, the process proceeds to step S6, where the label boundary of the processing target cycle T is corrected using the image frame having the most similar feature.

[0075] If it is determined in step S15 that there is no image frame with a maximum value of similarity, the process proceeds to step S16, where the number of image frames on the side where the similarity between the features at the label boundary and the features near the label is increasing is increased, and the image frame with the most similar features is extracted.

[0076] Next, in step S17, it is determined whether the image frame having the most similar features has been extracted from the added image frames a predetermined number of times or more. If it is determined that the image frame has not been extracted a predetermined number of times or more, the process proceeds to step S6, where the label boundary of the cycle T to be processed is corrected using the image frame having the most similar features.

[0077] If it is determined in step S17 that image frames have been extracted a predetermined number of times or more in succession, the process proceeds to step S18, where the user is notified of any change in the labeling tendency and the labeling information before and after the change.

[0078] Next, in step S19, the user determines whether the labeling is incorrect. If it is determined that the labeling is incorrect, the process proceeds to step S20, where the user corrects the labeling. The corrected labeling is fed back to the labeling assistance device.

[0079] If the user determines in step S19 that the labeling is not incorrect, the process proceeds to step S21, where the label boundary criteria are updated.

[0080] As described above, in the fourth embodiment, if there is no image frame with a maximum value of similarity between the label boundary feature and the label neighborhood feature among the multiple image frames having features near the label of the processing target cycle T, the label correction control unit 8 increases the number of the multiple image frames and searches for an image frame with a maximum value of similarity. Then, from the increased number of image frames, the image frame with the feature most similar to the label boundary feature of the reference cycle S is extracted, and this image frame is used to correct the label boundary assigned by the user in the processing target cycle T. For example, the accuracy of the rough labeling by the user may be significantly reduced or the labeling may be off by several seconds to several tens of seconds due to reasons such as a change in the labeling user or a long time having passed since the last labeling. However, even in such cases, by extracting the image frame with the feature most similar to the label boundary feature of the reference cycle S from the increased number of image frames as in this embodiment and correcting the labeling using this image frame, it is possible to flexibly address the reduction in accuracy or deviation of the rough labeling.

[0081] Furthermore, in this embodiment, if there is no image frame having features most similar to the features of the label boundary of the reference cycle S among the multiple image frames having features near the label of the target cycle T, the user is notified that the labeling trend has changed and is provided with information on the labeling before and after the change. More specifically, if a maximum value of similarity is obtained from a newly established range by increasing the number of image frames and such a maximum value is obtained consecutively a predetermined number of times or more, the user is notified that the labeling trend has changed and is provided with information on the labeling before and after the change. This allows the user to efficiently correct the labeling when the labeling trend has changed. Furthermore, if the user determines that the information on the change in labeling trend is incorrect, the user understands that the label correction control unit 8 of the computer of the labeling assistance device has performed different labeling in the expanded range of image frames, and can change the response of the label correction control unit 8 as necessary. [Explanation of symbols]

[0082] 1. Reference cycle label input section 2. Processing target cycle label input section 3. Image feature extraction 4...Similarity determination section 5. Label correction section 6. Skeleton estimation part 7. Label prediction part 8. Label correction control section S... Reference cycle T: Cycle to be processed T1: First processing cycle

Claims

1. A labeling support method for supporting a user in labeling time-series image data obtained by repeating a plurality of unit tasks, each of which is a cycle of a plurality of unit tasks performed in series by a worker, the method comprising: providing each label of the reference cycle labeled with a predetermined accuracy; preparing labels for each of the target cycles labeled by the user with less accuracy than the reference cycle; extracting, as label boundary features, image frame features that define a label boundary between two unit operations that are adjacent in a time series direction of the reference cycle; extracting, as label vicinity features, features of an image frame that defines a label boundary of the process target cycle corresponding to the two unit operations of the reference cycle, and features of a plurality of image frames in the vicinity before and after this image frame; determining a similarity between a feature of the label boundary of the reference cycle and a feature of the label vicinity of the processing target cycle; correcting the label boundary assigned by the user in the cycle to be processed using an image frame having features most similar to the features of the label boundary of the reference cycle among a plurality of image frames having features near the label of the cycle to be processed; Labeling assistance methods.

2. obtaining the label boundary features by estimating the worker's skeleton in image frames having the label boundary features of the reference cycle; The labeling support method according to claim 1, wherein the features near the label of the cycle to be processed are acquired by estimating the skeleton of the worker in a plurality of image frames having features near the label.

3. training a classification model that predicts a label boundary based on the features of the label boundary, the features near the label, and the modified label boundary; predicting a label boundary for a next cycle using the classification model; 2. The labeling assistance method according to claim 1, further comprising: when the predicted label boundary matches or is similar to the corrected label boundary, stopping labeling by the user and automatically performing labeling using the classification model.

4. If there is no image frame having a feature most similar to the feature of the label boundary of the reference cycle among the plurality of image frames having a feature near the label of the cycle to be processed, increasing the number of image frames on the side where the similarity between the feature of the label boundary and the feature near the label is increasing among the plurality of image frames before and after the image frame; extracting an image frame having characteristics most similar to the characteristics of the label boundary of the reference cycle from among the increased image frames; 2. The method of claim 1, further comprising: correcting a label boundary assigned by the user in the cycle to be processed using the image frame.

5. 2. The labeling support method according to claim 1, further comprising the step of notifying the user that a labeling trend is changing and providing information on the labeling before and after the change, when there is no image frame having a feature most similar to the feature on the label boundary of the reference cycle among a plurality of image frames having features near the label of the cycle to be processed.

6. A labeling support device that supports a user in labeling time-series image data obtained by repeating a plurality of unit tasks, each of which is a cycle of a plurality of unit tasks performed in series by a worker, the time-series image data including the plurality of unit tasks, the labeling support device comprising: a storage unit for storing each label of a reference cycle labeled with a predetermined accuracy and each label of a process target cycle labeled by the user with a lower accuracy than the reference cycle; an extracting unit that extracts, as label boundary features, image frame features that define a label boundary between two unit operations that are adjacent in a time series direction of the reference cycle, and further extracts, as label neighborhood features, image frame features that define a label boundary of the process target cycle corresponding to the two unit operations of the reference cycle, and features of a plurality of image frames in the vicinity before and after the image frame; a determination unit that determines similarity between a feature of the label boundary of the reference cycle and a feature near the label of the target cycle; a correction unit that corrects the label boundary assigned by the user in the target cycle using an image frame having a feature most similar to the feature of the label boundary of the reference cycle among a plurality of image frames having features near the label of the reference cycle; A labeling assistance device comprising:

7. A labeling support program that causes a computer to execute the labeling support method according to claim 1.

Images