Work process identification method, work process identification device and program

Abstract

This invention provides a method for identifying work processes that can accurately identify work processes from videos of work performed by machinery using an object detection algorithm. [Solution] The present invention relates to a work process identification method that identifies work processes from a video of work performed by a work machine 1 using an object detection algorithm. The work process identification method of the present invention includes a definition step S-DI which defines work processes according to the content of the work and defines multiple labels to correspond to multiple work processes; a label detection step S-LD which acquires initial data in which corresponding labels are detected from each frame of the video; an alignment step S-AL which acquires alignment data in which the identified multiple sections are aligned in chronological order; and a identification step S-ID which identifies the required time for each of the multiple work processes based on the alignment data, and further includes a filtering step SF which removes false detections contained in the initial data and alignment data.

Description

Technical Field

[0001] The present invention relates to a work process identification method, a work process identification device, and a program for identifying a work process from a moving image of a work performed by a work machine using an object detection algorithm.

Background Art

[0002] In general, work machines such as dismantling machines used for automobile dismantling work are complicated and difficult to operate. For this reason, when an unskilled operator operates a work machine to perform various types of work, unnecessary movements are likely to occur.

[0003] From the viewpoint of reducing such unnecessary movements, various work analysis methods have been conventionally proposed. For example, Patent Document 1 below proposes a work analysis method capable of automatically analyzing dismantling work performed by a crusher based on work trajectory data of the crusher attached to the tip of a work attachment.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0005] By the way, in recent years, work analysis is sometimes performed by causing an object detection algorithm to perform image recognition on moving images of various types of work. Therefore, it is conceivable to use such a method to analyze the work of work machines.

[0006] However, conventional object detection algorithms simply surround any object in the recognized image with a bounding box based on pre-trained data and detect a specific label. Therefore, conventional methods have difficulty recognizing which work process a machine is performing, making it difficult to obtain useful data for work analysis.

[0007] This invention was devised in view of the above-described circumstances, and its main objective is to provide a method for identifying work processes that can accurately identify work processes from videos of work performed by work machines using an object detection algorithm. [Means for solving the problem]

[0008] A first aspect of the present invention is a work process identification method that identifies work processes from a video of work performed by a work machine using an object detection algorithm, comprising: a definition step of defining a plurality of work processes performed between the start and end of a work according to the content of the work to be performed, and defining a plurality of labels that the object detection algorithm should detect from the video so as to correspond to the plurality of work processes; a label detection step of loading the target video into the object detection algorithm which has been pre-trained with the contents of the plurality of labels, and obtaining initial data in which the corresponding labels are detected from each frame of the video; an alignment step of identifying intervals in which the same type of label is continuously detected for each type of label based on the initial data, and obtaining alignment data in which the identified plurality of intervals are aligned in chronological order; and an identification step of identifying the required time for each of the plurality of work processes based on the alignment data, further comprising a filtering step of removing false detections contained in the initial data and the alignment data.

[0009] In the work process identification method of the present invention, it is preferable that the filtering step includes a first filtering step performed between the label detection step and the alignment step, and a second filtering step performed between the alignment step and the identification step.

[0010] In the method for specifying work processes of the present invention, the work involves dismantling an object to be dismantled using a crusher provided in the work machine, and it is desirable that the work process includes a plurality of steps of separating components contained in the object to be dismantled from the object to be dismantled using the crusher, and moving the separated components to a sorting destination.

[0011] In the method for specifying work processes of the present invention, the object to be dismantled is an automobile, the components are classified into one of the following: hood, vehicle bottom component, trunk, roof, interior remaining component, door, and other remaining component, and it is desirable that the work processes be classified into multiple processes for each component.

[0012] In the work process identification method of the present invention, it is preferable that the plurality of labels are classified based on information about the orientation of the automobile and information about the classification of the components.

[0013] In the work process identification method of the present invention, the object detection algorithm preferably detects the label by surrounding the object to be detected in the video with a bounding box, and the first filtering step preferably includes a step of removing the label as a false detection if a part of the bounding box is located outside the image frame of the video.

[0014] In the work process identification method of the present invention, the object detection algorithm preferably detects the label by surrounding the object to be detected in the video with a bounding box, and the first filtering step preferably includes a step of deciding whether to remove the corresponding label as a false detection based on the height and width information of the bounding box.

[0015] In the work process identification method of the present invention, it is preferable that the object detection algorithm detects the label and evaluates the detection confidence in the label detection step, and the first filtering step includes a step of deciding whether to remove the corresponding label as a false detection based on the integral value of the detection confidence.

[0016] In the work process identification method of the present invention, it is preferable that the second filtering step includes a step of removing the other type of label if a different type of label belonging to one of the intervals is detected at the same time as the first interval.

[0017] In the work process identification method of the present invention, it is preferable that one of the plurality of work processes includes a repetitive operation performed at a predetermined operating cycle, and the second filtering step includes a step of removing the corresponding section if the duration of the section is shorter than the operating cycle.

[0018] In the work process identification method of the present invention, the plurality of labels preferably include a specific label and dependent labels which, due to the nature of the work, are inevitably detected after the specific label, and the second filtering step preferably includes a step of removing the dependent label if the dependent label is detected before the specific label.

[0019] In the work process identification method of the present invention, the second filtering step preferably includes a step of removing subsequent sections if multiple sections having the same type of label are detected.

[0020] In the work process identification method of the present invention, the second filtering data obtained in the second filtering step is arranged in chronological order, with a plurality of sections whose start times are identified, and it is desirable that the identification step identifies the required time of the work process corresponding to the label included in the specific section by considering the start time of the section immediately following the specific section in the second filtering data as the end time of the specific section.

[0021] In the work process identification method of the present invention, it is desirable that the identification step further subdivides the work process corresponding to the label included in one section based on the second filtering data obtained in the second filtering step and the data obtained from the sensor provided on the work machine, and then identifies the time required for each process.

[0022] A second aspect of the present invention is a work process identification device that identifies work processes from a video of work performed by a work machine using an object detection algorithm, comprising: a storage unit that stores a plurality of predefined work processes and a plurality of predefined labels corresponding to the plurality of work processes; and a control unit that receives data from the storage unit and identifies the work processes, wherein the control unit can perform a label detection step of loading the target video into the object detection algorithm which has been pre-trained with the contents of the plurality of labels, and acquiring initial data in which the corresponding labels are detected from each frame of the video; based on the initial data, it can perform an alignment step of identifying intervals in which the same type of label is continuously detected for each type of label, and acquiring alignment data in which the identified plurality of intervals are aligned in chronological order; based on the alignment data, it can perform a identification step of identifying the required time for each of the plurality of work processes; and it can perform a filtering step of removing false detections contained in the initial data and the alignment data.

[0023] A third aspect of the present invention is a program for identifying work processes from a moving image of work performed by a work machine using an object detection algorithm, which causes a computer to: store a plurality of predefined work processes and a plurality of predefined labels corresponding to the plurality of work processes; cause the object detection algorithm, which has learned the content of the plurality of labels in advance, to read the target moving image and acquire initial data in which the corresponding label is detected from each frame of the moving image, this being a label detection step; identify, for each label type, a section in which the same type of label is continuously detected based on the initial data, and acquire aligned data obtained by aligning the plurality of identified sections in chronological order, this being an alignment step; and identify the required time for each of the plurality of work processes based on the aligned data, this being an identification step, and further causes the computer to execute a filtering step that removes false detections included in the initial data or the aligned data. [Effects of the Invention]

[0024] The work process identification method of the present invention, by adopting the above features, can accurately identify work processes from a moving image of work performed by a work machine. This also makes it possible to easily obtain useful data for work analysis. [Brief Description of the Drawings]

[0025] [Figure 1] It is a side view of the work machine used in the present embodiment. [Figure 2] It is a schematic diagram when the work machine in Fig. 1 is performing work. [Figure 3] It is a flowchart of the work process identification method of the present embodiment. [Figure 4] It is a schematic diagram conceptually showing initial data. [Figure 5] It is a flowchart of the first filtering step. [Figure 6] (a) and (b) are each schematic diagrams when an object within an image frame of a moving image is surrounded by a bounding box of the object detection algorithm. [Figure 7] This is a schematic diagram showing how objects within the video's image frame are enclosed by the bounding box of an object detection algorithm. [Figure 8] This is a magnified view of the detection confidence graph. [Figure 9] This is a schematic diagram conceptually illustrating the first filtering data. [Figure 10] This is a schematic diagram conceptually representing sorted data. [Figure 11] This is a flowchart of the second filtering step. [Figure 12] This is a conceptual schematic diagram of the processing at a specific step. [Figure 13] This is a schematic diagram showing the state of a shredder when separating the trunk or roof of a car. [Figure 14] (a) is an enlarged view of the section corresponding to the trunk / ceiling process shown in Figure 12, and (b) is a graph showing the change in the arm angle θ. [Figure 15] This is a block diagram showing an example of a work process identification device of the present invention. [Modes for carrying out the invention]

[0026] Hereinafter, one embodiment of the present invention will be described with reference to the drawings. The present invention relates to a work process identification method that identifies work processes from video footage of work performed by a work machine using an object detection algorithm. Figure 1 is a side view showing a work machine 1 used in the work process identification method of this embodiment. As shown in Figure 1, the work machine 1 used in this embodiment is equipped, for example, with a crusher 3 at the tip of an arm 2 that can grip and cut an object to be worked on. The work performed in this embodiment is to dismantle an object to be dismantled using the crusher 3 provided in this work machine 1. Specifically, the object to be dismantled is an automobile.

[0027] Figure 2 is a schematic diagram showing the situation when the work machine 1 is performing work. As shown in Figure 2, the work in this embodiment involves dismantling the object to be dismantled 4 using the crusher 3 (shown in Figure 1) provided by the work machine 1. Specifically, it includes multiple steps of separating the components contained in the automobile, which is the object to be dismantled 4, using the crusher 3, and moving the separated components to the sorting destination 5. The automobile to be dismantled is a sedan-type passenger car with a trunk and four doors. The work in this embodiment involves separating each component of one automobile and moving them to the sorting destination 5, and then moving all the components to the sorting destination 5. In the work process identification method of this embodiment, an object detection algorithm is used to identify the time required for the multiple work processes included in such work.

[0028] An object detection algorithm is an algorithm that identifies objects in an image and determines their location. Generally, object detection algorithms obtain information about the type and location of target objects by performing the following steps on the input image: feature extraction, candidate region generation, and classification.

[0029] In this embodiment, for example, YOLOv5 (You Only Look Once version 5) is used as the object detection algorithm. YOLOv5 is a deep learning-based object detection algorithm that uses a convolutional neural network (CNN), and is capable of classifying and estimating the location of multiple objects contained in an input image in a single process. This enables fast and highly accurate object detection. In this object detection algorithm, the objects to be detected in the video are enclosed in a bounding box, and the label corresponding to this object is detected. At the same time, the coordinates of the center position of this bounding box, as well as the height and width information, are also recorded as information of the detected label.

[0030] However, the object detection algorithm used in the present invention is not limited to those described above. Depending on the purpose, the present invention may appropriately employ known object detection algorithms such as Faster R-CNN (Faster Region-based Convolutional Neural Network), SSD (Single Shot MultiBox Detector), and EfficientDet.

[0031] Figure 3 shows a flowchart of the work process identification method of this embodiment. As shown in Figure 3, the work process identification method of this embodiment includes a definition step S-DI, a label detection step S-LD, an alignment step S-AL, a identification step S-ID, and further includes a filtering step SF. The filtering step SF of this embodiment includes a first filtering step S-F1 and a second filtering step S-F2. The first filtering step S-F1 is performed between the label detection step S-LD and the alignment step S-AL. The second filtering step S-F2 is performed between the alignment step S-AL and the identification step S-ID. The various steps in the work process identification method of this embodiment will be described below.

[0032] <Definition Step S-DI> In the definition step S-DI, depending on the content of the target task, multiple work processes to be performed from the start to the end of the task are defined, and multiple labels that the object detection algorithm should detect from the video are defined to correspond to these multiple work processes.

[0033] In this embodiment, the components included in the automobile, which is the object to be dismantled 4, are classified into one of the following: hood, vehicle bottom, trunk, roof, interior remaining components, doors, and other remaining components. Furthermore, the work process is divided into multiple steps for each of the above components. Specifically, the work process is divided into the hood process P1, the vehicle bottom component process P2, the trunk process P3, the roof process P4, the interior remaining components process P5, the door process P6, and the other remaining components process P7. However, at the stage when the object detection algorithm assigns labels, the trunk process P3 and the roof process P4 are treated as one process (trunk-roof process P3-4). This trunk-roof process P3-4 is later separated and identified as two processes (trunk process P3 and roof process P4) by a method described later.

[0034] The bonnet process P1 means that the crusher 3 grasps the bonnet and moves it to sorting destination 5, and includes the movement of the bonnet and all similar components to sorting destination 5. The vehicle bottom component process P2 means that the crusher 3 grasps the vehicle bottom components (various components that can be obtained from the underside of the vehicle) and moves them to sorting destination 5, and includes the movement of all vehicle bottom components to sorting destination 5. The same applies to other work processes.

[0035] The above-described definition of the work process is specific to the work in this embodiment (automobile dismantling). The present invention is not limited to this embodiment, and the definition of the work process can be modified in various ways depending on the content of the work.

[0036] In actual work, the orientation and state of the vehicle change as work processes P1 to P7 progress. Taking these changes into account, in definition step S-DI, multiple labels that the object detection algorithm should detect are defined to correspond to the multiple work processes P1 to P7 defined above. In other words, when the object detection algorithm recognizes an image and detects a label, the labels are defined so that the work process being performed in the image can be inferred from the detected label.

[0037] In this embodiment, labels L1 to L9 are defined as follows, and the corresponding work processes are as indicated in parentheses. Label L1: Car with hood visible in a vertical orientation (corresponds to hood process P1) Label L2: Car with hood visible at an angle (corresponds to hood process P1) Label L3: Automobile in an overturned state with the bottom facing upwards (corresponding to vehicle bottom component process P2) Label L4: A car with a roof, oriented vertically, showing the trunk side (corresponding to trunk / roof process P3-4) Label L5: A car with the trunk side visible in a vertical orientation and the roof removed (corresponding to process P5 for residual interior components). Label L6: A car with the trunk side visible in a vertical orientation, with the ceiling and seats removed (corresponding to process P5 for remaining interior components). Label L7: A car in which two doors can be observed from the side (corresponding to door process P6) Label L8: Automobile with one door visible from the side (corresponding to door process P6) Label L9: Automobile with all four doors removed, facing sideways (corresponding to process P7 for other residual materials)

[0038] As described above, in this embodiment, for example, labels L1 and L2 correspond to the hood process P1. Labels L5 and L6 correspond to the interior residual material process P5. Labels L7 and L8 correspond to the door process P6. In this way, multiple labels may correspond to a single work process.

[0039] In this invention, the method of defining labels is not limited to the embodiments described above. The definition of labels is determined as appropriate depending on the work content and the label detection capability of the object detection algorithm.

[0040] <Label detection step S-LD> In the label detection step S-LD, the target video is loaded into an object detection algorithm that has been pre-trained with the contents of multiple labels L1 to L9, and initial data is obtained in which the corresponding label is detected from each frame of the video.

[0041] When training an object detection algorithm, the following steps are typically taken. First, to create a training dataset, a video similar to the task to be performed (in this embodiment, a video of car dismantling) is prepared. This prepared training video is then divided into individual frames, and each image (in this embodiment, the car to be dismantled) is annotated based on a defined set of labels (in this embodiment, labels L1 to L9 mentioned above). After annotation is complete, a training dataset is created by linking images with text data indicating the objects within those images. It is desirable to prepare as many training datasets as possible. The object detection algorithm is then trained using the prepared training datasets.

[0042] As described above, the object detection algorithm that was trained is fed the target video (i.e., the video in which we intend to identify the work process), and initial data is obtained in which the corresponding label is detected from each frame of the video. Figure 4 shows a schematic diagram conceptually representing the initial data. In Figure 4, the detection confidence level for each label L1 to L9 is shown as a graph. The horizontal axis of each graph represents the time from the start of the work (t=0) to the end of the work (t=1600), and the unit is "seconds". The vertical axis of each graph represents the detection confidence level for the corresponding label.

[0043] In object detection algorithms, detection confidence is an index that indicates the degree of confidence that the algorithm is correct when it detects a particular object. In this embodiment, detection confidence is expressed as a value between 0 and 1, with a higher value indicating a higher probability that the detected object is correctly identified.

[0044] A general observation of Figure 4 reveals areas where waveforms indicating detection confidence are concentrated, and areas where these waveforms are sparsely detected. Furthermore, there are locations where waveforms for multiple labels are detected simultaneously. These indicate that the initial data included false positives from the object detection algorithm.

[0045] Assuming that the initial data contains false positives, further observation of Figure 4 allows us to infer, for example, the following: Observing the area A1 enclosed by the dashed line, we can see that the detection confidence for labels L1 and L2 is high within this area. From this, we can infer that the work process corresponding to labels L1 and L2 (in this embodiment, the bonnet process P1) was performed during the time period corresponding to this area A1. Similarly, observing the area A2 enclosed by the dashed line, we can see that the detection confidence for labels L5 and L6 is high within this area. From this, we can infer that the work process corresponding to labels L5 and L6 (in this embodiment, the in-vehicle residual material process P5) was performed during the time period corresponding to this area A2.

[0046] In this invention, the work process is accurately identified from a video of the work performed by the work machine using this estimation method. Furthermore, as shown in Figure 3, in this invention, a filtering step SF is performed to remove false detections contained in the initial data and the alignment data described later, from the viewpoint of improving the accuracy of estimation. In this embodiment, as the filtering step SF, the following first filtering step S-F1 is performed between the label detection step S-LD and the alignment step S-AL described later.

[0047] <First filtering step S-F1> Figure 5 is a flowchart of the first filtering step S-F1. In the first filtering step S-F1 of this embodiment, false detections are removed from the initial data through the steps shown in Figure 5 to obtain the first filtered data. One method for doing this is to use bounding box information. As described above, the object detection algorithm surrounds the objects to be detected in the video with bounding boxes and detects labels. False detections are removed based on the bounding box information obtained simultaneously.

[0048] Figures 6(a) and 6(b) show schematic diagrams of when an object 21 (car) within the video image frame 20 is enclosed by the bounding box 22 of the object detection algorithm. As shown in Figure 6(a), if the bounding box 22 is completely contained within the video image frame 20, the detection is likely to be correct. On the other hand, as shown in Figure 6(b), if a portion 22a of the bounding box 22 is located outside the video image frame 20, the detection is likely to be a false positive. Therefore, the first filtering step S-F1 of this embodiment includes a step of removing labels where a portion 22a of the bounding box 22 is located outside the video image frame 20 as false positives. This makes it possible to remove waveforms that are likely to be false positives from the initial data shown in Figure 4. Note that "removing a label as a false positive" means treating the detection confidence of that label as 0.

[0049] Furthermore, a misdetection by the object detection algorithm may occur due to a discrepancy between the shape of the bounding box 22 (relationship between height H and width W) and the content of the detected label. From the perspective of eliminating such misdetections, as shown in Figure 5, the first filtering step S-F1 includes a step of deciding whether to remove the corresponding label as a misdetection based on the height H and width W information of the bounding box 22. The specific details of this step will be explained with reference to Figure 7.

[0050] Similar to FIG. 6, FIG. 7 is a schematic diagram showing that an object 21 (automobile) within an image frame 20 of a moving image is surrounded by a bounding box 22 generated by an object detection algorithm. As shown in FIG. 7, for example, when a horizontally oriented automobile exists as the object 21 in the image, normally the object detection algorithm surrounds the object 21 with a horizontally long bounding box 22 (H<W), and a label corresponding to a horizontally oriented automobile (labels L7, L8 and L9 correspond thereto in the present embodiment) is detected.

[0051] However, in actual detection, although the object detection algorithm surrounds the object 21 with a horizontally long bounding box 22, the object detection algorithm may detect a label that presupposes the object 21 is vertically long (in the present embodiment, the labels L1, L4, L5 and L6 are labels for vertically oriented automobiles). Such detection is regarded as having a contradiction between the shape of the bounding box 22 and the content of the detected label, and is removed as a false detection.

[0052] From a similar point of view, in the first filtering step S-F1 of the present embodiment, although the object 21 is surrounded by a vertically long bounding box 22 (H>W), if a label that presupposes the object 21 is horizontally long (in the present embodiment, labels L7, L8 and L9 for horizontally oriented automobiles) is detected, this detection is removed as a false detection (not shown in the figures).

[0053] Furthermore, depending on how the label is defined, there may be labels that assume that the ratio of the height H to the width W of the bounding box 22 falls within a specific range. For example, in the case of label L2 in this embodiment, it is assumed that the car is oriented at an angle, so when this is detected, the bounding box 22 becomes close to a square, and the ratio of the height H to the width W falls within a specific range. In the first filtering step S-F1, if a label (label L2 in this embodiment) that assumes this is the case is assigned even though the ratio of the height H to the width W of the bounding box 22 falls outside the specific range, this detection is removed as a false positive.

[0054] As shown in Figure 5, the first filtering step S-F1 of this embodiment includes a step of deciding whether to remove the corresponding label as a false positive based on the integral value of the detection confidence. This step will be described below with reference to Figure 8. Figure 8 shows an enlarged view of an example of a detection confidence graph. As shown in Figure 8, the object detection algorithm detects labels and evaluates the detection confidence in the label detection step S-LD.

[0055] For example, in Figure 8, region A3 can be observed to have a larger integrated value of detection confidence compared to regions A4 and A5. On the other hand, in another region A4, the same integrated value can be observed to be smaller compared to regions A3 and A5. In the first filtering step S-F1, labels with such small integrated values ​​are removed as false positives. The range of the horizontal axis (time) for performing the integration and the threshold value of the integrated value used to determine whether or not it is a false positive are determined appropriately based on various experiments and empirical rules. Therefore, the determination of the range and threshold will determine, for example, whether or not to treat region A5 as a false positive. This method makes it possible to remove waveforms that are highly likely to be false positives from the initial data.

[0056] Figure 9 conceptually shows the first filtered data obtained by removing false detections in the first filtering step S-F1 from the initial data shown in Figure 4. In Figure 9, the waveforms removed as false detections are shown as dashed lines. In this embodiment, subsequent steps are performed using this first filtered data.

[0057] <Alignment Step S-AL> As shown in Figure 3, in the sorting step S-AL, based on the initial data (or data obtained from the initial data), intervals in which the same type of label is continuously detected are identified for each type of label, and sorted data is obtained by sorting the identified intervals in chronological order. In this embodiment, since the first filtering step S-F1 is performed before the sorting step S-AL, the sorting step S-AL is performed based on the first filtering data obtained from the initial data, and the sorted data described above is obtained.

[0058] Figure 10 shows a schematic diagram conceptually illustrating the acquired sorted data. As shown in Figure 10, sorted data is, for example, obtained by identifying intervals from the first filtering data (shown in Figure 9) where the same type of label is continuously detected, i.e., intervals where a waveform indicating the confidence level of label detection is continuously detected. In Figure 10, the detected intervals are labeled Z1 to Z17. The number after "Z" indicates the order in the time series.

[0059] From this sorted data, it becomes possible to infer which work process was performed at which time. For example, section Z1 is the section where label L1 was detected, and Z2 is the section where label 2 was detected. It can be inferred that the bonnet process P1, corresponding to labels L1 and L2, was performed during the time period encompassing sections Z1 and Z2. Similarly, section Z5 is the section where label L4 was detected. It can be inferred that the trunk / roof processes P3 and P4, corresponding to label L4, were performed during the time period of section Z5. Using this method, work processes can be identified from video footage of the work.

[0060] It is also possible to directly identify the work process from the alignment data. However, the alignment data may still contain false detections. Therefore, as shown in Figure 3, in this embodiment, from the viewpoint of improving the accuracy of work process identification, a second filtering step S-F2 is performed as a filtering step SF after the alignment step S-AL to remove false detections from the alignment data.

[0061] <Second filtering step S-F2> Figure 11 is a flowchart of the second filtering step S-F2. As shown in Figure 11, in the second filtering step S-F2, the second filtered data is obtained by removing intervals or labels that do not satisfy the conditions for the work process from the sorted data. Examples of methods for doing this include the following:

[0062] The second filtering step S-F2 includes removing any labels of a different type that belong to a given interval if such labels are detected at the same time as the interval. For example, in the sorted data shown in Figure 10, interval Z9 with a different type of label L1 is detected at the same time as interval Z8. Such interval Z9 is likely to be a false positive and is therefore removed.

[0063] Furthermore, as shown in Figure 2, the operation in this embodiment involves repeatedly dismantling a car and moving its components to their respective sorting locations. Therefore, the work processes included in the operation consist of repetitive operations performed at a predetermined operation cycle. Consequently, it is assumed that the time required for each work process is greater than the aforementioned operation cycle. In the sorting data, there may be intervals that contradict this assumption, so it is desirable to remove these.

[0064] From this perspective, as shown in Figure 11, the second filtering step S-F2 includes a step of removing a section if the duration of the section is shorter than the operation cycle. For example, in the sorted data shown in Figure 10, section Z6 is detected, but since it is shorter than the operation cycle, it is removed as a false detection.

[0065] Furthermore, labels L1 to L9 include a specific label and dependent labels that, due to the nature of the work, are inevitably detected after the aforementioned specific label. For example, label L9 is a label for a car in which all four doors have been removed, and naturally, it is inevitably detected after label L5 (a label in which two doors can be observed) and label L6 (a label in which one door can be observed). In other words, label L9 is a dependent label of labels L5 and L6.

[0066] In another example, labels L5 and L6 are labels for cars with the roof removed, and are therefore inevitably detected after label L4 (with roof). In other words, labels L7 and L8 are dependent labels of label L4, respectively. In sorted data, if such dependent labels are detected before a particular label, it is desirable to remove them.

[0067] From this perspective, as shown in Figure 11, the second filtering step S-F2 includes a step of removing the dependent label if it is detected before the specific label. For example, in the sorted data shown in Figure 10, the interval Z3 of label 7 is detected before the interval Z5 of label 4, and is therefore removed as a false positive in the second filtering step S-F2.

[0068] In principle, the work in this embodiment does not involve repeating a work process after it has been completed. Therefore, as shown in Figure 11, the second filtering step S-F2 of this embodiment includes a step of removing subsequent sections if multiple sections with the same type of label are detected. For example, in the sorted data shown in Figure 10, for label L1, after section Z1 is detected, sections Z10 and Z17 are detected. Similarly, for label L6, after section Z7 is detected, section Z15 is detected. These sections are removed as false positives in the second filtering step S-F2. The second filtering data is obtained in the second filtering step S-F2 using this method.

[0069] <Specific Step S-ID> As shown in Figure 3, in specific step S-ID, the required time for each of the multiple work processes is determined based on the sorted data (or data obtained from the sorted data). In this embodiment, since the second filtering step S-F2 is performed before specific step S-ID, specific step S-ID is performed using the second filtering data obtained from the sorted data.

[0070] Figure 12 is a schematic diagram conceptually illustrating the processing of a specific step S-ID. In Figure 12, the data for the intervals (regions where labels were continuously detected) obtained from the second filtering data are arranged by work process. Labels L1 and L2 correspond to the same bonnet process P1. Therefore, in Figure 12, the detection confidence of labels L1 and L2 are combined and shown as a single graph. Similarly, labels L5 and L6 correspond to the same interior residual material process P5. Labels L7 and L8 correspond to the same door process P6. For these labels as well, the detection confidence is combined and shown as a single graph.

[0071] Furthermore, the bottom section of Figure 12 shows a bar graph that identifies the required time for each of the multiple work processes. As shown in Figure 12, in specific step S-ID, the required time for each work process P1 to P7 is identified from the data of each section.

[0072] From the bar graph in Figure 12, for example, it can be seen that the bonnet process P1 is performed between times t0 and t1. It can also be seen that the interior residual material process P5 is performed between times t2 and t3. Similarly, the required time for other work processes can be determined from this bar graph.

[0073] Ideally, data should be detected in a manner that allows only one type of section to be detected at all times from the start to the end of the work, and that the detected section switches seamlessly and sequentially as the work progresses. However, in reality, the detection results are not as ideal as this, and often multiple sections are detected in a manner that is temporally separated or overlapping.

[0074] For example, if the second filtering data obtained in the second filtering step S-F2 consists of multiple intervals with specified start times that are arranged in chronological order without overlap, the identification step S-ID considers the start time of the interval immediately following the specific interval in the second filtering data as the end time of the specific interval, and identifies the required time for the work process corresponding to the label included in the specific interval.

[0075] As an example, section Za, which belongs to the vehicle bottom member process P2 shown in Figure 12, does not overlap in time with section Zb, which belongs to the subsequent trunk / ceiling processes P3 and P4, and is separated from it. In such a case, the start time of section Zb can be considered as the end time of the vehicle bottom member process P2 corresponding to section Za, and a specific step S-ID can be performed.

[0076] Furthermore, sections belonging to different work processes may overlap when viewed in a time series. For example, in Figure 12, sections Zc and Zd slightly overlap. In such cases, the start time of the later section in the time series can be considered as the end time of the earlier section in the time series, and a specific step S-ID can be performed.

[0077] Furthermore, as shown in Figure 12, if two sections Ze and Zf are detected in a single work process, and no other labels are detected in the blank period Ti between these two sections, then it can be assumed that the corresponding work process (in this example, the bonnet process P1) was performed during the two sections and the blank period, and the specific step S-ID can be performed.

[0078] In this embodiment, at the stage when the object detection algorithm assigns labels, the trunk process P3 and the ceiling process P4 are combined and treated as the trunk-ceiling process P3-4. This is because assigning labels to distinguish between the trunk and the ceiling would likely increase false detections given the current accuracy of the object detection algorithm.

[0079] From the perspective of addressing cases where such work processes are included, the specific step S-ID may further subdivide the work processes corresponding to the labels included in a single section based on the second filtering data obtained in the second filtering step S-F2 and the data acquired from the sensors installed on the work machine 1, and then specify the time required for each process. An example of this is described below.

[0080] Figure 13 shows a schematic diagram illustrating the state of the crusher 3 when separating the trunk 7 or roof 8 of the automobile 6 in this embodiment. As shown in Figure 13, the angle of the arm 2 connected to the crusher 3 differs significantly when separating the trunk 7 and when separating the roof 8. Specifically, the angle formed by the plane constituting the outer shape of the arm 2 along the direction from the base end to the tip of the arm 2 and the horizontal plane is larger when separating the roof 8 than when separating the trunk 7. For this reason, the work machine 1 is equipped with a sensor that can acquire the angle of the arm 2, and based on the data acquired from this sensor and the data of the sections corresponding to the trunk / roof processes P3 and P4 included in the second filtering data, it is possible to accurately identify the trunk / roof processes P3 and P4 as trunk process P3 and roof process P4.

[0081] Figure 14(a) is an enlarged view of section Zb corresponding to the trunk / ceiling processes P3 and P4 shown in Figure 12, and (b) is a graph showing the change in the angle θ of arm 2 obtained from the sensor described above. As shown in Figure 14(b), it can be inferred that the trunk process P3 is being performed in region A6 where the angle θ of arm 2 is relatively small. Also, it can be inferred that the ceiling process P4 is being performed in region A7 where the angle θ of arm 2 is relatively large. Therefore, by appropriately setting a threshold for the angle θ, in section Zb (between time ta and time tb) corresponding to the trunk / ceiling processes P3 and P4, it can be inferred that the trunk process P3 is being performed when the angle θ is smaller than the threshold (between time ta and time tc), and that the ceiling process P4 is being performed when the angle θ is larger than the threshold (between time tc and time tb). By this method, in the specific step S-ID of this embodiment, the work process corresponding to the label included in one section can be further subdivided in detail and its required time can be determined.

[0082] The work process identification method of the present invention, by adopting the above-described features, can accurately identify work processes from video footage of work performed by a work machine using an object detection algorithm. In other words, it is possible to determine how long each of multiple work processes takes to perform without a human having to watch the video from start to finish. Therefore, it is possible to easily obtain data useful for work analysis. In this embodiment, both the first filtering step S-F1 and the second filtering step S-F2 are included, but the present invention is not limited to this embodiment, and any method including a filtering step SF that removes false detections in some manner is acceptable.

[0083] Figure 15 is a block diagram showing an example of the work process identification device of the present invention. As shown in Figure 15, the work process identification device of the present invention includes a storage unit and a control unit. The storage unit stores a plurality of predefined work processes and a plurality of predefined labels corresponding to the plurality of work processes. The control unit receives data from the storage unit and identifies the work process. The processing performed by this device is as previously described.

[0084] Furthermore, another aspect of the present invention includes a program that uses an object detection algorithm to identify work processes from a video of the work performed by the work machine 1. This program causes a computer to store a plurality of predefined work processes and a plurality of predefined labels corresponding to the plurality of work processes, and then performs the various steps already described.

[0085] Although particularly preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the illustrated embodiments and can be implemented in various modified forms. [Industrial applicability]

[0086] As described above, the present invention is useful as a work process identification method, work process identification device, and program that can accurately identify work processes from videos of work performed by a work machine using an object detection algorithm. [Explanation of symbols]

[0087] 1. Working Machinery S-DI Definition Steps S-LD Label Detection Step S-AL Alignment Step S-ID Identification Steps SF filtering step

Claims

1. A method for identifying work processes from a video of a work machine performing an action, using an object detection algorithm, A definition step that defines a plurality of work steps to be performed between the start and end of the work, depending on the content of the work to be performed, and defines a plurality of labels that the object detection algorithm should detect from the video to correspond to the plurality of work steps, A label detection step involves loading the target video into the object detection algorithm, which has been pre-trained with the contents of the multiple labels, and obtaining initial data in which the corresponding labels are detected from each frame of the video. A sorting step involves identifying intervals in which the same type of label is continuously detected based on the initial data, and obtaining sorted data by arranging the identified intervals in chronological order. This includes a specific step of determining the required time for each of the multiple work processes based on the aforementioned alignment data, The further step includes a filtering step to remove false detections contained in the initial data and the sorted data, Work process identification method.

2. The filtering step described above is: A first filtering step performed between the label detection step and the alignment step, A method for identifying a work process according to claim 1, comprising a second filtering step performed between the alignment step and the identification step.

3. The aforementioned operation involves dismantling the object to be dismantled using a crusher provided by the aforementioned work machine. The method for specifying a work process according to claim 1, wherein the work process includes a plurality of steps of separating members contained in the object to be demolished from the object to be demolished using the crusher, and moving the separated members to a sorting destination.

4. The object to be dismantled is an automobile. The aforementioned components are classified into one of the following: hood, vehicle bottom components, trunk, ceiling, interior components, doors, and other components. The method for identifying work processes according to claim 3, wherein the work process is classified into multiple processes for each of the members.

5. The method for identifying a work process according to claim 4, wherein the plurality of labels are classified based on information about the orientation of the automobile and information about the classification of the components.

6. The object detection algorithm detects the labels by surrounding the objects to be detected in the video with bounding boxes. The method for identifying a work process according to any one of claims 2 to 5, wherein the first filtering step includes removing as false detection any label in which a portion of the bounding box is located outside the image frame of the video.

7. The object detection algorithm detects the labels by surrounding the objects to be detected in the video with bounding boxes. The work process identification method according to any one of claims 2 to 5, wherein the first filtering step includes a step of determining whether to remove the corresponding label as a false detection based on information of the height and width of the bounding box.

8. The object detection algorithm, in the label detection step, detects the label and evaluates the detection confidence level. The work process identification method according to any one of claims 2 to 5, wherein the first filtering step includes a step of determining whether to remove the corresponding label as a false detection based on the integral value of the detection confidence.

9. The method for identifying a work process according to any one of claims 2 to 5, wherein the second filtering step includes removing the other type of label if a different type of label belonging to one of the intervals is detected at the same time as the interval.

10. One of the aforementioned multiple work processes includes a repetitive operation performed at a predetermined operating cycle, The method for identifying a work process according to any one of claims 2 to 5, wherein the second filtering step includes removing the section if the duration of the section is shorter than the operation cycle.

11. The plurality of labels include a specific label and dependent labels that, due to the nature of the operation, are inevitably to be detected after the specific label. The method for identifying a work process according to any one of claims 2 to 5, wherein the second filtering step includes removing the dependent label if the dependent label is detected before the specific label.

12. The work process identification method according to any one of claims 2 to 5, wherein the second filtering step includes removing subsequent sections if multiple sections having the same type of label have been detected.

13. The second filtering data obtained in the second filtering step is arranged in chronological order, with multiple intervals whose start times are specified, without any overlap between them. The method for identifying a work process according to any one of claims 2 to 5, wherein the identifying step involves determining the start time of the section immediately following the specific section in the second filtering data as the end time of the specific section, and identifying the required time of the work process corresponding to the label included in the specific section.

14. The method for identifying a work process according to any one of claims 2 to 5, wherein the identification step further subdivides the work process corresponding to the label included in one section based on the second filtering data obtained in the second filtering step and the data obtained from the sensor provided on the work machine, and then identifies the time required for each work process.

15. A work process identification device that identifies work processes from videos of work performed by a work machine using an object detection algorithm, The system includes a storage unit that stores a plurality of predefined work processes and a plurality of predefined labels corresponding to the plurality of work processes, and a control unit that receives data from the storage unit and identifies the work processes. The control unit, The object detection algorithm, which has been pre-trained on the contents of the multiple labels, can be given the target video, and a label detection step can be performed to obtain initial data in which the corresponding label is detected from each frame of the video. Based on the initial data, a sorting step can be performed to identify intervals in which the same type of label is continuously detected for each type of label, and to obtain sorted data obtained by arranging the identified intervals in chronological order. Based on the aforementioned alignment data, a specific step can be performed to determine the required time for each of the multiple work processes. A filtering step can be performed to remove false detections contained in the initial data and the sorted data. Work process identification device.

16. A program that uses an object detection algorithm to identify work processes from videos of work performed by a work machine, On the computer, A plurality of predefined work processes and a plurality of predefined labels corresponding to the plurality of work processes are stored. A label detection step involves loading the target video into the object detection algorithm, which has been pre-trained with the contents of the multiple labels, and obtaining initial data in which the corresponding labels are detected from each frame of the video. A sorting step involves identifying intervals in which the same type of label is continuously detected based on the initial data, and obtaining sorted data by arranging the identified intervals in chronological order. Based on the aforementioned alignment data, a specific step is performed to determine the required time for each of the multiple work processes. Further, a filtering step is performed to remove false detections contained in the initial data and the sorted data. program.

Images