Skeletal detection system and work management device

Abstract

A skeleton detection system includes a first storage unit that stores reference motion information related to at least one cycle of a reference motion that is a repetitive motion performed by a worker; a skeleton information detection unit that detects skeleton information of the worker from a dynamic image of the worker in a work; and a target determination unit that determines, as a target, skeleton information of the worker indicating a motion satisfying a target necessary condition in each of the skeleton information of a plurality of workers in the work detected by the skeleton information detection unit, the target necessary condition being that a difference between the reference motion information and check target information that is motion information in the work is within a predetermined motion allowable range.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a skeleton detection system and a work management device. BACKGROUND

[0002] A method of checking a work performed by a worker by using a computer is known. In this method, as shown in Patent Literature 1, for example, an image data of a worker photographed in a work is analyzed by a computer, and an action of the worker appearing in the image data is compared with a determination reference to determine whether the worker is performing the work appropriately.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Application Publication No. 2011-048547 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] However, in the method disclosed in Patent Literature 1, in a case where a plurality of workers are photographed at the same time, a worker who should be a check target can be mistaken for another worker by the computer due to a change in the action of the worker. In addition, a load of the checking system can increase due to a capacity of the image data for checking and the like.

[0008] Therefore, an object of the present disclosure is to reduce a load of information processing in a system that senses a skeleton of a worker in a case where a work of the worker is checked by a checking system using image data of the worker photographed in the work and the like.

[0009] TECHNICAL SOLUTION

[0010] To solve the above problem, the skeleton detection system of the present disclosure includes a first storage section that stores reference action information related to at least one cycle of a reference action that is a repetitive action performed by a worker, a skeleton information detection section that detects skeleton information of the worker from a moving image of a work performed by the worker photographed in the work, and a target determination section that determines, as a target, skeleton information of the worker indicating an action satisfying a target necessary condition from among each of skeleton information of a plurality of workers in the work detected by the skeleton information detection section, the target necessary condition being that a difference between the reference action information and check target information that is action information of the work is within a predetermined action allowable range.

[0011] According to the above configuration, the target can be determined from the skeleton information of the worker detected from the dynamic image during the work. That is, by comparing the examination target information of the worker with the reference motion information of the worker stored in the first storage unit, the target can be determined. In particular, the skeleton detection system of the present disclosure sets the repeated motion of the worker as the examination target, and the motion of the worker of the examination target has regularity and periodicity. Therefore, the determination of the target can be performed accurately and quickly using the skeleton detection system of the present disclosure. Further, the target is determined by the target determination unit, whereby the worker of the examination target can be determined from the worker photographed in the dynamic image during the work. Therefore, the load of the skeleton detection system can be suppressed.

[0012] Effects of Invention

[0013] According to the present disclosure, in a case where the work of the worker is examined using the image data photographed the worker in the work and by the examination system, even in a case where a plurality of workers are photographed at the same time, the worker can be appropriately examined, and the load of the examination system can be suppressed. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a functional block diagram of the skeleton detection system of the first embodiment.

[0015] Figure 2 is a graph showing the skeleton information of the front side of the worker detected by the skeleton information detection unit of Figure 1

[0016] Figure 3 is a graph showing the skeleton information of the side surface side of the worker detected by the skeleton information detection unit of Figure 1

[0017] Figure 4 is a flowchart of the target determination processing of the skeleton detection system of Figure 1

[0018] Figure 5 is a graph schematically showing the situation until the target is determined from the photographed data of the dynamic image during the work in the target determination processing of the first embodiment.

[0019] Figure 6 is a graph showing the position information of the front side of the worker included in the skeleton information of the third modification example.

[0020] Figure 7 is a graph showing the position information of the side surface side of the worker included in the skeleton information of the third modification example.

[0021] Figure 8 is a flowchart showing the overall operation of the skeleton detection system of the fourth modification example. ​​​

[0022] Figure 9 is a functional block diagram of the work management apparatus of the second embodiment.

[0023] Figure 10 is a flowchart showing the overall operation of the work management apparatus of Figure 9 .

[0024] Figure 11 is a sub-flowchart of the eligibility determination process regarding Figure 10 . DETAILED DESCRIPTION

[0025] Hereinafter, each embodiment will be described with reference to the drawings.

[0026] (First Embodiment)

[0027] [Skeleton detection system]

[0028] Figure 1 is a functional block diagram of the skeleton detection system 1 of the first embodiment. The skeleton detection system 1 is an inspection system that inspects the motion of a worker. The skeleton detection system 1 detects skeleton information indicating the motion of a worker from work-in-progress dynamic images (hereinafter, also simply referred to as work-in-progress dynamic images) that are captured of a worker in work. The skeleton information detected by the skeleton detection system 1 is used for inspection of repetitive motions performed by a worker in work. As the repetitive motions that are the inspection targets, for example, assembly work of products can be given, but is not limited thereto.

[0029] The skeleton detection system 1 includes a photographing apparatus 2 that photographs work-in-progress dynamic images and a detection apparatus 3 that detects skeleton information from the work-in-progress dynamic images. The photographing apparatus 2 is a video camera having, as one example, a CCD (Charge Coupled Device) or the like as an imaging element, generates work-in-progress dynamic images, and outputs the data. The photographing apparatus 2 is disposed at a position at which a worker in work can be photographed. As one example, the photographing apparatus 2 is disposed so as to photograph a certain area of a factory in which a worker performs work. The photographing apparatus 2 and the detection apparatus 3 are connected by wireless or wired connection.

[0030] The detection device 3 has an arithmetic unit 30 that receives data of the work-in-progress dynamic image output from the photographing device 2, and a first storage unit 31, a second storage unit 32, and an output unit 33 that are individually controlled by the arithmetic unit 30. The first storage unit 31 stores reference movement information that relates to the amount of at least one cycle of a reference movement that is obtained from a reference dynamic image of a reference movement of the worker and that is repeatedly performed by the worker. The reference movement is a movement that becomes an example of the repeated movement of the worker, and is set for each work. In the case of the reference movement in the present embodiment, the reference movement can be set based on the reference movements of a plurality of workers who perform the same work. As one example, the first storage unit 31 of the present embodiment stores reference movement information of the worker that is obtained from a reference dynamic image (hereinafter, also simply referred to as a reference dynamic image) of a reference movement of the worker that is photographed. As one example, the reference dynamic image is a dynamic image of the worker that is photographed in a state in which the movement of the worker can be clearly confirmed (for example, a state in which there is no obstacle between the worker and the photographing device 2). The reference dynamic image includes reference movement information that corresponds to the reference movement of at least one cycle or more of the worker. The time of one cycle of the reference movement is, for example, several seconds to several minutes or so, but is not limited thereto.

[0031] The second storage unit 32 stores inspection object information that relates to the amount of at least one cycle of a repeated movement (main movement) of the worker that is performed as an actual work. The repeated movement (main movement) is photographed by the photographing device 2. By storing the inspection object information of the repeated movement (main movement) in the second storage unit 32, post-verification can be performed. The information stored in the first storage unit 31 and the second storage unit 32 can be read out by the arithmetic unit 30 at a prescribed timing, but the information stored in the first storage unit 31 and the inspection object information of the repeated movement (main movement) photographed by the photographing device 2 can be set to be appropriately read out by the arithmetic unit 30. The output unit 33 outputs the information stored in the first storage unit 31 and the second storage unit 32. As one example, the output unit 33 is a display unit.

[0032] As an example, the detection device 3 is realized by a computer provided with a processor such as a CPU, a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive) or the like as a recording medium, and an LCD (Liquid Crystal Display) or the like as a display. The operation section 30 is realized by the processor. The number of processors can be either one or a plurality. The first storage section 31 and the second storage section 32 are realized by the above-described recording medium. Further, the above-described recording medium stores therein a program for causing the operation section 30 to execute each process of the present embodiment and information related to the candidate necessary condition and the target necessary condition described later. The above-described recording medium can be an external storage device provided outside the detection device 3.

[0033] The operation section 30 has a skeleton information detection section 301, a candidate selection section 302, and a target determination section 303. The skeleton information detection section 301 detects the skeleton information of the worker from the work dynamic image output from the imaging device 2. As a method of detecting the skeleton information, a publicly known method can be employed. As the method of detecting the skeleton information, for example, a method using the "Open Pose" technology (refer to Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, "Realtime Multi-Person 2D Pose Estimation using PartAffinity Fields", arXiv:1611.08050v2) which is an open source library (Open source library) published by Carnegie Mellon University and which enables detection of the positions of a plurality of persons in an imaged image after recording or in real time can be cited. According to this method, even in a case where the bodies of a plurality of workers are overlapped in the work dynamic image, the skeleton information of each worker can be distinguished and detected.

[0034] Figure 2 is a graph representing the skeleton information of the front side of the worker W detected by the skeleton information detection section 301 of the Figure 1 Figure 3 is a graph representing the skeleton information of the side side of the worker W detected by the skeleton information detection section 301 of the Figure 1 Figure 2 and Figure 3 schematically show a case where the skeleton information of the worker W is displayed on the display surface 331 of the output section 33. Further, in addition to Figure 2 and Figure 3 , the following Figure 5 ​​Skeleton information of the worker W is schematically shown, and a prescribed position of the skeleton information is shown by a black circle.

[0035] As shown in FIG. 1, as an example, the skeleton information is at least any one of a position of a plurality of parts P of the worker W or a length of a line L connecting two or more of the plurality of parts P of the worker W (for example, a joint part or the like) or a skeleton shape formed by the plurality of parts P and the line L. The information includes position information of two-dimensional coordinates. The skeleton information changes in accordance with a posture of the worker W in the in-work dynamic image. The operation section 30 grasps a change in the skeleton information over a plurality of continuous frames included in the in-work dynamic image. Figure 2 Figure 3 As shown in FIG. 1, as an example, the skeleton information is at least any one of a position of a plurality of parts P of the worker W or a length of a line L connecting two or more of the plurality of parts P of the worker W (for example, a joint part or the like) or a skeleton shape formed by the plurality of parts P and the line L. The information includes position information of two-dimensional coordinates. The skeleton information changes in accordance with a posture of the worker W in the in-work dynamic image. The operation section 30 grasps a change in the skeleton information over a plurality of continuous frames included in the in-work dynamic image.

[0036] The candidate selection section 302 selects a candidate satisfying a prescribed candidate necessary condition from among the plurality of skeleton information of the worker W detected from the in-work dynamic image. Thus, the candidate selection section 302 screens the number of workers W of the inspection object. The candidate necessary condition is set in advance as a necessary condition for the candidate selection section 302 to screen the skeleton information targeted from among the plurality of skeleton information in the in-work dynamic image. In this way, the candidate selection section 302 screens the skeleton information photographed in the in-work dynamic image using the candidate necessary condition, and thus can reduce the burden on the skeleton detection system 1.

[0037] Here, as an example, the candidate necessary condition of the present embodiment includes a condition that the skeleton information is located within a prescribed region of a photographed range shown by the in-work dynamic image. As the prescribed region, for example, a central region of the in-work dynamic image, a region of at least one of the left and right sides, or a work region of the worker W (including a region including a work machine and its vicinity as an example) can be cited, but is not limited thereto. According to such a candidate necessary condition, the candidate selection section 302 can select a candidate having a high possibility of being targeted from among the plurality of skeleton information in the in-work dynamic image by setting the prescribed region described above, and can further reduce the burden on the skeleton detection system 1.

[0038] ​The target determination section 303 determines the skeleton information of the worker W indicating the action satisfying the target necessary condition as a target. The aforementioned target necessary condition is a necessary condition in which the difference between the reference action information and the check target information in each of the skeleton information of the plurality of workers W in the work detected by the skeleton information detection section 301 is within the action allowable range set in advance. If the difference between the check target information and the reference action information of the candidate (worker W) detected by the candidate selection section 302 satisfies the target necessary condition, the target determination section 303 determines the candidate as a target. Note that, in the present embodiment, if the target determination section 303 determines that there is no target, the candidate selection section 302 selects another candidate satisfying the candidate necessary condition. The candidate selection section 302 and the target determination section 303 repeatedly perform the same processing until the target is determined.

[0039] The action allowable range (action range satisfying the target necessary condition) can be appropriately set. As the action allowable range, for example, a range in which the positional displacement of the prescribed position (as one example, the position of the hand of either side) of the skeleton information of the reference action information and the check target information in the repetitive action of one cycle is less than a prescribed distance, a range in which the positional displacement of the track traced by the prescribed position in the repetitive action of one cycle is less than a prescribed distance, or the like can be exemplified. Alternatively, as the action allowable range, for example, a range in which the displacement time of the action involved in at least a part of one cycle of the repetitive action of the skeleton information is less than a prescribed time can be exemplified. Note that, a case in which the time required for the action is too short with respect to the prescribed time can also be excluded from the action allowable range. Thus, the prescribed time is set in accordance with the work content.

[0040] The target determination section 303 of the present embodiment calculates a difference between the reference motion information and the inspection target information in a state in which the reference motion information and the inspection target information are synchronized. Specifically, the target determination section 303 makes the start time of the repetitive motion indicated by the reference motion information and the inspection target information coincide by setting a timing at which the skeleton information of the prescribed position of the worker's body included in the inspection target information and the skeleton information of the prescribed position of the worker's body included in the reference motion information coincide within a certain range (for example, a range in which the positional displacement is within several centimeters) as the reference time. Then, the target determination section 303 calculates a difference between the reference motion information and the inspection target information in the repetitive motion of at least one cycle after the reference time. The difference can be, for example, the maximum positional displacement of the skeleton information of the prescribed position of the worker's body included in the reference motion information and the inspection target information when the start time of the motion is made to coincide, the maximum positional displacement of the track outlined by the skeleton information of the prescribed position of the worker's body included in the reference motion information and the inspection target information when the start time of the motion is made to coincide, or the displacement of the period of the motion between the reference motion information and the inspection target information.

[0041] [Action of skeleton detection system]

[0042] In the action of the skeleton detection system 1, first, the target determination processing in which the operation section 30 determines a target from among the skeleton information of the plurality of workers W in the work dynamic image is performed. In the target determination processing of the present embodiment, first, the skeleton information detection section 301 detects the skeleton information of the worker W from the work dynamic image. Thereafter, the candidate selection section 302 selects a candidate from among the detected skeleton information, and screens the number of workers W who are the inspection targets. Next, the target determination section 303 determines a target of the inspection target from among the selected candidates. Thus, the target determination processing is performed. As long as there is no instruction of the inspection suspension from the inspector, the operation section 30 repeatedly executes the flow of the present processing.

[0043] In this way, according to the skeleton detection system 1 of the present embodiment, it is possible to determine a target from among the skeleton information of the worker W detected from the work dynamic image. That is, by comparing the inspection target information of the worker W with the reference motion information of the worker W stored in the first storage section 31, it is possible to determine a target. In particular, the skeleton detection system 1 sets the repetitive motion of the worker W as the inspection target, and the motion of the worker W who is the inspection target has regularity and periodicity. Therefore, it is possible to accurately and quickly perform the determination of the target using the skeleton detection system 1. Furthermore, by the target determination section 303 determining a target, it is possible to determine the worker W who is the inspection target from among the workers W photographed in the work dynamic image. Therefore, it is possible to suppress the load of the skeleton detection system 1.

[0044] Further, the first storage section 31 stores the reference motion information of the worker W obtained from the reference dynamic image. Therefore, the operation section 30 can utilize the reference motion information at any time by referring to the first storage section 31. Therefore, the operation section 30 can efficiently perform the check of the repetitive motion of the target by referring to the reference motion information at the timing requested by the operation section 30.

[0045] Further, the skeleton detection system 1 of the present embodiment is provided with a second storage section 32 that stores the check target information. Therefore, the operation section 30 can utilize the check target information at any time by referring to the second storage section 32. Therefore, for example, even after the worker W performs the repetitive motion, the check of the repetitive motion of the target can be performed a posteriori by the operation section 30 by referring to the check target information at the timing requested by the checker or the operation section 30.

[0046] Further, the target determination section 303 of the present embodiment determines the skeleton information of the worker W that represents the motion satisfying the target necessary condition as the target. In this way, by using the target necessary condition, the target determination section 303 can stably determine the skeleton information of the worker W that should be determined as the target.

[0047] Further, as one example, the skeleton information includes at least any one of the position of the plurality of parts P of the worker W or the length of the line L connecting two or more of the plurality of parts P of the worker W or the skeleton shape formed by these plurality of parts P and the line L. In this way, the skeleton information is body information that simplifies the body of the worker W to the extent that the repetitive motion can be grasped, and therefore, for example, the amount of information is less than the image data of the worker W itself. Therefore, the operation section 30 can perform the check of the repetitive motion of the target with high accuracy while reducing the work burden based on the skeleton information including such data.

[0048] Hereinafter, the specific content of the target determination processing of the present embodiment will be exemplified. Figure 4 is a flowchart of the target determination processing of the skeleton detection system 1. This Figure 4 The flowchart shown in FIG. 13 is the target determination processing of the present embodiment. In this flowchart, the operation section 30 first determines whether the reference motion information of the work content is stored in the first storage section 31 in the skeleton detection system 1 (step S11). In step S11, the operation section 30 advances the step to step S13 after acquiring and storing the necessary reference motion information in the first storage section 31 (step S12) in a case where it is determined that the reference motion information of the work content is not stored in the first storage section 31 (step S11: No). In step S11, the operation section 30 advances the step to step S13 in a case where it is determined that the reference motion information of the work content is stored in the first storage section 31 (step S11: Yes).

[0049] In step S12, the operation section 30 can notify the examiner of the fact that the reference motion information of the work content is insufficient, for example, by the output section 33, and prompt the examiner to input data of a reference dynamic image including necessary reference motion information to the detection device 3. In the operation section 30, when it is sensed that the examiner has input data of a reference dynamic image to the detection device 3, the skeleton information detection section 301 can detect skeleton information from the data of the reference dynamic image as the reference motion information of the worker W.

[0050] Next, the skeleton information detection section 301 detects skeleton information of the worker W from the work-in-progress dynamic image that is captured to the worker W in the work (step S13). Thereafter, the operation section 30 counts the number of the workers W captured in the work-in-progress dynamic image on the basis of the skeleton information detected by the skeleton information detection section 301 (step S14).

[0051] Thereafter, the candidate selection section 302 selects a candidate that satisfies the candidate necessary condition from among the skeleton information of the worker W detected from the work-in-progress dynamic image (step S15). Thus, the skeleton information to be the examination target can be further filtered. As one example, the candidate necessary condition in the present flow includes a condition that the skeleton information is located within a prescribed region of the captured range shown by the work-in-progress dynamic image. The prescribed region can be appropriately set, for example, to a work region in which the worker W performs the work (as one example, a region including a work machine and its vicinity). Alternatively, the prescribed region can be set, for example, to a central region of the work-in-progress dynamic image. Thus, the skeleton information of the worker W located within the prescribed region among the workers W captured in the work-in-progress dynamic image is selected as a candidate.

[0052] Next, the target determination section 303 determines whether the candidate selected in step S15 satisfies the target necessary condition (S16). In step S16, the target determination section 303 returns the step to step S13 in a case where it is determined that the candidate does not satisfy the target necessary condition (step S16: No). In step S16, the target determination section 303 advances the step to step S17 in a case where it is determined that the candidate satisfies the target necessary condition (step S16: Yes).

[0053] In step S17, the target determination section 303 determines the candidate as the target if the number of candidates that satisfy the target necessary condition is one in step S16. In step S17, the target determination section 303 determines the candidate in which the offset of the motion allowable range of the target necessary condition is the smallest as the target if the number of candidates that satisfy the target necessary condition is plural in step S16. As one example, the operation section 30 stores the result of the determination of the target in the first storage section 31. The operation section 30 ends the present flow after step S17 is implemented.

[0054] Here, Figure 5 (a)~ Figure 5 (d) is a diagram schematically illustrating the situation in the target determination process of the first embodiment up to the point where the target is determined based on the dynamic image during the operation. Figure 5 In the example shown, in step S13, the skeleton information detection unit 301 detects the skeleton information of four workers W1 to W4 from the work screen. Figure 5 (a)). Furthermore, in step S15, the candidate selection unit 302 selects the skeleton information of three operators W1 to W3 located within a designated area S of the dynamic image during the operation as candidates. Figure 5 (b) Therefore, skeleton information located outside the specified area S (in this case, the skeleton information of operator W4) in the skeleton information of the shooting area of ​​the dynamic image during the operation is not considered as a candidate and is removed from the candidate.

[0055] Furthermore, in step S17, the target determination unit 303 determines, based on the skeleton information of candidate operators W1 to W3, whether the following target necessity condition is met: the difference between the baseline motion information set for each job content and the inspection object information, which is the motion information in the job, is within a pre-set allowable motion range. As an example, the inspection object information of each operator is compared with the baseline motion information in the order of W3, W2, and W1, and the operator who meets the target necessity condition (e.g., W1) is determined as the target. Furthermore, the skeleton information of operators W2 and W3, which are determined not to meet the target necessity condition, is not considered as the target object and is removed from the target list. Figure 5 (c) Figure 5 (d)).

[0056] According to this embodiment, all reference motion information required for the operation W's motion inspection is stored in the first storage unit 31. Therefore, in this process, information to be designated as an object is extracted from all the reference motion information stored in the first storage unit 31, and the target is determined by comparing the skeleton information detected based on the dynamic image during the operation with the reference motion information. Therefore, a thorough inspection can be performed. Furthermore, the candidate selection unit 302 selects candidates based on candidate necessity conditions including the condition that the candidate is located within a predetermined area S of the shooting range shown by the dynamic image during the operation. Therefore, the load on the skeleton detection system 1 when selecting candidates can be reduced. Therefore, the target can be determined quickly.

[0057] The following describes a modification of the present embodiment. As one example, the candidate necessary condition of the first modification includes the following necessary condition: the difference between the reference motion information and the inspection target information according to the skeleton information detected from the in-work dynamic image is within a pre-set motion allowable range. In this way, in the present modification, the candidate is selected by the candidate selection section 302 based on the repetitive motion of the skeleton information within the in-work dynamic image. Thus, according to the present modification, the skeleton information of a more appropriate repetitive motion is selected as a candidate before the target determination section 303 determines the target. Therefore, the candidate selection section 302 can select an appropriate candidate with high precision, and the burden on the target determination section 303 when determining the target can be reduced.

[0058] Further, the worker W of the inspection target sometimes performs work at a position closest to the imaging device 2 among the plurality of workers W, for example. In this case, the worker W of the inspection target is photographed the most among the plurality of workers W within the in-work dynamic image. Therefore, as one example, the candidate necessary condition of the second modification includes the following condition: the size of the skeleton represented by the skeleton information detected from the in-work dynamic image is the largest among the skeleton information of the plurality of workers W detected from the in-work dynamic image. The size of the skeleton represented by the skeleton information is expressed by, for example, any one of the following: the area of a polygon having three or more positions of the body of the worker W detected from the in-work dynamic image as vertices (for example, the area of a triangle having the positions PI to P3 as vertices shown in Figure 6 Figure 2 and Figure 3 The above-described polygon can have a shape having four or more vertices. Further, the sum of the area of the polygon and the skeleton length can be combined. By being based on such a candidate necessary condition, the worker W photographed the most in the in-work dynamic image is selected as a candidate.

[0059] Figure 6 is a diagram showing position information of the front side of the worker W included in the skeleton information of the third modification. Figure 7 is a diagram showing position information of the side surface side of the worker W included in the skeleton information of the third modification. In Figure 6 and Figure 7 , the skeleton information displayed in the display surface 331 of the output section 33 is schematically shown.

[0060] As Figure 6 and Figure 7 ​As shown, the target determination section 303 of the third modification example determines the target based on the motion represented by the skeleton information expressed by the respective changes of the first position Pl and the second position P2 (the respective positions of the right shoulder and the left shoulder) separated on both sides across the center line X of the body of the worker W detected in the dynamic image during the work, and the third position P3 (the central position of the head) separated from the first position Pl and the second position P2.

[0061] According to the present modification example, the target determination section 303 determines the target based on the skeleton information including less position information, whereby the target determination section 303 can quickly determine the target, and can further reduce the burden on the skeleton detection system 1 at the time of determining the target, for example, as compared with the case of using the skeleton information of the entire body of the worker W.

[0062] Further, in the present modification example, the position data of the first to third positions Pl to P3 is included in the inspection target information. Thereby, the target determination section 303 can quickly determine the target from among the candidates based on the data of the first to third positions Pl to P3. Note that in the first modification example, the candidate selection section 302 can select the candidate based on the motion represented by the skeleton information expressed by the respective changes of the first to third positions Pl to P3, at the time of selecting the candidate by the candidate selection section 302.

[0063] Figure 8 is a flowchart showing the overall operation of the skeleton detection system of the fourth modification example. The skeleton detection system of the present modification example performs the target determination processing (step S1) and then performs the target inspection processing of the repeated motion of the inspection target (step S2). After step S2, the operation section 30 determines whether there is an instruction of the inspection suspension from the inspector (step S3). In the skeleton detection system of the present modification example, as long as the operation section 30 determines that there is no instruction of the inspection suspension from the inspector in step S3 (step S3: No), the flow of steps S1 to S3 is repeatedly executed. Note that the skeleton detection system of the present modification example can be provided with a recording section that records the target inspection result. In this case, the operation section 30 records the target inspection result in the recording section.

[0064] As one example, the skeleton detection system of the present modification performs the target determination processing (step S1) and the target inspection processing (step S2) in real time based on the in-process dynamic image of the work. Thereby, the inspector can use the skeleton detection system to inspect the repetitive action of the work of the worker W in real time and promptly. Therefore, the inspection result can be immediately reflected in the repetitive action of the current worker W. Further, since the target inspection processing (step S2) is performed in real time, if a device that analyzes the inspection result of the inspection processing is used, the inspector can easily determine the assembly product even in a case where the assembly product that does not reach the quality criterion is assembled due to the inappropriate action of the worker W, for example. Thereby, the waste of the assembly product can be reduced. In this way, according to the present modification, various work analyses can be performed using the inspection result of the target inspection processing (step S2).

[0065] In the skeleton detection system, in a case where the target inspection processing is performed in real time, the operation section 30 can not store the in-process dynamic image of the work captured by the imaging device 2 in the second storage section 32, for example, and can perform the target determination processing (step S1) and the target inspection processing (step S2). Alternatively, the target determination processing (step S1) and the target inspection processing (step S2) can be performed while the in-process dynamic image of the work captured by the imaging device 2 is stored in the second storage section 32. Further, at least one of the reference dynamic image and the in-process dynamic image can be recorded in the storage section (for example, at least one of the first storage section 31 and the second storage section 32) provided in the skeleton detection system.

[0066] Next, the specific content of the target inspection processing (step S2) is exemplified. In the present processing, as one example, the operation section 30 performs the action inspection of the repetitive action in the main action based on the skeleton information within the in-process dynamic image with respect to the target (for example, W1) determined in step S17. The action inspection is performed by the operation section 30 determining whether or not the difference between the reference action information and the inspection target information is within a predetermined action allowable range, for example. The action allowable range of the target inspection necessary condition can be set to a narrower range than the action allowable range of the target necessary condition, but is not limited thereto.

[0067] According to the above configuration, the inspector can appropriately and promptly perform the action inspection of the repetitive action of the target based on the notified inspection result of the target. Hereinafter, the second embodiment is described focusing on the difference from the first embodiment.

[0068] (Second Embodiment)

[0069] The second embodiment will be described below. In the second embodiment, a device is shown that performs the same processing as the target inspection processing of the fourth modified example of the first embodiment described above, and based on the inspection result of the processing, performs processing that determines whether or not the action of the worker corresponding to the target is qualified.

[0070] Figure 9 is a functional block diagram of the work management device 100 of the second embodiment. As shown in Figure 9 , the work management device 100 is provided with the photographing device 2 and the determination device 4 connected with the photographing device 2. The determination device 4 has the arithmetic unit 40, the first storage unit 41, the second storage unit 42, and the output unit 43. The arithmetic unit 40 has the determination unit 404 in addition to the skeleton information detection unit 401, the candidate selection unit 402, and the target determination unit 403. This determination unit 404 performs the pass / fail determination processing that determines whether or not the action of the worker corresponding to the target is qualified based on whether or not the difference between the inspection target information of the target and the reference action information corresponding to the target is within the reference range set in advance at a prescribed timing. The output unit 43 outputs the determination result of the pass / fail determination processing performed by the determination unit 404. In this way, the work management device 100 has a configuration that substantially has the skeleton detection system 1 of the first embodiment and the determination unit 404.

[0071] Figure 10 is a flowchart showing the overall action of the work management device 100 of Figure 9 . The overall action of the work management device 100 will be described below using Figure 10 . As shown in Figure 10 , when the work management device 100 acts, first, the arithmetic unit 40 performs the target determination processing as in step S1 (step S4). After that, the determination unit 404 determines whether or not the difference between the inspection target information of the target and the reference action information corresponding to the target is within the reference range set in advance. By this, the pass / fail determination processing is performed (step S5).

[0072] Next, the arithmetic unit 40 determines whether or not there is an instruction of inspection suspension from the inspector (step S6). In step S6, the arithmetic unit 40 suspends the execution of the present flow in a case where it is determined that there is an instruction of inspection suspension from the inspector (step S6: Yes). In step S6, the arithmetic unit 40 returns to step S4 in a case where it is determined that there is no instruction of inspection suspension from the inspector (step S6: No). By this, the arithmetic unit 40 repeatedly executes the present flow as long as there is no instruction of inspection suspension from the inspector. The above is the overall action of the work management device 100.

[0073] The specific processing content in the pass / fail determination processing (step S5) of Figure 10 will be exemplified below.Figure 11 is about Figure 10 the pass / fail determination process. In the present subflow, the operation section 40 performs the action check of the determined target repetitive action (step S51). In the action check, as described above, the determination section 404 determines whether the difference between the check target information of the target and the reference action information corresponding to the target is within the reference range (as one example, the action allowance range of the target check necessary condition described above).

[0074] Next, the operation section 40 determines whether the action of the worker corresponding to the target is a pass or a fail based on whether the difference determined in the check result of step S51 is within the reference range (step S52). In step S52, in a case where the action of the worker corresponding to the target is determined to be a fail with respect to the reference due to the difference in the check result of step S51 not being within the reference range (here, the target check necessary condition is not satisfied) (step S52: No), the determination section 40 outputs a determination result indicating this to the output section 43 (step S54), and ends the execution of the present flow. In step S52, in a case where the action of the worker corresponding to the target is determined to be a pass with respect to the reference due to the difference in the check result of step S51 being within the reference range (here, the target check necessary condition is satisfied) (step S52: Yes), the determination section 40 outputs a determination result indicating this to the output section 43 (step S53), and ends the execution of the present flow.

[0075] Even with the work management apparatus 100 performing such an action, the same effects as the skeleton detection system 1 of the first embodiment can be achieved. Furthermore, since the operation section 40 has the determination section 404, the checker can confirm the determination result of the determination section 404 with respect to the repetitive action indicated by the target by the content output from the output section 43. Thus, the checker can be relieved of the burden of personally performing the action check. Furthermore, the checker can stably and highly accurately check the repetitive action of the worker W even over a long period of time.

[0076] Note that the output section 43 can be a sound output section. In this case, it can be that the operation section 40 outputs a warning sound to the output section 43 in a case where the determination section 404 determines that the difference in the check result of step S51 is not within the reference range (step S52: No), in step S54. Furthermore, the work management apparatus 100 can also have a recording section that records the determination result of the determination section 404. In this case, the operation section 40 records the determination result of the determination section 404 obtained in step S52 in the recording section.

[0077] Each of the configurations and combinations thereof in the embodiments is one example, and additional, omission, substitution, and other changes of the configurations can be appropriately made without departing from the scope of the gist of the present disclosure. The present disclosure is not limited by the embodiments, but is defined only by the claims. Furthermore, each of the various aspects disclosed in the present specification can be combined with any of the other features disclosed in the present specification.

[0078] The skeleton detection system 1 and the work management apparatus 100 can be provided with a plurality of imaging devices 2. In this case, the skeleton detection system 1 and the work management apparatus 100 can check, for example, repetitive movements of the worker W at a plurality of sites.

[0079] Further, in the first embodiment, an example in which a plurality of candidates are sequentially determined to determine the target is shown, but a plurality of candidates can be simultaneously determined to determine the target. In this case, the skeleton detection system 1 can have, for example, a plurality of arithmetic units 30.

[0080] Explanation of Reference Numerals

[0081] W, W1 to W4: worker

[0082] 1: skeleton detection system

[0083] 31, 41: first storage

[0084] 32, 42: second storage

[0085] 43: output

[0086] 100: work management apparatus

[0087] 301, 401: skeleton information detection

[0088] 302, 402: candidate selection

[0089] 303, 403: target determination

[0090] 404: determination

Claims

1. A skeleton detection system, the skeleton detection system comprising: The first storage unit stores reference motion information, which relates to the amount of at least one cycle of a reference motion that is a repetitive motion performed by an operator. The skeleton information detection unit detects the skeleton information of the worker based on the captured dynamic images of the worker during the operation. as well as The target determination unit, while synchronizing the reference motion information and the inspection object information which is motion information in the operation, calculates the difference between the reference motion information and the inspection object information, and determines the skeleton information of the operator that represents the motion that meets the necessary conditions of the target among the skeleton information of multiple operators in the operation detected by the skeleton information detection unit as the target. The necessary conditions of the target refer to the difference being within a pre-set allowable range of motion.

2. The skeleton detection system according to claim 1, wherein, The first storage unit stores the reference motion information of the operator obtained from the reference motion video of the operator's reference motion captured by the camera.

3. The skeleton detection system according to claim 2, further comprising: The second storage unit stores the information of the object being inspected.

4. The skeleton detection system according to any one of claims 1 to 3, wherein the skeleton detection system further comprises, The candidate selection unit selects candidates that meet predetermined candidate necessity conditions from the skeleton information of the plurality of operators detected based on dynamic images during the operation. The target determination unit determines the skeleton information of the operator that represents an action that meets the necessary conditions for the target as the target. The necessary conditions for the target refer to the difference between the baseline action information and the inspection object information of the operator selected by the candidate selection unit being within a pre-set action allowable range.

5. The skeleton detection system according to claim 4, wherein, The candidate necessary conditions include the following condition: among the skeleton information of the plurality of operators detected based on the dynamic images in the operation, the size of the skeleton represented by the skeleton information detected based on the dynamic images in the operation is the largest.

6. The skeleton detection system according to claim 5, wherein, The size of the skeleton represented by the skeleton information can be expressed by either of the following: the area of ​​a polygon with three or more points on the operator's body detected from the dynamic image of the operation as vertices, and the sum of the skeleton length of the operator detected from the dynamic image of the operation.

7. The skeleton detection system according to claim 4, wherein, The candidate necessary conditions include the following: the skeleton information is located within a specified area of ​​the shooting range shown by the dynamic image in the operation.

8. The skeleton detection system according to any one of claims 1 to 3, wherein, The target determination unit determines the target based on the action represented by the skeletal information shown by various changes in the first position, the second position, and the third position. The first and second positions are separated on both sides by the center line of the operator's body detected according to the dynamic image during the operation, and the third position is separated from the first and second positions of the body, respectively.

9. The skeleton detection system according to claim 8, wherein, The position data of the first to third positions of the dynamic image in the operation are included in the inspection object information.

10. The skeleton detection system according to any one of claims 1 to 3, wherein, The skeleton information includes data representing the position of at least one of multiple parts of the operator or lines connecting two or more of the multiple parts of the operator.

11. A job management device, the job management device comprising: The skeleton detection system according to any one of claims 1 to 10; The judgment unit performs a pass / fail judgment process based on whether the difference between the inspection object information represented by the target and the reference action information corresponding to the target is within a preset reference range; and The output unit outputs the determination result of the determination unit.

Images