Task recognition device, task recognition method, and task recognition program

Abstract

To enable improvement of accuracy in work recognition.SOLUTION: A work recognition apparatus according to the present invention acquires a captured image obtained by capturing work of an operator, detects first detection information relating to at least one of a right hand and a left hand of the operator and an object of the work based on the captured image, detects second detection information relating to a skeleton of the operator based on the captured image, executes first recognition processing for recognition of the work based on the first detection information and the second detection information detected previously, executes second recognition processing for recognition of the work based on the detected second detection information, determines whether or not a switch condition for use in switching between the first recognition processing and the second recognition processing for recognition of the work is satisfied based on the first detection information, outputs a work recognition result of a first recognition unit if the switch condition is not satisfied, and outputs a work recognition result of a second recognition unit if the switch condition is satisfied.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The disclosed technology relates to an activity recognition device, an activity recognition method, and an activity recognition program. [Background technology]

[0002] Patent Document 1 discloses a human behavior inference system that discriminates behaviors performed by a person using a tool, the human behavior inference system having: a video acquisition unit that acquires video of the behavior; a human behavior discrimination unit that outputs human behavior candidates for the behavior captured in the video, based on the video from the video acquisition unit, from a predetermined human behavior definition; a tool data acquisition unit that acquires sensor information from a sensor attached to the tool; a tool action discrimination unit that outputs tool action candidates for the tool for which the sensor information was acquired, based on the sensor information from the tool data acquisition unit, from a predetermined tool action definition; and a comprehensive human behavior discrimination unit that infers the behavior captured in the video from the video acquisition unit, based on the human behavior candidates output from the human behavior discrimination unit and the tool action candidates output from the tool action discrimination unit.

[0003] Patent Document 2 discloses a task recognition device including a sensor data acquisition unit that acquires sensor data, a body part information acquisition unit that detects a body part of a worker based on the sensor data acquired by the sensor data acquisition unit and acquires body part information related to the body part of the worker, an object information acquisition unit that detects an object based on the sensor data acquired by the sensor data acquisition unit and acquires object information related to the object, an association unit that associates the object with a body part of the worker who performed a task using the object based on the body part information acquired by the body part information acquisition unit and the object information acquired by the object information acquisition unit, and a recognition result analysis unit that recognizes the task performed by the worker based on association information related to the association results associated by the association unit. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-12328 [Patent Document 2] Patent No. 6444573 Summary of the Invention [Problem to be solved by the invention]

[0005] In the technology described in Patent Document 1, when recognizing human actions through video analysis, blind spots often occur in the video, making it difficult to accurately estimate a person's actions. Therefore, in addition to estimating actions based on people's position information, the technology recognizes work comprehensively using predefined human action information and tool action information.

[0006] Furthermore, the technology described in Patent Document 2 recognizes the work of a worker by using the coordinates of the worker's body parts and the position coordinates of the object from sensor data, linking the body parts with the object parts, and recognizing the type of work. If the object cannot be detected, processing such as complementation is performed, and the system tries to match the action and object as much as possible.

[0007] However, in an actual work site, there are many parts, and even if parts are detected, they may be erroneously detected, making it difficult to improve the accuracy of work recognition.

[0008] The disclosed technology has been made in consideration of the above points, and aims to provide an activity recognition device, an activity recognition method, and an activity recognition program that can improve the accuracy of activity recognition. [Means for solving the problem]

[0009] A first aspect of the disclosure is a task recognition device comprising: an acquisition unit that acquires a captured image of a worker performing a task; a first detection unit that detects, based on the captured image, first detection information related to at least one of the worker's right hand and left hand and an object of the task; a second detection unit that detects, based on the captured image, second detection information related to the worker's skeleton; a first recognition unit that executes a first recognition process to recognize the task based on the detected first detection information and second detection information; a second recognition unit that executes a second recognition process to recognize the task based on the detected second detection information; a determination unit that determines, based on the first detection information, whether a switching condition is satisfied for switching between recognizing the task using the first recognition unit and recognizing the task using the second recognition unit; and an output unit that outputs the task recognition result by the first recognition unit if the switching condition is not satisfied, and outputs the task recognition result by the second recognition unit if the switching condition is satisfied.

[0010] In the first aspect, the determination unit may determine that the switching condition is satisfied when the distance between the hand and the object is equal to or greater than a predetermined threshold.

[0011] In the first aspect, the determination unit may determine that the switching condition is satisfied when the distance between the right hand and the left hand is equal to or greater than a predetermined threshold.

[0012] In the first aspect, the determination unit may determine that the switching condition is satisfied when a bounding box of the hand and a bounding box of the object do not overlap.

[0013] In the first aspect, the determination unit may determine whether the switching condition is satisfied based on a size of the detected object.

[0014] In the first aspect, the determination unit may determine that the switching condition is satisfied when a size of a bounding box of the detected object is smaller than a predetermined minimum size of the object.

[0015] In the first aspect, the determination unit may determine that the switching condition is satisfied when a size of a bounding box of the detected object is larger than a predetermined maximum size of the object.

[0016] In the first aspect, the first detection unit may calculate a reliability of the detected object, and the determination unit may determine that the switching condition is satisfied when the reliability of the object is equal to or less than a predetermined threshold.

[0017] In the first aspect, the determination unit may determine that the switching condition is satisfied when the detected speed of at least one of the hand and the object is equal to or greater than a predetermined threshold.

[0018] In the first aspect, the determination unit may determine that the switching condition is satisfied when the detected position of the hand is outside a range of a predetermined working space.

[0019] A second aspect of the disclosure is a task recognition method in which a computer executes a process including: acquiring a captured image of a worker performing a task; detecting, based on the captured image, first detection information relating to at least one of the worker's right hand and left hand and an object of the task; detecting, based on the captured image, second detection information relating to the worker's skeleton; performing a first recognition process to recognize the task based on the detected first detection information and the second detection information; performing a second recognition process to recognize the task based on the detected second detection information; determining, based on the first detection information, whether a switching condition is met for switching between recognizing the task using the first recognition process or the second recognition process; outputting a recognition result of the task by the first recognition process if the switching condition is not met; and outputting a recognition result of the task by the second recognition process if the switching condition is met.

[0020] A third aspect of the disclosure is a task recognition program that causes a computer to execute a process including: acquiring a captured image of a worker performing a task; detecting, based on the captured image, first detection information relating to at least one of the worker's right hand and left hand and an object of the task; detecting, based on the captured image, second detection information relating to the worker's skeleton; executing a first recognition process to recognize the task based on the detected first detection information and the second detection information; executing a second recognition process to recognize the task based on the detected second detection information; determining, based on the first detection information, whether a switching condition is satisfied for switching between recognizing the task by the first recognition process and recognizing the task by the second recognition process; if the switching condition is not satisfied, outputting a recognition result of the task by the first recognition process; and if the switching condition is satisfied, outputting a recognition result of the task by the second recognition process. [Effects of the Invention]

[0021] According to the disclosed technology, it is possible to improve the accuracy of task recognition. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a configuration diagram of a task recognition system. [Figure 2] FIG. 2 is a configuration diagram illustrating a hardware configuration of the task recognition device. [Figure 3] FIG. 2 is a functional block diagram of the task recognition device. [Figure 4] FIG. 2 is a diagram for explaining first detection information detected based on a captured image of a task. [Figure 5] FIG. 10 is a diagram for explaining second detection information detected based on a captured image of a task. [Figure 6] FIG. 10 is a diagram showing a recognition result of a task. [Figure 7] 10 is a flowchart of an activity recognition process. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of the present invention will be described below with reference to the drawings. The same or equivalent components and parts in each drawing are designated by the same reference numerals. The dimensional proportions of the drawings may be exaggerated for the sake of explanation and may differ from the actual proportions.

[0024] 1 shows the configuration of an activity recognition system 10. The activity recognition system 10 includes an activity recognition device 20 and a camera 30.

[0025] The task recognition device 20 recognizes the task content of the task performed by the worker W based on the image captured by the camera 30.

[0026] As an example, a worker W takes out a work object M placed on a workbench T and performs a predetermined task in a work space S. The workbench T is installed in a place with sufficient brightness to allow the worker's actions to be recognized.

[0027] The camera 30 captures, for example, RGB color images. The camera 30 is installed in a position where the work being done by the worker W can be easily recognized. Specifically, the camera 30 is installed in a position that satisfies conditions such as a position where an area including at least the work space S is not hidden by other objects, a position where the work being done by the worker W is not hidden by the workbench T, and a position where the finger movements of the worker W while working are not hidden by other objects. In this embodiment, as an example, a case will be described where the camera 30 is installed in a position where it can look down on at least the upper body of the worker W from diagonally above.

[0028] In this embodiment, a case where one camera 30 is provided will be described, but a configuration may be adopted in which multiple cameras 30 are provided. In addition, in this embodiment, a case where there is one worker W will be described, but there may be two or more workers W.

[0029] Fig. 2 is a block diagram showing the hardware configuration of the task recognition device 20 according to this embodiment. As shown in Fig. 2, the task recognition device 20 includes a controller 21. The controller 21 is configured as a device including a general computer.

[0030] 2, the controller 21 includes a central processing unit (CPU) 21A, a read-only memory (ROM) 21B, a random access memory (RAM) 21C, and an input / output interface (I / O) 21D. The CPU 21A, the ROM 21B, the RAM 21C, and the I / O 21D are connected to each other via a bus 21E. The bus 21E includes a control bus, an address bus, and a data bus.

[0031] Furthermore, an operation unit 22, a display unit 23, a communication unit 24, and a storage unit 25 are connected to the I / O 21D.

[0032] The operation unit 22 includes, for example, a mouse and a keyboard.

[0033] The display unit 23 is configured by, for example, a liquid crystal display.

[0034] The communication unit 24 is an interface for performing data communication with an external device such as a camera 30 .

[0035] The storage unit 25 is configured as a non-volatile external storage device such as a hard disk, etc. As shown in Fig. 2, the storage unit 25 stores an activity recognition program 25A and the like.

[0036] The CPU 21A is an example of a computer. The term "computer" here refers to a processor in a broad sense, and includes a general-purpose processor (e.g., a CPU) or a dedicated processor (e.g., a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, etc.).

[0037] The task recognition program 25A may be realized by being stored in a non-volatile, non-transitory recording medium or distributed via a network and installed in the task recognition device 20 as appropriate.

[0038] Examples of non-volatile non-transitory recording media include CD-ROMs (Compact Disc Read Only Memory), magneto-optical disks, HDDs (Hard Disk Drives), DVD-ROMs (Digital Versatile Disc Read Only Memory), flash memory, memory cards, etc.

[0039] Fig. 3 is a block diagram showing the functional configuration of the CPU 21A of the task recognition device 20. As shown in Fig. 3, the CPU 21A functionally includes an acquisition unit 40, a first detection unit 41, a second detection unit 42, a first recognition unit 43, a second recognition unit 44, a determination unit 45, and an output unit 46. The CPU 21A functions as each functional unit by reading and executing a task recognition program 25A stored in the storage unit 25.

[0040] The acquisition unit 40 acquires from the camera 30 a photographed image of the worker W working.

[0041] The first detector 41 detects first detection information related to at least one of the worker W's right and left hands and the work object M based on the captured image acquired from the camera 30. Specifically, the first detection information includes at least one of the coordinates of the four corners of a bounding box representing an area including at least one of the worker W's right and left hands and the coordinates of the four corners of a bounding box representing an area of ​​the work object M in contact with at least one of the right and left hands. Here, the bounding box refers to a rectangular shape, such as a rectangle or a square, that circumscribes the object to be detected. Specifically, the reliability of the object to be detected is calculated for each of anchor boxes (rectangular regions) of various sizes. Then, the coordinates of the four corners of the anchor box with the highest reliability are set as the coordinates of the four corners of the bounding box. Such a bounding box detection method can be a known method, such as Faster R-CNN (Regions with Convolutional Neural Networks), or the method described in Reference 1 below.

[0042] (Reference 1) "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.

[0043] Fig. 4 shows a captured image G1 captured from above the worker W, showing the worker W handling an object M with his right hand RH in a workspace S. In this case, as shown in Fig. 4, the first detection unit 41 detects the coordinates of the four corners of a bounding box BR that surrounds the right hand RH, the coordinates of the four corners of a bounding box BL that surrounds the left hand LH, and the coordinates of the four corners of a bounding box BM that surrounds the object M. In the following, when there is no need to distinguish between the right hand RH and the left hand LH, they will simply be referred to as hand H.

[0044] As a method for detecting the first detection information based on the captured image, a first detection trained model can be used, which is a learning model that uses the captured image as input and the first detection information as output, trained using a large number of captured images as training data. As a training method for obtaining such a first detection trained model, a known method such as CNN can be used, and for example, the method described in Reference 2 below can be used.

[0045] (Reference 2) "Understanding Human Hands in Contact at Internet Scale", pp.9869-9878, Dandan Shan1, Jiaqi Geng, Michelle Shu, David F. Fouhey, University of Michigan, Johns Hopkins University, CVPR2020.

[0046] By detecting such first detection information in time series, it becomes possible to understand what kind of work the worker W is performing on the object M using his / her hand H.

[0047] The second detection unit 42 detects second detection information related to the skeleton of the worker W based on the captured image acquired from the camera 30. Specifically, the second detection information includes coordinates of feature points such as body parts and joints of the worker W, and link information that defines links connecting each feature point. For example, the feature points include facial parts of the worker W, such as the eyes and nose, and joints such as the neck, shoulders, elbows, wrists, waist, knees, and ankles.

[0048] An example of the detected second detection information is shown in Fig. 5. In the example of Fig. 5, the second detection information includes the coordinates of feature points P1 to P11 detected across the right hand, right arm, right shoulder, head, left shoulder, left arm, and left hand of the worker W, and link information that defines links L1 to L10 connecting the feature points.

[0049] As a method for detecting the second detection information based on the captured image, a second detection trained model can be used, which is a learning model that uses the captured image as input and the second detection information as output, trained using a large number of captured images as training data. As a training method for obtaining such a second detection trained model, a known method such as CNN (Regions with Convolutional Neural Networks) can be used, and for example, the method described in Reference 3 below can be used.

[0050] (Reference 3) "OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", Zhe Cao, Student Member, IEEE, Gines Hidalgo, Student Member, IEEE, Tomas Simon, Shih-En Wei, and Yaser Sheikh, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE.

[0051] By detecting such second detection information in time series, it becomes possible to understand what work the worker W is performing on the object M using his / her hand H.

[0052] The first recognition unit 43 executes a first recognition process for recognizing an operation based on the first detection information detected by the first detection unit 41 and the second detection information detected by the second detection unit 42. Specifically, the first recognition unit 43 recognizes an operation using a first recognition trained model that uses the first detection information and the second detection information as input and outputs an operation recognition result as training data.

[0053] The second recognition unit 44 executes a second recognition process for recognizing the work based on the second detection information detected by the second detection unit 42. Specifically, the second recognition unit 44 recognizes the work using a second recognition trained model that uses the second detection information as input and the work recognition result as output, and that has trained a large amount of the second detection information as training data.

[0054] Incidentally, the recognition results of the first recognition process, which recognizes an activity using both the first detection information and the second detection information, often have higher recognition accuracy than the recognition results of the second recognition process, which recognizes an activity based only on the second detection information. However, the recognition accuracy of the first recognition process may be reduced depending on the state of the worker W and the object M. For example, this may occur when the object M is small and difficult to detect, when the object M is hidden by the hand H of the worker W and cannot be accurately detected, when the entire body of the worker W is erroneously detected as the hand H, or when an object other than the object M is erroneously detected as the object M. If the hand H of the worker W or the object M is erroneously detected in this way, the recognition accuracy of the first recognition process will be reduced.

[0055] Specifically, for example, suppose that the work performed by worker W is packing work in which a product and an instruction manual are placed in a packing box, packed, and a label is attached to the box. In this case, the objects M are the box, the product, the instruction manual, and the label. If the packing work is divided into chronological order for each object M, it can be divided into work S1 handling the box, work S2 handling the product, work S3 handling the instruction manual, and work S4 handling the label. When such packing work is recognized by the first recognition process and the second recognition process, for example, the recognition accuracy of work S1 may be higher in the second recognition process than in the first recognition process, and conversely, the recognition accuracy of work S3 may be higher in the first recognition process than in the second recognition process.

[0056] Figure 6 shows an example of the recognition results for each task when the above-mentioned packing task is performed in two cycles. As shown in Figure 6, in the correct recognition results, there is no period in which each task, Task 1 to Task 4, is mistakenly recognized as another task. In contrast, in the recognition results of the first recognition process, there is a period in which task S1 in the second cycle is mistakenly recognized as task S3. Furthermore, in the recognition results of the second recognition process, there is a period in which task S3 in the first cycle is mistakenly recognized as task S4.

[0057] As described above, there are cases where a correct recognition result cannot be obtained by only the first recognition process by the first recognition unit 43 or by only the second recognition process by the second recognition unit 44.

[0058] Therefore, the determination unit 45 determines, based on the first detection information, whether a switching condition for switching between recognizing the work by the first recognition unit 43 and recognizing the work by the second recognition unit 44 is satisfied.

[0059] For example, the determination unit 45 determines that the first switching condition is satisfied when the distance between the detected hand H and the object M is equal to or greater than a predetermined threshold T1. Specifically, the determination unit 45 determines that the first switching condition is satisfied when, for example, at least one of the distance between the center position of the bounding box BR of the right hand RH and the center position of the bounding box BM of the object M and the distance between the center position of the bounding box BL of the left hand LH and the center position of the bounding box BM of the object M is equal to or greater than the predetermined threshold T1.

[0060] Here, if the coordinates of the center position C1 of the bounding box of the hand H are (x1, y1) and the coordinates of the center position C2 of the bounding box BM of the object M are (x2, y2), the distance D1 between the center positions C1 and C2 is calculated using the following formula.

[0061] D1=[(x1-x2) 2 +(y1-y2) 2 ] 1 / 2 ···(1)

[0062] The threshold T1 is set to a value (e.g., 10 cm or more) that can determine that there is a high possibility that at least one of the detected hand H and object M is a false detection when the distance D1 is equal to or greater than the threshold T1, and is set in advance based on, for example, experimental results. Note that when calculating the distance D1, the position of one of the four corners of the bounding box may be used instead of the center position of the bounding box.

[0063] Furthermore, the determination unit 45 may determine that the second switching condition is met when the distance D2 between the right hand RH and the left hand LH is equal to or greater than a predetermined threshold T2. The distance D2 is the distance between the center position of the bounding box BR of the right hand RH and the center position of the bounding box BL of the left hand LH, and can be calculated in the same manner as in equation (1) above. The threshold T2 is set in advance in the same manner as the threshold T1.

[0064] Furthermore, the determination unit 45 may determine that a third switching condition is satisfied when the bounding box of the hand H does not overlap with the bounding box of the object M. Specifically, for example, the determination unit 45 determines that the third switching condition is satisfied when at least one of the bounding box BR of the right hand RH and the bounding box BM of the object M does not overlap with the bounding box BM of the object M. This is because when the bounding boxes do not overlap with each other, the distance between the hand H and the object M is large, and it is highly likely that at least one of the hand H and the object M has been erroneously detected.

[0065] Furthermore, the determination unit 45 may determine whether or not the fourth switching condition is satisfied based on the size of the detected object M. Specifically, the determination unit 45 may determine that the fourth switching condition is satisfied when the size of the bounding box BM of the detected object M is smaller than the size of a predetermined smallest object M. Here, the size of the bounding box refers to, for example, the area of ​​the bounding box. In this way, when the size of the bounding box BR of the detected object M is smaller than the size of the smallest object M among the multiple objects M, it is highly likely that the detected object M is a false detection, and therefore the determination unit 45 determines that the fourth switching condition is satisfied.

[0066] Furthermore, as a fifth switching condition, the determination unit 45 may determine that the fifth switching condition is satisfied when the size of the bounding box of the detected object M is larger than the size of a predetermined largest object M (for example, when the size is 1.5 times or more the size of the largest object M). In this way, when the size of the bounding box BR of the detected object M is larger than the size of the largest object M among the multiple objects M, it is highly likely that the detected object M is a false detection, and therefore the determination unit 45 determines that the fifth switching condition is satisfied.

[0067] Furthermore, the determination unit 45 may determine that the sixth switching condition is satisfied when the reliability of the detected object is equal to or less than a predetermined threshold T3. In this case, the first detection unit 41 calculates the reliability of the detected object. As described above, the first detection unit 41 can detect the object using a first trained model for detection using a CNN or the like. The reliability of the detected object is calculated by using a so-called softmax function in the output layer of the first trained model for detection. The reliability is expressed, for example, as a numerical value ranging from 0 to 1, with a larger value indicating a higher reliability. Therefore, for example, if the threshold T3 is set to 0.5 and the reliability of the detected object is equal to or less than 0.5, it is determined that the sixth switching condition is satisfied, since it is highly likely that the object has been erroneously detected.

[0068] Furthermore, the determination unit 45 may determine that the seventh switching condition is satisfied when the speed of at least one of the detected hand H and object M is equal to or greater than a predetermined threshold T4. Specifically, the determination unit 45 determines that the seventh switching condition is satisfied when, for example, at least one of the speed of the center position of the bounding box BR of the right hand RH, the speed of the center position of the bounding box BL of the left hand LH, and the speed of the center position of the bounding box BM of the object M is equal to or greater than a predetermined threshold T4. The threshold T4 is set to a value (e.g., 1 m / s) at which it can be determined that there is a high possibility of erroneous detection when the speed is equal to or greater than the threshold T4, and is set in advance based on, for example, experimental results.

[0069] Furthermore, as an eighth switching condition, the determination unit 45 may determine that the eighth switching condition is satisfied when the detected position of the hand H is outside the range of a predetermined working space S.

[0070] For example, the coordinates of the center position of the bounding box BR of the right hand RH are (xr, yr), the coordinates of the center position of the bounding box BL of the left hand LH are (xl, yl), and the coordinates of the two corners on one of the two diagonals of the workspace S are (x1, y1) and (x2, y2).<x2、y1> In this case, if the following expressions (2) and (3) are satisfied, it can be determined that the center position of the bounding box BR of the right hand RH and the center position of the bounding box BL of the left hand LH are within the range of the workspace S.

[0071] x1≦xr, xl≦x2 (2) y1≦yr, yl≦y2 (3)

[0072] Therefore, if at least one of the above formulas (2) and (3) is not satisfied, it is assumed that the right hand RH and the left hand LH are outside the range of the working space S, and it is determined that the eighth switching condition is satisfied.

[0073] The first to eighth switching conditions may be appropriately combined, and it may be determined that the switching conditions are satisfied when two or more of the switching conditions are satisfied. In other words, it may be determined that the switching conditions are satisfied when at least one of the multiple switching conditions is satisfied.

[0074] If the switching condition is not satisfied, output unit 46 outputs the work recognition result obtained by first recognition unit 43, and if the switching condition is satisfied, output unit 46 outputs the work recognition result obtained by second recognition unit 44. The recognition result may be displayed by being output to display unit 23, or stored by being output to memory unit 25, for example.

[0075] In this way, the first recognition process by the first recognition unit 43 and the second recognition process by the second recognition unit 44 are switched depending on whether the switching condition is satisfied. For example, as shown in "Recognition result when switching" in FIG. 6, task S1 in the second cycle, indicated by a thick solid frame, is likely to be erroneously recognized using the first recognition process, so the task is recognized using the second recognition process. For tasks other than this, the task is recognized using the first recognition process. This makes it possible to prevent a decrease in task recognition accuracy compared to when tasks are recognized using only the first recognition process or only the second recognition process.

[0076] Next, the task recognition process executed by the CPU 21A of the task recognition device 20 will be described with reference to the flowchart shown in FIG.

[0077] In step S100, the CPU 21A acquires a captured image of the worker W working from the camera 30.

[0078] In step S101, the CPU 21A detects first detection information related to at least one of the right hand and the left hand of the worker W and the work object M, based on the captured image acquired in step S100. That is, the captured image is input to a first trained model for detection, and the first detection information is acquired.

[0079] In step S102, the CPU 21A detects second detection information related to the skeleton of the worker W based on the captured image acquired in step S100. That is, the CPU 21A inputs the captured image into a second trained model for detection to acquire the second detection information.

[0080] In step S103, the CPU 21A determines whether a switching condition for switching between recognizing the work by the first recognition process and recognizing the work by the second recognition process is satisfied based on the first detection information acquired in step S101. Specifically, it determines whether at least one of the first to eighth switching conditions described above is satisfied. If the determination in step S103 is negative, the process proceeds to step S104, and if the determination is affirmative, the process proceeds to step S105.

[0081] In step S104, CPU 21A executes a first recognition process based on the first detection information acquired in step S101 and the second detection information acquired in step S102. That is, the first detection information and the second detection information are input to a first trained model for recognition, and a recognition result of the task is acquired.

[0082] In step S105, CPU 21A executes a second recognition process based on the second detection information acquired in step S102. That is, the second detection information is input to a second trained model for recognition, and a recognition result of the task is acquired.

[0083] In step S106, the CPU 21A outputs the work recognition result acquired in step S104 or step S105 to the display unit 23 or the storage unit 25, for example.

[0084] In this manner, in this embodiment, whether to execute the first recognition process or the second recognition process is determined based on the first detection information, and if there is a high possibility of erroneous detection, the first recognition process is switched to the second recognition process, thereby improving the recognition accuracy of the task.

[0085] The above-described embodiment merely exemplifies the configuration of the present invention, and the present invention is not limited to the specific embodiment described above, and various modifications are possible within the scope of the technical concept thereof.

[0086] Furthermore, the task recognition process executed by the CPU after reading the software (program) in the above embodiments may be executed by various processors other than the CPU. Examples of such processors include dedicated electrical circuits, such as programmable logic devices (PLDs) (such as field-programmable gate arrays (FPGAs)) whose circuit configuration can be changed after manufacture, and application-specific integrated circuits (ASICs) that are processors with circuit configurations specifically designed to execute the recognition process. The task recognition process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices. [Explanation of symbols]

[0087] 10 Task Recognition System 20 Work recognition device 22 Control section 23 Display section 24 Communications Department 25 Memory section 25A Work Recognition Program 30 Camera 40 Acquisition Department 41 Detector 42 Detection unit 43 Recognition part 44 Recognition part 45 Judgment section 46 Output section

Claims

1. an acquisition unit that acquires a captured image of a worker's work; a first detection unit that detects first detection information related to at least one of the worker's right hand and left hand and the object of the work based on the captured image; a second detection unit that detects second detection information related to a skeleton of the worker based on the captured image; a first recognition unit that executes a first recognition process to recognize the work based on the detected first detection information and the detected second detection information; a second recognition unit that executes a second recognition process to recognize the work based on the detected second detection information; a determination unit that determines whether a switching condition for switching between recognizing the work by the first recognition unit and recognizing the work by the second recognition unit is satisfied based on the first detection information; and an output unit that outputs the recognition result of the task by the first recognition unit when the switching condition is not satisfied, and outputs the recognition result of the task by the second recognition unit when the switching condition is satisfied; Equipped with The determination unit determines that the switching condition is satisfied in at least one of the following cases: when the distance between the right hand and the left hand is equal to or greater than a predetermined threshold; when the speed of at least one of the detected hand and the object is equal to or greater than a predetermined threshold; and when the position of the detected hand is outside a range of a predetermined working space. Work recognition device.

2. The determination unit determines that the switching condition is satisfied when the distance between the hand and the object is equal to or greater than a predetermined threshold. The task recognition device according to claim 1 .

3. The determination unit determines that the switching condition is satisfied when the bounding box of the hand and the bounding box of the object do not overlap. The task recognition device according to claim 1 .

4. The determination unit determines whether the switching condition is satisfied based on the size of the detected object. The task recognition device according to any one of claims 1 to 3.

5. The determination unit determines that the switching condition is satisfied when the size of a bounding box of the detected object is smaller than a predetermined minimum size of the object. The task recognition device according to claim 4.

6. The determination unit determines that the switching condition is satisfied when the size of a bounding box of the detected object is larger than a predetermined maximum size of the object. The task recognition device according to claim 4.

7. the first detection unit calculates a reliability of the detected object; The determination unit determines that the switching condition is satisfied when the reliability of the object is equal to or less than a predetermined threshold. The task recognition device according to claim 1 .

8. The computer Acquire photographed images of the worker's work, Detecting first detection information related to at least one of the worker's right hand and left hand and the object of the work based on the captured image; detecting second detection information related to the worker's skeleton based on the captured image; executing a first recognition process for recognizing the work based on the detected first detection information and the detected second detection information; performing a second recognition process for recognizing the work based on the detected second detection information; determining whether a switching condition for switching between recognizing the work by the first recognition processing and recognizing the work by the second recognition processing is satisfied based on the first detection information; If the switching condition is not satisfied, output the recognition result of the task by the first recognition processing, and if the switching condition is satisfied, output the recognition result of the task by the second recognition processing. This includes: The switching condition is determined to be satisfied in at least one of the following cases: when the distance between the right hand and the left hand is equal to or greater than a predetermined threshold; when the speed of at least one of the detected hand and the object is equal to or greater than a predetermined threshold; and when the position of the detected hand is outside a range of a predetermined workspace. A work-aware method for performing processing.

9. On the computer, Acquire photographed images of the worker's work, Detecting first detection information related to at least one of the worker's right hand and left hand and the object of the work based on the captured image; detecting second detection information related to the worker's skeleton based on the captured image; executing a first recognition process for recognizing the work based on the detected first detection information and the detected second detection information; performing a second recognition process for recognizing the work based on the detected second detection information; determining whether a switching condition for switching between recognizing the work by the first recognition processing and recognizing the work by the second recognition processing is satisfied based on the first detection information; If the switching condition is not satisfied, output the recognition result of the task by the first recognition processing, and if the switching condition is satisfied, output the recognition result of the task by the second recognition processing. This includes: The switching condition is determined to be satisfied in at least one of the following cases: when the distance between the right hand and the left hand is equal to or greater than a predetermined threshold; when the speed of at least one of the detected hand and the object is equal to or greater than a predetermined threshold; and when the position of the detected hand is outside a range of a predetermined workspace. A task recognition program that executes the process.

Images