Action detection device, action detection system, program, and action detection method

Abstract

An action detection device (130) comprises: a hand surrounding area image acquisition unit (133) which acquires a first target image that indicates a detection target in a first time and a second target image that indicates the detection target in a second time; an encoding processing unit (137) which extracts a first feature amount and a second feature amount respectively from the first target image and the second target image using an encoder trained by performing training processing involving training an autoencoder including the encoder that encodes one target image included in a plurality of target images, thereby extracting a feature amount and a decoder that decodes the feature amount and that outputs a decoded image, and training processing involving training the encoder such that the total amount of change in a plurality of feature amounts extracted from a plurality of target images becomes smaller; and a component operation detection processing unit (141) which, when the difference between the first feature amount and the second feature amount exceeds a predetermined threshold value, and the detection target is in a predetermined region, detects that a predetermined action is being performed.

Description

Motion detection device, motion detection system, program, and motion detection method

[0001] The present disclosure relates to a motion detection device, a motion detection system, a program, and a motion detection method.

[0002] Traceability information is information that associates product numbers with part lot numbers, and is used for quality control. Because part lot numbers are affixed only to part boxes, not to parts themselves, ensuring the accuracy of traceability information requires a system equipped with a means to observe a series of operations performed by workers, such as taking out and putting in.

[0003] For example, the technology described in Patent Document 1 detects work when the skeleton of a worker extracted from a video of the worker enters a designated area.

[0004] Japanese Patent Application Laid-Open No. 2022-162314

[0005] However, conventional technology detects work when the skeleton extracted from the worker's video enters a specified area, so if there are other parts boxes or products on the worker's path, it may mistakenly detect a different work.

[0006] Therefore, one or more aspects of the present disclosure aim to prevent erroneous detection due to passing through a predetermined area.

[0007] a first learning process for training an autoencoder including an encoder that extracts features by encoding one target image included in a plurality of target images that show the target to be trained in time series over a predetermined period of time, and a decoder that decodes the features to output a decoded image; and a second learning process for training the encoder so that the decoded image approaches the one target image, and a second learning process for training the encoder so that a total sum of changes in the features extracted from the plurality of target images by the encoder is reduced. The motion detection device according to one aspect of the present disclosure is characterized by comprising: a detection target image acquisition unit that acquires a first target image that shows a detection target that is a target to be detected at a first time, and a second target image that shows the detection target at a second time that is later than the first time; an encoding processing unit that extracts a first feature from the first target image and a second feature from the second target image using the trained encoder; and a detection unit that detects that a predetermined motion is being performed at the second time when a difference between the first feature and the second feature exceeds a predetermined threshold and the detection target is in a predetermined area at the second time.

[0008] A motion detection system according to one aspect of the present disclosure includes a first learning unit that trains an autoencoder including a training image acquisition unit that acquires a plurality of target images showing a target to be trained in time series over a predetermined period, an encoder that extracts features by encoding one of the plurality of target images, and a decoder that decodes the features and outputs a decoded image, so that the decoded image approaches the one target image, and a second learning unit that trains the encoder so that a total amount of change in the plurality of features extracted from the plurality of target images by the encoder becomes small, thereby generating a trained encoder. a detection target image acquisition unit that acquires a first target image showing a detection target that is an object to be detected at a first time and a second target image showing the detection target at a second time that is later than the first time; an encoding processing unit that uses the trained encoder to extract a first feature amount from the first target image and a second feature amount from the second target image; and a detection unit that detects that a predetermined operation is being performed at the second time when a difference between the first feature amount and the second feature amount exceeds a predetermined threshold and the detection target is in a predetermined area at the second time.

[0009] A program according to one aspect of the present disclosure includes a computer, a detection target image acquisition unit that acquires a first target image showing a detection target that is a target to be detected at a first time and a second target image showing the detection target at a second time later than the first time, an encoder that extracts features by encoding one target image included in a plurality of target images that show the target to be learned in chronological order over a predetermined period of time, and a decoder that decodes the features to output a decoded image, a first learning process that learns the autoencoder so that the decoded image approaches the one target image, and The encoder is trained by performing a second learning process to train the encoder so that the total amount of change in the plurality of feature quantities extracted from the plurality of target images by the encoder becomes small, and the encoder is made to function as an encoding processing unit that extracts a first feature quantity from the first target image and a second feature quantity from the second target image using the trained encoder, and a detection unit that detects that a predetermined action is being performed at the second time when the difference between the first feature quantity and the second feature quantity exceeds a predetermined threshold and the detection target is in a predetermined area at the second time.

[0010] A motion detection method according to one aspect of the present disclosure includes acquiring a first target image showing a detection target that is an object to be detected at a first time and a second target image showing the detection target at a second time later than the first time, and performing a first learning process to train an autoencoder including an encoder that extracts features by encoding one target image included in a plurality of target images that show the object to be learned in chronological order over a predetermined period of time, and a decoder that decodes the features and outputs a decoded image so that the decoded image approaches the one target image, and a second learning process to train the encoder so that a total sum of changes in the plurality of features extracted from the plurality of target images by the encoder is reduced, thereby extracting a first feature from the first target image and a second feature from the second target image, and detecting that a predetermined motion is being performed at the second time when a difference between the first feature and the second feature exceeds a predetermined threshold and the detection target is in a predetermined area at the second time.

[0011] According to one or more aspects of the present disclosure, it is possible to prevent erroneous detection due to passing through a predetermined area.

[0012] FIG. 1 is a block diagram schematically showing the configuration of a scene change extraction system according to an embodiment; FIG. 2 is a block diagram schematically showing the configuration of a learning device; FIG. 3 is a schematic diagram for explaining processing in an image difference evaluation processing unit and processing in a change frequency evaluation processing unit; (A) and (B) are schematic diagrams for explaining feature quantities learned in an embodiment; FIG. 4 is a block diagram schematically showing the configuration of a PC; FIG. 5 is a block diagram schematically showing the configuration of a motion detection device; FIG. 6 is a flowchart showing the overall operation of the learning device; FIG. 7 is a flowchart showing the learning operation in the learning device; FIG. 8 is a flowchart showing the overall operation of the motion detection device.

[0013] 1 is a block diagram showing a schematic configuration of a scene change extraction system 100 as a motion detection system according to an embodiment. The scene change extraction system 100 includes a learning device 110 and a motion detection device 130. The learning device 110 and the motion detection device 130 are connected to a network 101, such as the Internet or a local area network (LAN). A camera 102 that captures video may be connected to the network 101.

[0014] 2 is a block diagram showing a schematic configuration of the learning device 110. The learning device 110 includes a communication unit 111, a video acquisition unit 112, a hand peripheral image acquisition unit 113, an auto-encoding processing unit 117, an image difference evaluation processing unit 120, and a change frequency evaluation processing unit 121.

[0015] The communication unit 111 performs communication. Here, the communication unit 111 performs communication via the network 101.

[0016] The image acquisition unit 112 acquires an image of the worker working. The acquired image is provided to the hand surrounding image acquisition unit 113. For example, the image acquisition unit 112 acquires the image from the camera 102 via the communication unit 111. Note that the image acquisition unit 112 may acquire the image by reading the image stored in a storage unit (not shown).

[0017] The hand-periphery image acquisition unit 113 functions as a learning-target image acquisition unit that acquires a plurality of target images showing the target for learning in chronological order over a predetermined period. Here, the hand-periphery image acquisition unit 113 acquires, as target images, hand-periphery images, which are images of the area around the worker's hand, from each frame included in the video. In this embodiment, the target for learning is the worker's hand. The target image acquired here is also referred to as a learning target image. The hand-periphery image acquisition unit 113 includes a foreground extraction processing unit 114, a skeleton extraction processing unit 115, and a hand-periphery image extraction processing unit 116.

[0018] The foreground extraction processing unit 114 extracts the foreground corresponding to the worker from each frame included in the video. A known technique may be used to extract the foreground. For example, the foreground extraction processing unit 114 may extract, as the foreground, a portion that shows movement in the previous and next frames.

[0019] The skeleton extraction processing unit 115 extracts the skeleton of the worker from each frame included in the video. The skeleton extraction may be performed using a known technique. For example, the skeleton extraction processing unit 115 may extract the skeleton using OpenPose.

[0020] Hand-periphery image extraction processing section 116 extracts, as a hand-periphery image, a predetermined region including the hand in the skeleton extracted by skeleton extraction processing section 115, from the foreground extracted by foreground extraction processing section 114. The extracted hand-periphery image is provided to auto-encoding processing section 117 and image difference evaluation processing section 120.

[0021] The auto-encoding processing unit 117 uses an auto-encoder to take the hand-periphery image as an input image, encodes the input image to calculate features, and decodes the features to generate an output image. The output image is provided to the image difference evaluation processing unit 120 and is also called a decoded image. The auto-encoding processing unit 117 includes an encoding processing unit 118 and a decoding processing unit 119.

[0022] The encoding unit 118 calculates the feature amounts by encoding the image around the hand using an encoder included in the autoencoder. The calculated feature amounts are provided to the decoding unit 119 and the change frequency evaluation unit 121.

[0023] The decoding unit 119 generates an output image by decoding the feature amount using a decoder included in the autoencoder. The output image is provided to the image difference evaluation unit 120.

[0024] The image difference evaluation processing unit 120 functions as a first learning unit that learns an autoencoder including an encoder that extracts features by encoding a target image, i.e., a hand-periphery image, and a decoder that decodes the features and outputs a decoded image, so that the decoded image approaches the hand-periphery image. This learning is also referred to as a first learning process.

[0025] For example, the image difference evaluation processing unit 120 trains the autoencoder so that the difference between the image of the vicinity of the hand as an input image to the autoencode processing unit 117 and the output image from the autoencode processing unit 117 becomes small.

[0026] The change frequency evaluation processing unit 121 functions as a second learning unit that generates a trained encoder by training the encoder so that the total amount of change in multiple feature quantities extracted by the encoder becomes small. This learning is also referred to as a second learning process.

[0027] For example, the change frequency evaluation processing unit 121 calculates the time change of the feature amount as the absolute value of the difference between the feature amount of a certain frame and the feature amount of the next frame from the encoding processing unit 118, and then calculates the total feature amount change by aggregating the time change of the feature amount over a predetermined period.The change frequency evaluation processing unit 121 then learns the encoder used by the encoding processing unit 118 of the auto-encoding processing unit 117 so as to reduce the total feature amount change.

[0028] 3 is a schematic diagram for explaining the processing of the image difference evaluation processing unit 120 and the change frequency evaluation processing unit 121. As shown in Fig. 3, a hand surrounding image M as an input image is input to the autoencoder, and a decoded image M' is output as an output image.

[0029] The image difference evaluation processing unit 120 calculates the difference E between the hand surrounding image M and the decoded image M′. １ Calculate the difference E １ Update the autoencoder parameters so that is small.

[0030] The change frequency evaluation processing unit 121 also receives the feature value f ｔ Change in |Δf ｔ | is calculated as the time change of the feature amount, and the time change of the feature amount is totaled for a predetermined period, thereby obtaining the total change of the feature amount E ２ Then, the change frequency evaluation processing unit 121 calculates the total feature change amount E ２ The parameters of the encoder used by the encoding processing unit 118 are updated so that .times. ...

[0031] By performing the above-described learning in the image difference evaluation processing unit 120, separable features are acquired, as shown in FIG. 4A. This makes it possible to easily perform class classification using smooth hyperplanes. However, learning in the image difference evaluation processing unit 120 does not necessarily result in the acquisition of discriminative features with little variation between similar images, and it may be difficult to detect changes in the scene from changes in the features.

[0032] In contrast, after learning is performed in the image difference evaluation processing unit 120, the change frequency evaluation processing unit 121 trains the encoder to reduce the amount of change between consecutive frames, thereby outputting features that are less likely to vary along the time axis, making it possible to acquire discriminative features as shown in Figure 4 (B).

[0033] The learning device 110 described above can be realized by, for example, a computer such as the PC 10 shown in Fig. 5. The PC 10 includes a storage 11 such as a hard disk drive (HDD) and a solid state drive (SSD), a memory 12, a processor 13 such as a central processing unit (CPU), a communication interface (I / F) 14 such as a network interface card (NIC), an input interface 15 such as a keyboard and a mouse, and a display 16.

[0034] For example, the video acquisition unit 112, the hand-periphery image acquisition unit 113, the auto-encoding processing unit 117, the image difference evaluation processing unit 120, and the change frequency evaluation processing unit 121 can be realized by the processor 13 loading a program stored in the storage 11 into the memory 12 and executing the program. The communication unit 111 can be realized by the communication I / F 14.

[0035] The program may be downloaded to the storage 11 from a recording medium (not shown) via a reader / writer (not shown) or from the network 101 via the communication I / F 14, and then loaded onto the memory 12 and executed by the processor 13. Alternatively, the program may be directly loaded onto the memory 12 from a recording medium via the reader / writer or from the network 101 via the communication I / F 14, and then executed by the processor 13. In other words, the program may be provided by a computer program product such as a recording medium.

[0036] 6 is a block diagram showing a schematic configuration of the motion detection device 130. The motion detection device 130 includes a communication unit 131, an image acquisition unit 132, a hand peripheral image acquisition unit 133, an encoding processing unit 137, a grip state change detection unit 138, an area information storage unit 139, a set area entry / exit detection unit 140, a part operation detection processing unit 141, and a traceability information storage unit 142.

[0037] The communication unit 131 performs communication. In this example, the communication unit 111 performs communication via the network 101.

[0038] The image acquisition unit 132 acquires an image of the worker working. The acquired image is provided to the hand surrounding image acquisition unit 133. For example, the image acquisition unit 132 acquires the image from the camera 102 via the communication unit 131. Note that the image acquisition unit 132 may acquire the image by reading the image stored in a storage unit (not shown).

[0039] The hand-periphery image acquisition unit 133 acquires hand-periphery images, which are images of the periphery of the worker's hands, from each frame included in the video. For example, the hand-periphery image acquisition unit 133 functions as a detection target image acquisition unit that acquires a first target image showing a detection target (here, the worker's hand) that is an object to be detected at a first time, and a second target image showing the detection target at a second time that is later than the first time. The hand-periphery image acquisition unit 133 includes a foreground extraction processing unit 134, a skeleton extraction processing unit 135, and a hand-periphery image extraction processing unit 136.

[0040] The foreground extraction processing unit 134 extracts the foreground corresponding to the worker from each frame included in the video. The foreground extraction may be performed using a known technique.

[0041] The skeleton extraction processing unit 135 extracts the worker's skeleton from each frame included in the video. The skeleton extraction may be performed using a known technique. The extracted skeleton is notified to the hand surrounding image extraction processing unit 136 and the set area entry / exit detection unit 140.

[0042] Hand-periphery image extraction processing section 136 extracts, as a hand-periphery image, a predetermined region including the hand in the skeleton extracted by skeleton extraction processing section 135, from the foreground extracted by foreground extraction processing section 114. The extracted hand-periphery image is provided to encoding processing section 137.

[0043] The encoding processing unit 137 is a processing unit that receives an encoder from the autoencoder trained by the training device 110 and performs encoding using the encoder. For example, the motion detection device 130 receives the encoder from the training device 110 via the communication unit 131 and operates the encoder to generate the encoding processing unit 137.

[0044] The encoding processing unit 137 calculates feature amounts by encoding the hand-periphery image. The calculated feature amounts are provided to the grip state change detection unit 138. For example, the encoding processing unit 137 uses a trained encoder that is an encoder trained by the training device 110 to extract a first feature amount from the first target image and a second feature amount from the second target image.

[0045] The grip state change detection unit 138 detects a change in the feature quantity from the encoding processing unit 137 over time. The grip state change detection unit 138 is also referred to as a first detection unit. For example, the grip state change detection unit 138 detects a change in the state of the detection target when the change between the first feature quantity and the second feature quantity exceeds a predetermined threshold. Specifically, the grip state change detection unit 138 calculates a grip state change amount indicating the magnitude of the change in the feature quantity from the encoding processing unit 137 over time. Specifically, the grip state change detection unit 138 calculates the grip state change amount by calculating the absolute value of the difference between the feature quantity calculated from a certain frame and the feature quantity calculated from the next frame. Then, the grip state change detection unit 138 detects a change in the feature quantity from the encoding processing unit 137 over time when the calculated grip state change amount exceeds a predetermined threshold. The detection result of the grip state change detection unit 138 is provided to the part operation detection processing unit 141.

[0046] The area information storage unit 139 stores area information indicating, as a designated area, an area from which a worker takes out a part or an area into which a worker puts a part in the video acquired by the video acquisition unit 132. For example, the area information indicates the coordinates of the area from which a worker takes out a part or an area into which a worker puts a part.

[0047] The set area entry / exit detection unit 140 detects that the worker's hand has entered or left the designated area indicated by the area information stored in the area information storage unit 139, based on the time series of the hand coordinates in the skeleton extracted by the skeleton extraction processing unit 135. The set area entry / exit detection unit 140 is also referred to as a second detection unit.

[0048] The part operation detection processing unit 141 functions as a detection unit that detects that a predetermined action is being performed at a second time when, based on the detection result from the handle state change detection unit 138 and the detection result from the set area entry / exit detection unit 140, the difference between the first feature amount and the second feature amount exceeds a predetermined threshold and the detection target is in the predetermined area at the second time. In this embodiment, the predetermined action is the work of the worker. The part operation detection processing unit 141 is also referred to as a third detection unit.

[0049] For example, the part operation detection processing unit 141 detects that a worker has removed a part or inserted a part from the detection result from the handle state change detection unit 138 and the detection result from the set area entry / exit detection unit 140, and generates traceability information indicating the detection result. The traceability information is stored in the traceability information storage unit 142.

[0050] Specifically, if the detection result from the set area entry / exit detection unit 140 indicates that a change in the handle state is also detected by the detection result from the handle state change detection unit 138 between the time when the worker's hand enters a specified area and the time when the worker's hand is detected to have left the specified area, the part operation detection processing unit 141 will assume that work has been performed in the specified area and generate traceability information that includes at least the time and the work that is scheduled to be performed in the specified area.

[0051] The traceability information storage unit 142 stores the traceability information.

[0052] The above-described motion detection device 130 can also be realized by a computer such as the PC 10 shown in Fig. 5. For example, the video acquisition unit 132, the hand peripheral image acquisition unit 133, the encoding processing unit 137, the grip state change detection unit 138, the set area entry / exit detection unit 140, and the part operation detection processing unit 141 can be realized by the processor 13 loading a program stored in the storage 11 into the memory 12 and executing the program. The area information storage unit 139 and the traceability information storage unit 142 can be realized by the storage 11 or the memory 12. The communication unit 131 can be realized by the communication I / F 14.

[0053] 7 is a flowchart showing the overall operation of the learning device 110. First, the video acquisition unit 112 acquires a video of the worker working, for example, via the communication unit 111 (S10). The acquired video is provided to the hand surrounding image acquisition unit 113.

[0054] Next, the hand-periphery image acquisition unit 113 acquires hand-periphery images, which are images of the area around the worker's hands, from each frame included in the video (S11). The acquired hand-periphery images are provided to the auto-encoding processing unit 117 and the image difference evaluation processing unit 120.

[0055] Next, the image difference evaluation processing unit 120 and the change frequency evaluation processing unit 121 train the autoencoder using the processing results from the autoencoder processing unit 117 (S12). The training here is unsupervised training, the details of which will be described later. The encoder of the trained autoencoder is sent to the motion detection device 130 via the communication unit 111.

[0056] 8 is a flowchart showing the learning operation of the learning device 110. First, the encoding processing unit 118 of the auto-encoding processing unit 117 identifies one hand-periphery image from all hand-periphery images to be learned, and encodes the identified hand-periphery image to calculate feature amounts (S20). The calculated feature amounts are provided to the decoding processing unit 119 and the change frequency evaluation processing unit 121.

[0057] The decoding processing unit 119 generates a decoded image as an output image by decoding the feature amount from the encoding processing unit 118 (S21). The decoded image is provided to the image difference evaluation processing unit 120.

[0058] The image difference evaluation processing unit 120 calculates the difference E between the image around the hand and the decoded image. １ is calculated (S22).

[0059] The image difference evaluation processing unit 120 calculates the difference E １ It is determined whether the calculated difference E is equal to or smaller than a predetermined allowable value (S23). １ If the calculated difference E is greater than the predetermined allowable value (No in S23), it is determined that learning is necessary, and the process proceeds to step S24. １ If is equal to or smaller than the predetermined allowable value (Yes in S23), it is determined that learning is not necessary, and the process proceeds to step S25.

[0060] In step S24, the image difference evaluation processing unit 120 calculates the calculated difference E １ The difference E １ The parameters of the autoencoder are updated by multiplying the differential value of by the learning rate. Then, the process returns to step S23.

[0061] In step S25, the image difference evaluation processing unit 120 determines whether learning for a predetermined maximum number of epochs has been completed for all hand peripheral images to be learned. If learning for the maximum number of epochs has been completed for all hand peripheral images (Yes in S25), the process proceeds to step S26. If learning for the maximum number of epochs has not been completed for all hand peripheral images, in other words, if there are hand peripheral images for which learning for the maximum number of epochs has not been completed (No in S25), the process returns to step S20, and hand peripheral images for which learning for the maximum number of epochs has not yet been completed become the subject of learning.

[0062] In step S26, the change frequency evaluation processing unit 121 calculates the change in the feature amount from the encoding processing unit 118 as the time change in the feature amount, and calculates the time change in the feature amount over a predetermined period of time to obtain a total feature amount change E ２ Calculate.

[0063] The change frequency evaluation processing unit 121 calculates the calculated total feature change amount E ２ It is determined whether the calculated total feature amount change E is equal to or less than a predetermined allowable value (S27). ２ is greater than the predetermined allowable value (No in S27), it is determined that learning is necessary, and the process proceeds to step S28. ２ If is equal to or smaller than the predetermined allowable value (Yes in S27), it is determined that learning is not necessary, and the process proceeds to step S29.

[0064] In step S28, the change frequency evaluation processing unit 121 calculates the calculated total feature amount change E ２ The total feature change amount E ２ The parameters of the encoding processing unit 118 corresponding to the encoder of the autoencoder are updated by multiplying the differential value of by the learning rate. Then, the process returns to step S20, where one hand-periphery image is identified from all the hand-periphery images to be learned, and the process is repeated.

[0065] In step S29, the change frequency evaluation processing unit 121 determines whether learning for a predetermined maximum number of epochs has been completed for all hand peripheral images to be learned. If learning for the maximum number of epochs has been completed for all hand peripheral images (Yes in S29), the process ends. If learning for the maximum number of epochs has not been completed for all hand peripheral images, in other words, if there are hand peripheral images for which learning for the maximum number of epochs has not been completed (No in S29), the process returns to step S26, and multiple hand peripheral images for which learning for the maximum number of epochs has not yet been completed become the target of learning. The maximum number of epochs in step S29 may be the same as or different from the maximum number of epochs in step S25.

[0066] 9 is a flowchart showing the overall operation of the motion detection device 130. It is assumed that the motion detection device 130 has already acquired the encoder from the learning device 110 via the communication unit 131, and that the encoding processing unit 137 is capable of performing encoding using the acquired encoder.

[0067] First, the image acquisition unit 132 acquires an image of the worker working, for example, via the communication unit 131 (S30). The acquired image is provided to the hand surrounding image acquisition unit 133.

[0068] Next, the hand-periphery image acquisition unit 133 acquires a hand-periphery image, which is an image of the area around the worker's hand, from each frame included in the video (S31). The acquired hand-periphery image is provided to the encoding processing unit 137.

[0069] Next, the encoding processing unit 137 calculates feature amounts by encoding the image around the hand using the encoder trained by the learning device 110 (S32). The calculated feature amounts are provided to the grip state change detection unit 138.

[0070] Next, based on the detection results from the handle state change detection unit 138 and the detection results from the set area entry / exit detection unit 140, if a change in the handle state is detected by the detection results from the handle state change detection unit 138 between the time when the worker's hand is detected entering a designated area and the time when the worker's hand is detected leaving that designated area, the part operation detection processing unit 141 assumes that work has been performed in that designated area and generates traceability information that includes at least the time and the work that is scheduled to be performed in that designated area (S33).

[0071] As described above, according to this embodiment, excessive changes in feature quantities can be suppressed and scene changes can be captured with high accuracy by imposing constraints on the time changes in feature quantities using the change frequency evaluation processing unit 121. When this is applied to the motion detection device 130 that functions as a traceability device, scenes in which a held object has changed, in other words, operations such as the removal or insertion of a part, can be detected with high accuracy from the time series of images of the area around the hand.

[0072] Furthermore, since the learning device 110 uses an autoencoder that allows learning without a teacher, the effort required to create learning data can be reduced.

[0073] In the embodiment described above, the hand peripheral image acquisition unit 113 and the hand peripheral image acquisition unit 133 acquire the hand peripheral image by extracting the worker's skeleton from the image, but the embodiment is not limited to this example. For example, a marker may be attached to the object (here, the hand) whose peripheral image is to be acquired, and the hand peripheral image acquisition unit 113 and the hand peripheral image acquisition unit 133 may detect the marker from the image and acquire an image around the marker as the hand peripheral image.

[0074] In the embodiment described above, the set area entrance / exit detection unit 140 determines whether the hand area within the skeleton extracted from the image enters or exits a predetermined area, but the embodiment is not limited to this example. For example, the set area entrance / exit detection unit 140 may determine whether the entire skeleton enters or exits a predetermined area. Even in this case, the object within the skeleton (here, the hand) can be treated as having entered or exited the predetermined area.

[0075] The processing in the scene change extraction system 100 described above is performed by the learning device 110 and the motion detection device 130, but the embodiment is not limited to this example. The processing in the scene change extraction system 100 may be performed by one or more computers that are capable of communicating with each other. When the processing in the scene change extraction system 100 is performed by a single computer, that computer may be a motion detection device.

[0076] REFERENCE SIGNS LIST 100 Scene change extraction system, 110 Learning device, 111 Communication unit, 112 Video acquisition unit, 113 Hand surrounding image acquisition unit, 114 Foreground extraction processing unit, 115 Skeleton extraction processing unit, 116 Hand surrounding image extraction processing unit, 117 Auto-encoding processing unit, 118 Encoding processing unit, 119 Decoding processing unit, 120 Image difference evaluation processing unit, 121 Change frequency evaluation processing unit, 130 Action detection device, 131 Communication unit, 132 Video acquisition unit, 133 Hand surrounding image acquisition unit, 134 Foreground extraction processing unit, 135 Skeleton extraction processing unit, 136 Hand surrounding image extraction processing unit, 137 Encoding processing unit, 138 Grip state change detection unit, 139 Area information storage unit, 140 Set area entry / exit detection unit, 141 Part operation detection processing unit, 142 Traceability information storage unit.

Claims

1. A detection target image acquisition unit acquires a first target image showing a detection target to be detected at a first time and a second target image showing the detection target at a second time later than the first time; an autoencoder that extracts features by encoding one target image included in a plurality of target images showing the target to be learned in a time series over a predetermined period of time, and a decoder that decodes the features and outputs a decoded image; an encoding unit that extracts a first feature from the first target image and a second feature from the second target image using the autoencoder that has been trained by performing a first learning process to train the encoder so that the decoded image approaches the one target image, and a second learning process to train the encoder so that a total amount of change in the plurality of feature amounts extracted from the plurality of target images by the encoder becomes small; and a detection unit that detects that a predetermined action is being performed at the second time when a difference between the first feature and the second feature exceeds a predetermined threshold and the detection target is in a predetermined area at the second time. A motion detection device comprising:

2. The motion detection device according to claim 1, wherein the target is a worker's hand, and the predetermined motion is an operation performed by the worker.

3. A training target image acquisition unit that acquires a plurality of target images that show a target to be trained in time series over a predetermined period of time; a first learning unit that trains an autoencoder that includes an encoder that extracts features by encoding one of the plurality of target images and a decoder that decodes the features and outputs a decoded image so that the decoded image approaches the one target image; a second learning unit that generates a trained encoder by training the encoder so that the total amount of change in the plurality of features extracted from the plurality of target images by the encoder becomes small; a detection target image acquisition unit that acquires a first target image that shows a detection target that is a target to be detected at a first time and a second target image that shows the detection target at a second time that is later than the first time; and an encoding processing unit that uses the trained encoder to extract a first feature from the first target image and a second feature from the second target image. a detection unit that detects that a predetermined action is being performed at the second time when a difference between the first feature amount and the second feature amount exceeds a predetermined threshold and the detection target is in a predetermined area at the second time.

4. A computer is caused to function as: a detection target image acquisition unit that acquires a first target image showing a detection target that is an object to be detected at a first time, and a second target image showing the detection target at a second time later than the first time; an autoencoder that performs a first learning process to train an autoencoder including an encoder that extracts features by encoding one target image included in a plurality of target images that show the object to be learned in chronological order over a predetermined period, and a decoder that decodes the features and outputs a decoded image so that the decoded image approaches the one target image; and a second learning process to train the encoder so that the total amount of change in the plurality of features extracted from the plurality of target images by the encoder becomes small; an encoding processing unit that extracts a first feature from the first target image and a second feature from the second target image; and a detection unit that detects that a predetermined action is being performed at the second time when the difference between the first feature and the second feature exceeds a predetermined threshold and the detection target is in a predetermined area at the second time. A program characterized by.

5. A motion detection method comprising: acquiring a first target image showing a detection target to be detected at a first time, and a second target image showing the detection target at a second time later than the first time; extracting features by encoding one target image included in a plurality of target images showing the target to be learned in a time series over a predetermined period of time; and performing a first learning process to train an autoencoder including an encoder that extracts features by encoding one target image included in a plurality of target images that show the target to be learned in a time series over a predetermined period of time, and a decoder that decodes the features and outputs a decoded image so that the decoded image approaches the one target image; and performing a second learning process to train the encoder so that a total amount of change in the plurality of features extracted from the plurality of target images by the encoder becomes small, thereby extracting a first feature from the first target image and extracting a second feature from the second target image; and detecting that a predetermined motion is being performed at the second time when a difference between the first feature and the second feature exceeds a predetermined threshold and the detection target is in a predetermined area at the second time.

Images