Detection device, detection method, and detection program

Abstract

A detection device (200) includes an acquisition unit (220) that acquires one or more normal feature vectors, which are vector representations of skeleton information indicating normal behavior, and a plurality of images including an observation subject, an extraction unit (230) that extracts a plurality of pieces of the skeleton information using the plurality of images, a generation unit (240) that converts each of the plurality of pieces of the skeleton information into a vector representation, and a determination unit (250) that, on the basis of feature vectors of a plurality of observation subjects obtained by conversion, and the normal feature vector, determines that behavior of the observation subject shows signs of abnormal behavior when distance between the feature vector of the observation subject and the normal feature vector increases over a period that is set in advance, or when a total value of distances between the feature vector of the observation subject and the normal feature vector is greater than a threshold value set in advance, over a period that is set in advance.

Description

Detection device, detection method, and detection program

[0001] The present disclosure relates to a detection device, a detection method, and a detection program.

[0002] Surveillance cameras are installed in many places. A technology has been proposed for detecting individuals engaging in abnormal behavior based on images obtained from the surveillance cameras (see Patent Literature 1). The detection device in Patent Literature 1 detects the behavior of an active subject captured in an image. If the detected behavior differs from normal behavior, the detection device extracts the active subject from the image as a candidate for an abnormal subject.

[0003] Patent No. 6692086

[0004] The above techniques can detect abnormal behavior that has already occurred, but they cannot detect signs of abnormal behavior.

[0005] The objective of the present disclosure is to detect signs of abnormal behavior.

[0006] According to one aspect of the present disclosure, there is provided a detection device, the detection device comprising: an acquisition unit that acquires one or more normal feature vectors, which are vector representations of skeletal information indicating normal behavior, and a plurality of images including an observation target; an extraction unit that uses the plurality of images to extract a plurality of pieces of skeletal information; a generation unit that converts each of the plurality of pieces of skeletal information into a vector representation; and a determination unit that, based on the plurality of feature vectors of the observation target obtained by the conversion and the normal feature vector, determines that the behavior of the observation target is a sign of abnormal behavior if a distance between the feature vectors of the observation target and the normal feature vector increases for a predetermined period of time or if a total value of the distances between the feature vectors of the observation target and the normal feature vectors for a predetermined period of time is greater than a predetermined threshold.

[0007] According to the present disclosure, it is possible to detect signs of abnormal behavior.

[0008] 1 is a diagram illustrating a generation device and a detection device according to a first embodiment. FIG. 1 is a diagram illustrating hardware included in the generation device according to the first embodiment. FIG. 2 is a block diagram illustrating functions of the generation device according to the first embodiment. FIG. 3 is a diagram illustrating an example of a group of normal feature vectors according to the first embodiment. FIG. 4 is a flowchart illustrating an example of processing performed by the generation device according to the first embodiment. FIG. 5 is a diagram illustrating a case where normal feature vectors are generated using an autoencoder according to the first embodiment. FIG. 6 is a diagram illustrating hardware included in the detection device according to the first embodiment. FIG. 7 is a block diagram illustrating functions of the detection device according to the first embodiment. FIG. 8 is a graph illustrating a case where distance increases according to the first embodiment. FIG. 9 is a diagram in which feature vectors of multiple observation targets according to the first embodiment are plotted against a group of normal feature vectors. FIG. 10 is a flowchart illustrating an example of processing performed by the detection device according to the first embodiment. FIG. 11 is a graph illustrating a case where the sum of distances according to the first modification of the first embodiment is greater than a threshold. FIG. 12 is a flowchart illustrating an example of processing performed by the detection device according to the first modification of the first embodiment. FIG. 13 is a flowchart illustrating an example of processing performed by the detection device according to the second modification of the first embodiment. 10 is a flowchart (part 1) illustrating an example of processing executed by the detection device according to the third embodiment; 11 is a flowchart (part 2) illustrating an example of processing executed by the detection device according to the third embodiment;

[0009] Hereinafter, an embodiment will be described with reference to the drawings.

[0010] Embodiment 1. Fig. 1 is a diagram showing a generation device and a detection device according to embodiment 1. Fig. 1 shows a generation device 100 and a detection device 200. The generation device 100 generates information used in the detection device 200. The detection device 200 can acquire images from a camera 300. The detection device 200 uses the information to detect signs of abnormal behavior.

[0011] First, the generating device 100 will be described.

[0012] 2 is a diagram showing hardware included in the generating device according to embodiment 1. The generating device 100 includes a processor 101, a volatile storage device 102, and a non-volatile storage device 103.

[0013] The processor 101 controls the entire generating device 100. For example, the processor 101 is a central processing unit (CPU) or a field programmable gate array (FPGA). The processor 101 may be a multiprocessor. The generating device 100 may also include a processing circuit.

[0014] The volatile storage device 102 is a main storage device of the generating device 100. For example, the volatile storage device 102 is a random access memory (RAM). The non-volatile storage device 103 is an auxiliary storage device of the generating device 100. For example, the non-volatile storage device 103 is a hard disk drive (HDD) or a solid state drive (SSD).

[0015] Next, a description will be given of the functions of the generation device 100. Fig. 3 is a block diagram showing the functions of the generation device according to embodiment 1. The generation device 100 includes a storage unit 110, an acquisition unit 120, an extraction unit 130, a conversion unit 140, an output unit 150, and a learning generation unit 160.

[0016] The storage unit 110 may be realized as a storage area secured in the volatile storage device 102 or the non-volatile storage device 103. Some or all of the acquisition unit 120, extraction unit 130, conversion unit 140, output unit 150, and learning generation unit 160 may be realized by a processing circuit. Furthermore, some or all of the acquisition unit 120, extraction unit 130, conversion unit 140, output unit 150, and learning generation unit 160 may be realized as program modules executed by the processor 101.

[0017] The storage unit 110 stores various information.

[0018] The acquisition unit 120 acquires a plurality of images showing normal behavior. For example, the acquisition unit 120 acquires the plurality of images from the storage unit 110. Also, for example, the acquisition unit 120 acquires the plurality of images from an external device. Note that the external device is a device that exists outside the generation device 100. For example, the external device is a cloud server, an external memory, etc. The external device is not shown in the drawing.

[0019] The extraction unit 130 extracts a plurality of pieces of skeletal information using a plurality of images. For example, the extraction unit 130 extracts a plurality of pieces of skeletal information using OpenPose.

[0020] The conversion unit 140 converts each of the multiple pieces of skeletal information into a vector representation. Specifically, the conversion unit 140 converts multiple joints indicated by the skeletal information into a vector representation. The vector obtained by the conversion is called a normal feature vector. The conversion unit 140 performs the conversion process on each piece of skeletal information. As a result, multiple normal feature vectors are obtained. In other words, a group of normal feature vectors is obtained.

[0021] Furthermore, the conversion unit 140 may use a multivariate Gaussian distribution or a mixed Gaussian distribution for each piece of skeleton information to generate a probability density of the coordinate values ​​of each joint indicated by the skeleton information as a normal feature vector. The normal feature vector v is expressed by Equation (1), where p is the probability density and n is the number of joints.

[0022]

[0023] Furthermore, the conversion unit 140 may use a multivariate Gaussian distribution or a mixed Gaussian distribution to calculate the probability density of the coordinate values ​​of each joint indicated by the skeletal information for each piece of skeletal information, and generate a normal feature vector based on the probability density and the Fisher vector. In this way, a group of normal feature vectors is generated. An example of a group of normal feature vectors is shown below.

[0024] Fig. 4 is a diagram showing an example of a normal feature vector group according to the first embodiment. Fig. 4 shows a feature vector space 400. Fig. 4 shows a state in which the normal feature vector group is plotted in the feature vector space 400.

[0025] The output unit 150 outputs the group of normal feature vectors to the detection device 200. The output unit 150 may output the group of normal feature vectors to an external device. The output unit 150 may output a multivariate Gaussian distribution or a mixed Gaussian distribution based on skeletal information indicating normal behavior to the detection device 200 or the external device.

[0026] Next, the processing executed by the generating device 100 will be described using a flowchart. FIG. 5 is a flowchart showing an example of the processing executed by the generating device of embodiment 1. (Step S11) The acquiring unit 120 acquires a plurality of images showing normal behavior. (Step S12) The extracting unit 130 uses the plurality of images to extract a plurality of pieces of skeletal information. (Step S13) The converting unit 140 converts each of the plurality of pieces of skeletal information into a vector representation. (Step S14) The outputting unit 150 outputs a group of normal feature vectors.

[0027] The above describes the case where multiple normal feature vectors are generated. The generation device 100 may acquire one image showing normal behavior and generate one normal feature vector based on the image.

[0028] Furthermore, the normal feature vector may be generated using an autoencoder. A case where a normal feature vector is generated will be described using a diagram. FIG. 6 is a diagram showing a case where a normal feature vector is generated using the autoencoder of the first embodiment. The learning generation unit 160 trains the autoencoder so that, when skeletal information is input to the autoencoder, the autoencoder outputs the same skeletal information. For example, when skeletal information M1 is input to the autoencoder, the learning generation unit 160 trains the autoencoder so that the autoencoder outputs skeletal information M2. Note that the skeletal information M1 and the skeletal information M2 are the same. The skeletal information used for learning is the skeletal information extracted by the extraction unit 130. The number of joints in the skeletal information M1 and the skeletal information M2 in FIG. 6 is an example.

[0029] After the autoencoder has completed learning, the learning generation unit 160 generates a normal feature vector, which is the difference between the skeletal information input to the autoencoder and the skeletal information output by the autoencoder. When the content of Figure 6 is used, the normal feature vector is the difference between skeletal information M1 and skeletal information M2.

[0030] The learning generation unit 160 may generate information output by the encoder of the autoencoder as a normal feature vector. Alternatively, the learning generation unit 160 may generate a vector obtained by converting skeletal information output by the autoencoder into a vector representation as a normal feature vector. For example, the learning generation unit 160 generates a vector obtained by converting skeletal information M2 into a vector representation as a normal feature vector. The output unit 150 may output the normal feature vector and the autoencoder to the detection device 200 or an external device.

[0031] Next, the detection device 200 will be described.

[0032] First, the hardware of the detection device 200 will be described. Fig. 7 is a diagram showing the hardware of the detection device according to the first embodiment. The detection device 200 is also called a computer. The detection device 200 includes a processor 201, a volatile storage device 202, a nonvolatile storage device 203, and an interface 204.

[0033] The processor 201 controls the entire detection device 200. For example, the processor 201 is a CPU, an FPGA, or the like. The processor 201 may be a multiprocessor. The detection device 200 may also include a processing circuit.

[0034] The volatile storage device 202 is a main storage device of the detection device 200. For example, the volatile storage device 202 is a RAM. The non-volatile storage device 203 is an auxiliary storage device of the detection device 200. For example, the non-volatile storage device 203 is a HDD or SSD. The interface 204 communicates with the camera 300.

[0035] Next, a description will be given of the functions of the detection device 200. Fig. 8 is a block diagram showing the functions of the detection device according to embodiment 1. The detection device 200 includes a storage unit 210, an acquisition unit 220, an extraction unit 230, a generation unit 240, a determination unit 250, and an output unit 260.

[0036] The storage unit 210 may be realized as a storage area secured in the volatile storage device 202 or the non-volatile storage device 203. The storage unit 210 may also be called a memory. Some or all of the acquisition unit 220, extraction unit 230, generation unit 240, determination unit 250, and output unit 260 may be realized by a processing circuit. Furthermore, some or all of the acquisition unit 220, extraction unit 230, generation unit 240, determination unit 250, and output unit 260 may be realized as program modules executed by the processor 201. For example, the program executed by the processor 201 is also called a detection program or a detection program product. For example, the detection program is recorded on a recording medium.

[0037] The storage unit 210 stores various information.

[0038] The acquisition unit 220 acquires one or more normal feature vectors. For example, the acquisition unit 220 acquires the normal feature vectors from the storage unit 210. Alternatively, for example, the acquisition unit 220 acquires the normal feature vectors from an external device. Note that the external device is a device that exists outside the detection device 200. For example, the external device is a cloud server, an external memory, or the like. An illustration of the external device is omitted. Note that the normal feature vector is a vector representation of skeletal information indicating normal behavior.

[0039] The acquisition unit 220 acquires a plurality of images including the observation target. For example, the acquisition unit 220 acquires a plurality of images including the observation target from the camera 300. The acquisition unit 220 may also acquire a plurality of images including the observation target from an external device. The observation target is a human being. The observation target may also be an animal or a machine. The plurality of images may also be considered as a video.

[0040] The extraction unit 230 extracts a plurality of pieces of skeletal information using a plurality of images. For example, the extraction unit 230 extracts the skeletal information using OpenPose.

[0041] The generation unit 240 converts each of the multiple pieces of skeletal information into a vector representation. Specifically, the generation unit 240 converts multiple joints indicated by the skeletal information into a vector representation. Through the conversion, a feature vector of the observation target is obtained. The generation unit 240 performs the conversion process on each piece of skeletal information. As a result, a feature vector of the observation target is obtained.

[0042] Furthermore, the generation unit 240 may use a multivariate Gaussian distribution or a mixed Gaussian distribution for each piece of skeletal information to generate a probability density of the coordinate values ​​of each joint indicated by the skeletal information as a feature vector of the object of observation. Note that the multivariate Gaussian distribution or the mixed Gaussian distribution is a multivariate Gaussian distribution or a mixed Gaussian distribution based on skeletal information indicating normal behavior. Note that the technology for calculating the probability density using the multivariate Gaussian distribution or the mixed Gaussian distribution is a publicly known technology.

[0043] Furthermore, the generation unit 240 may generate a probability density of the coordinate values ​​of each joint indicated by the skeletal information using a multivariate Gaussian distribution or a mixed Gaussian distribution for each piece of skeletal information, and generate a feature vector of the object of observation based on the probability density and the Fisher vector. Note that the technique for calculating the feature vector using the probability density and the Fisher vector is a well-known technique.

[0044] The detection device 200 may generate feature vectors using an autoencoder. A case where an autoencoder is used will be described below. The acquisition unit 220 acquires the autoencoder from the storage unit 210 or an external device. The generation unit 240 generates, for each piece of skeletal information, a feature vector that is the difference between the skeletal information input to the autoencoder and the skeletal information output by the autoencoder. As described above, the autoencoder is trained using skeletal information of normal behavior. In other words, the autoencoder is not trained using skeletal information indicating a sign of abnormal behavior. Therefore, when skeletal information indicating a sign of abnormal behavior is input to the autoencoder, the autoencoder cannot output skeletal information identical to the input skeletal information. Therefore, when skeletal information indicating a sign of abnormal behavior is input to the autoencoder, a feature vector whose difference is not zero is generated.

[0045] The generating unit 240 may input the skeletal information to an autoencoder for each piece of skeletal information and generate information output by the encoder of the autoencoder as a feature vector. Alternatively, the generating unit 240 may input the skeletal information to an autoencoder for each piece of skeletal information and generate a vector representation of the skeletal information output by the autoencoder as a feature vector.

[0046] The determination unit 250 determines that the behavior of the observation target is a sign of abnormal behavior if the distance between the feature vector of the observation target and the normal feature vector increases for a predetermined period based on the feature vectors of the observation target and the normal feature vector. If the number of normal feature vectors is two or more, the distance is the distance between the feature vector of the observation target and the normal feature vector that is closest to the feature vector of the observation target among the group of normal feature vectors. The distance is the Euclidean distance. The determination is expressed by equation (2). S is the distance. t is the time. N is the predetermined period.

[0047]

[0048] The distance S may also be the length of the feature vector of the observation target, or the Mahalanobis distance.

[0049] The graph shows a case where the distance increases. FIG. 9 is a diagram showing a graph of a case where the distance increases in the first embodiment. The vertical axis represents distance. The horizontal axis represents time. As shown in the graph in FIG. 9, when the distance increases for a predetermined period, the determination unit 250 determines that the behavior of the observed object is a sign of abnormal behavior.

[0050] Also shown is a diagram in which the feature vectors of multiple observation targets are plotted against a group of normal feature vectors. FIG. 10 is a diagram in which the feature vectors of multiple observation targets of embodiment 1 are plotted against a group of normal feature vectors. As shown in FIG. 10, when the observation target exhibits abnormal behavior, the feature vector of the observation target gradually deviates from the group of normal feature vectors. In such a situation, the determination unit 250 determines that the behavior of the observation target is a sign of abnormal behavior. In this way, a sign of abnormal behavior is detected by the determination process.

[0051] When a sign of abnormal behavior is detected, the output unit 260 outputs information indicating that a sign of abnormal behavior has been detected. For example, the output unit 260 outputs the information to a display of the detection device 200. Furthermore, for example, the output unit 260 outputs an audio signal of the information from a speaker.

[0052] Next, the processing executed by the detection device 200 will be described using a flowchart. FIG. 11 is a flowchart showing an example of processing executed by the detection device of embodiment 1. (Step S21) The acquisition unit 220 acquires a normal feature vector. (Step S22) The acquisition unit 220 acquires an image including the observation target from the camera 300. (Step S23) The extraction unit 230 extracts skeletal information using the image. (Step S24) The generation unit 240 converts the skeletal information into a vector representation. This results in a feature vector. (Step S25) The determination unit 250 calculates the distance between the feature vector of the observation target and the normal feature vector.

[0053] (Step S26) The determination unit 250 determines whether the acquired image is the first image. If the acquired image is the first image, the process proceeds to step S22. If the acquired image is the second or later image, the process proceeds to step S27. (Step S27) The determination unit 250 determines whether the distance calculated this time is greater than the distance calculated previously. If the condition is met, the process proceeds to step S28. If the condition is not met, the process ends. (Step S28) The determination unit 250 determines whether a predetermined time has elapsed since the first image was acquired. If the time has elapsed, the process proceeds to step S29. If the time has not elapsed, the process proceeds to step S22. (Step S29) The determination unit 250 determines that the behavior of the observed subject is a sign of abnormal behavior. In other words, a sign of abnormal behavior is detected. (Step S30) The output unit 260 outputs information indicating that a sign of abnormal behavior has been detected.

[0054] According to the first embodiment, the detection device 200 can detect a sign of abnormal behavior. When the detection device 200 detects a sign of abnormal behavior, the detection device 200 outputs information indicating that a sign of abnormal behavior has been detected. This allows the user to recognize the sign of abnormal behavior. Therefore, the user can take action before an incident or accident occurs.

[0055] Variation 1 of Embodiment 1. In the above, the determination unit 250 determines that the behavior of the observation target is a sign of abnormal behavior when the distance between the feature vector of the observation target and the normal feature vector increases for a predetermined period. The determination unit 250 may determine that the behavior of the observation target is a sign of abnormal behavior when the total value of the distances between the feature vectors of the observation target and the normal feature vector for a predetermined period is greater than a predetermined threshold, based on the feature vectors of multiple observation targets and the normal feature vector. This determination is expressed by equation (3). S is the distance. t is the time. N is the predetermined period.

[0056]

[0057] When the number of normal feature vectors is two or more, the distance S is the distance between the feature vector of the observation target and the normal feature vector that is closest to the feature vector of the observation target among the normal feature vectors. The distance S may also be the length of the feature vector of the observation target. Furthermore, the distance S may be the Mahalanobis distance.

[0058] A graph showing a case where the total distance value is greater than the threshold value. FIG. 12 is a diagram showing a graph showing a case where the total distance value is greater than the threshold value in Variation 1 of Embodiment 1. The vertical axis represents the total distance value. The horizontal axis represents time. As shown in the graph in FIG. 12, when the total distance value for a predetermined period is greater than the threshold value, the determination unit 250 determines that the behavior of the observed subject is a sign of abnormal behavior.

[0059] 13 is a flowchart showing an example of processing executed by the detection device of Variation 1 of Embodiment 1. (Step S31) The acquisition unit 220 acquires a normal feature vector. (Step S32) The acquisition unit 220 acquires an image including the observation target from the camera 300. (Step S33) The extraction unit 230 extracts skeletal information using the image. (Step S34) The generation unit 240 converts the skeletal information into a vector representation. This results in a feature vector. (Step S35) The determination unit 250 calculates the distance between the feature vector of the observation target and the normal feature vector.

[0060] (Step S36) The determination unit 250 determines whether a predetermined time has elapsed since the first image was acquired. If the time has elapsed, the process proceeds to step S37. If the time has not elapsed, the process proceeds to step S32. (Step S37) The determination unit 250 determines whether the total distance value is greater than a threshold value. If the total value is greater than the threshold value, the process proceeds to step S38. If the total value is equal to or less than the threshold value, the process ends. (Step S38) The determination unit 250 determines that the behavior of the observed object is a sign of abnormal behavior. (Step S39) The output unit 260 outputs information indicating that a sign of abnormal behavior has been detected.

[0061] According to the first modification of the first embodiment, the detection device 200 can detect signs of behavior such as tailing by using the total distance value as an index.

[0062] Variation 2 of Embodiment 1 The determination unit 250 may determine, based on the feature vectors of a plurality of observation targets and the normal feature vector, that the behavior of the observation targets is a sign of abnormal behavior if the distance between the feature vectors of the observation targets and the normal feature vectors has increased for a predetermined period of time and the total value of the distances between the feature vectors of the observation targets and the normal feature vectors for the predetermined period of time is greater than a predetermined threshold.

[0063] 14 is a flowchart showing an example of processing executed by the detection device of Modification 2 of Embodiment 1. (Step S41) The acquisition unit 220 acquires a normal feature vector. (Step S42) The acquisition unit 220 acquires an image including the observation target from the camera 300. (Step S43) The extraction unit 230 extracts skeletal information using the image. (Step S44) The generation unit 240 converts the skeletal information into a vector representation. This results in a feature vector. (Step S45) The determination unit 250 calculates the distance between the feature vector of the observation target and the normal feature vector.

[0064] (Step S46) The determination unit 250 determines whether the acquired image is the first image. If the acquired image is the first image, the process proceeds to step S42. If the acquired image is the second or later image, the process proceeds to step S47. (Step S47) The determination unit 250 determines whether the distance calculated this time is greater than the distance calculated previously. If the condition is met, the process proceeds to step S48. If the condition is not met, the process ends. (Step S48) The determination unit 250 determines whether a predetermined time has elapsed since the first image was acquired. If the time has elapsed, the process proceeds to step S49. If the time has not elapsed, the process proceeds to step S42. (Step S49) The determination unit 250 determines whether the sum of the distances is greater than a threshold value. If the sum is greater than the threshold value, the process proceeds to step S50. If the sum is equal to or less than the threshold value, the process ends. (Step S50) The determination unit 250 determines that the behavior of the observation target is a sign of abnormal behavior. (Step S51) The output unit 260 outputs information indicating that a sign of abnormal behavior has been detected.

[0065] According to the second modification of the first embodiment, the detection device 200 can perform highly accurate detection by combining the conditions of the first embodiment and the conditions of the first modification of the first embodiment.

[0066] Second Embodiment Next, a second embodiment will be described. In the second embodiment, differences from the first embodiment will be mainly described. In the second embodiment, descriptions of the commonalities between the first embodiment and the second embodiment will be omitted.

[0067] An overview of the second embodiment will be described using the drawings. FIG. 15 is a diagram for explaining the overview of the second embodiment. FIG. 15 shows a group of normal feature vectors. In addition, FIG. 15 indicates the low variance direction of the group of normal feature vectors with an arrow 10. FIG. 15 shows the trajectories of the feature vectors of observation target A and the feature vectors of observation target B. The feature vector of observation target A is moving in the low variance direction. Therefore, the behavior of observation target A is more likely to develop into abnormal behavior than the behavior of observation target B. Therefore, the detection device 200 weights the distance when the feature vector of observation target A moves in the low variance direction. In addition, the detection device 200 does not weight the distance because the feature vector of observation target B moves in the high variance direction.

[0068] The low variance direction is a direction identified based on the variance ratio of all normal feature vectors belonging to the normal feature vector group. For example, suppose the ratio of multiple normal feature vectors existing in the direction in which the feature vector of observation object B moves is "8". Suppose the ratio of multiple normal feature vectors existing in the direction of arrow 10 is "2". In this case, the direction of arrow 10 is the low variance direction.

[0069] The weight may be a predetermined value or the inverse of the ratio of the variance value. Alternatively, the weight may be set as follows: A matrix indicating a group of normal feature vectors is subjected to singular value decomposition. The inverse of the similarity between the vector corresponding to the maximum singular value obtained by the singular value decomposition and the feature vector of the observation target is set as the weight.

[0070] 16 is a block diagram showing the functions of the detection device according to the second embodiment. The detection device 200 further includes a weighting unit 270. Part or all of the weighting unit 270 may be implemented by a processing circuit. Alternatively, part or all of the weighting unit 270 may be implemented as a program module executed by the processor 201.

[0071] The weighting unit 270 uses the feature vector of the observation target for which the distance is calculated to detect the movement direction of the feature vector. The weighting unit 270 may detect the movement direction of the feature vector based on the feature vector of the previous observation target and the feature vector of the current observation target. If the movement direction is the same as or similar to the low-variance direction of the normal feature vector group, the weighting unit 270 weights the distance calculated based on the feature vector of the observation target.

[0072] Next, the processing executed by the detection device 200 will be described using a flowchart. FIG. 17 is a flowchart (part 1) showing an example of the processing executed by the detection device of embodiment 2. FIG. 18 is a flowchart (part 2) showing an example of the processing executed by the detection device of embodiment 2. The processing in FIGS. 17 and 18 differs from the processing in FIG. 11 in that steps S26a to S26d are executed. Therefore, steps S26a to S26d will be described in FIGS. 17 and 18. Description of the processing other than steps S26a to S26d will be omitted.

[0073] (Step S26a) The determination unit 250 determines whether the acquired image is a first image. If the acquired image is a first image, the process proceeds to step S22. If the acquired image is a second or more image, the process proceeds to step S26b. (Step S26b) The weighting unit 270 uses the feature vector of the observation target to detect the movement direction of the feature vector. (Step S26c) The weighting unit 270 determines whether the movement direction is the same as or similar to the low-variance direction of the normal feature vector group. If the condition is met, the process proceeds to step S26d. If the condition is not met, the process proceeds to step S27. (Step S26d) The weighting unit 270 weights the distance calculated in step S25. Specifically, the weighting unit 270 adds or multiplies the weight to the distance.

[0074] According to the second embodiment, the detection device 200 can improve the accuracy of detecting signs of abnormal behavior by performing weighting.

[0075] Furthermore, by executing steps S26a to S26d every time a distance is calculated, it is possible to combine the first modification of the first embodiment with the second embodiment. For example, steps S26a to S26d are executed between steps S35 and S36. Furthermore, by executing steps S26a to S26d every time a distance is calculated, it is possible to combine the second modification of the first embodiment with the second embodiment. For example, steps S26a to S26d are executed after step S45. Note that step S46 is replaced with step S26a.

[0076] Embodiment 3 Next, embodiment 3 will be described. In embodiment 3, differences from embodiment 1 will be mainly described. Furthermore, in embodiment 3, descriptions of matters common to embodiment 1 will be omitted.

[0077] An overview of the third embodiment will be described using the drawings. FIG. 19 is a diagram for explaining the overview of the third embodiment. FIG. 19 shows a group of normal feature vectors. FIG. 19 also shows a region 20 where the density of the group of normal feature vectors is high and a region 21 where the density of the group of normal feature vectors is low. FIG. 19 shows the trajectory of the feature vector of observation target C and the trajectory of the feature vector of observation target D. The feature vector of observation target C moves from the high-density region 20 in a direction other than the low-density region 21. Therefore, there is a high possibility that the behavior of observation target C will suddenly develop into abnormal behavior. On the other hand, the feature vector of observation target D moves from the low-density region 21 in a direction where the density decreases. Therefore, there is a low possibility that the behavior of observation target D will suddenly develop into abnormal behavior.

[0078] The detection device 200 weights the distance when the feature vector of the observation target C moves from the high-density region 20 in a direction other than the low-density region 21. The weight may be a predetermined value. Alternatively, the weight may be calculated using a local outlier factor method.

[0079] 20 is a block diagram showing the functions of the detection device according to the third embodiment. The detection device 200 further includes a weighting unit 271. Part or all of the weighting unit 271 may be implemented by a processing circuit. Alternatively, part or all of the weighting unit 271 may be implemented as a program module executed by the processor 201.

[0080] The weighting unit 271 determines whether the feature vector of the previously observed object was present in a high-density region in the normal feature vector group. High density and low density may be determined using criteria such as a threshold or a numerical range. If the feature vector of the previously observed object was present in a high-density region, the weighting unit 271 detects the movement direction of the feature vector using the feature vector of the current observed object. The weighting unit 271 may detect the movement direction of the feature vector based on the feature vector of the previously observed object and the feature vector of the current observed object. If the movement direction is not a low-density direction in the normal feature vector group, the weighting unit 271 weights the distance calculated based on the feature vector of the current observed object. A non-low-density direction may be expressed as a direction away from the normal feature vector group. The distance may be a value calculated using the local outlier factor method.

[0081] Next, the processing executed by the detection device 200 will be described using a flowchart. FIG. 21 is a flowchart (part 1) showing an example of the processing executed by the detection device of embodiment 3. FIG. 22 is a flowchart (part 2) showing an example of the processing executed by the detection device of embodiment 3. The processing in FIGS. 21 and 22 differs from the processing in FIG. 11 in that steps S26e to 26i are executed. Therefore, steps S26e to 26i will be described in FIGS. 21 and 22. Description of the processing other than steps S26e to 26i will be omitted.

[0082] (Step S26e) The determination unit 250 determines whether the acquired image is the first image. If the acquired image is the first image, the process proceeds to step S22. If the acquired image is the second or later image, the process proceeds to step S26f. (Step S26f) The weighting unit 271 determines whether the feature vector of the previous observation target was present in a high-density area. If the condition is met, the process proceeds to step S26g. If the condition is not met, the process proceeds to step S27. (Step S26g) The weighting unit 271 uses the feature vector of the current observation target to detect the movement direction of the feature vector. (Step S26h) The weighting unit 271 determines whether the movement direction is a low-density direction. If the movement direction is a low-density direction, the process proceeds to step S27. If the movement direction is not a low-density direction, the process proceeds to step S26i. (Step S26i) The weighting unit 271 weights the distance calculated in step S25. Specifically, the weighting unit 271 adds or multiplies the weight to the distance.

[0083] According to the third embodiment, the detection device 200 can improve the accuracy of detecting signs of abnormal behavior by performing weighting.

[0084] Furthermore, by executing steps S26e to S26i every time a distance is calculated, it is possible to combine the first modification of the first embodiment with the third embodiment. For example, steps S26e to S26i are executed between steps S35 and S36. Furthermore, by executing steps S26e to S26i every time a distance is calculated, it is possible to combine the second modification of the first embodiment with the third embodiment. For example, steps S26e to S26i are executed after step S45. Note that step S46 is replaced with step S26e.

[0085] The embodiments are merely examples, and various modifications are possible within the scope of the present disclosure. Furthermore, the features of the embodiments can be combined with each other as appropriate.

[0086] 10 Arrow, 20 Area, 21 Area, 100 Generation device, 101 Processor, 102 Volatile storage device, 103 Non-volatile storage device, 110 Storage unit, 120 Acquisition unit, 130 Extraction unit, 140 Conversion unit, 150 Output unit, 160 Learning generation unit, 200 Detection device, 201 Processor, 202 Volatile storage device, 203 Non-volatile storage device, 204 Interface, 210 Storage unit, 220 Acquisition unit, 230 Extraction unit, 240 Generation unit, 250 Judgment unit, 260 Output unit, 270 Weighting unit, 271 Weighting unit, 300 Camera, 400 Feature vector space.

Claims

1. A detection device having: an acquisition unit that acquires one or more normal feature vectors, which are vector representations of skeletal information indicating normal behavior, and a plurality of images including an object of observation; an extraction unit that extracts a plurality of pieces of skeletal information using the plurality of images; a generation unit that converts each of the plurality of pieces of skeletal information into a vector representation; and a determination unit that, based on the plurality of feature vectors of the object of observation obtained by the conversion and the normal feature vector, determines that the behavior of the object of observation is a sign of abnormal behavior if the distance between the feature vector of the object of observation and the normal feature vector increases for a predetermined period of time, or if the total value of the distance between the feature vector of the object of observation and the normal feature vector for a predetermined period of time is greater than a predetermined threshold.

2. The detection device according to claim 1, wherein the judgment unit judges that the behavior of the observed object is a sign of abnormal behavior if, based on the feature vectors of the observed object and the normal feature vector, the distance between the feature vector of the observed object and the normal feature vector increases for a predetermined period of time and the total value of the distance between the feature vector of the observed object and the normal feature vector for the predetermined period of time is greater than a predetermined threshold.

3. A detection device as described in claim 1 or 2, further comprising a weighting unit that detects a movement direction using the feature vector of the object to be observed, for which distance is calculated, and weights the distance calculated based on the feature vector of the object to be observed if the movement direction is the same as or similar to the low variance direction of the normal feature vector, and the low variance direction is a direction identified based on the variance ratio of multiple normal feature vectors.

4. A detection device as described in claim 1 or 2, further comprising a weighting unit that detects a movement direction using the feature vector of the current observation object when the feature vector of the previous observation object was present in a high-density area among the plurality of normal feature vectors, and that weights a distance calculated based on the feature vector of the current observation object when the movement direction is not a direction in which the density is low among the plurality of normal feature vectors, wherein the high density is determined based on a predetermined criterion, and the low density is determined based on the criterion.

5. A detection device according to any one of claims 1 to 4, wherein the generation unit uses a multivariate Gaussian distribution or a mixed Gaussian distribution for each piece of skeletal information to generate a probability density of the coordinate values ​​of each joint indicated by the skeletal information as a feature vector of the object to be observed.

6. A detection device according to any one of claims 1 to 4, wherein the generation unit generates a probability density of the coordinate values ​​of each joint indicated by the skeletal information using a multivariate Gaussian distribution or a mixed Gaussian distribution for each piece of skeletal information, and generates a feature vector of the object of observation based on the probability density and a Fisher vector.

7. The detection device described in claim 1 or 2, wherein the acquisition unit acquires an autoencoder trained using skeletal information of the normal behavior, and the generation unit generates, for each piece of skeletal information, the difference between the skeletal information input to the autoencoder and the skeletal information output by the autoencoder as a feature vector of the observed object.

8. The detection device according to claim 1 or 2, wherein the acquisition unit acquires an autoencoder trained using skeletal information of the normal behavior, and the generation unit inputs skeletal information into the autoencoder for each piece of skeletal information, and generates information output by the encoder of the autoencoder as a feature vector of the object of observation.

9. The detection device according to claim 1 or 2, wherein the acquisition unit acquires an autoencoder trained using skeletal information of the normal behavior, and the generation unit inputs skeletal information into the autoencoder for each piece of skeletal information, and generates a vector representation of the skeletal information output by the autoencoder as a feature vector of the object of observation.

10. A detection device according to any one of claims 1 to 9, further comprising an output unit that, when a sign of abnormal behavior is detected, outputs information indicating that a sign of abnormal behavior has been detected.

11. A detection method in which a detection device acquires one or more normal feature vectors, which are vector representations of skeletal information indicating normal behavior, and a plurality of images including an object of observation, extracts a plurality of pieces of skeletal information using the plurality of images, and converts each of the plurality of pieces of skeletal information into a vector representation, and, based on the plurality of feature vectors of the object of observation obtained by the conversion and the normal feature vector, determines that the behavior of the object of observation is a sign of abnormal behavior if the distance between the feature vector of the object of observation and the normal feature vector increases for a predetermined period of time, or if the total value of the distance between the feature vector of the object of observation and the normal feature vector for a predetermined period of time is greater than a predetermined threshold.

12. A detection program that causes a detection device to execute the following process: acquire one or more normal feature vectors, which are vector representations of skeletal information indicating normal behavior, and a plurality of images including an object of observation; use the plurality of images to extract a plurality of pieces of skeletal information; convert each of the plurality of pieces of skeletal information into a vector representation; and, based on the plurality of feature vectors of the object of observation obtained by the conversion and the normal feature vector, determine that the behavior of the object of observation is a sign of abnormal behavior if the distance between the feature vector of the object of observation and the normal feature vector increases for a predetermined period of time, or if the total value of the distance between the feature vector of the object of observation and the normal feature vector for a predetermined period of time is greater than a predetermined threshold.

Images