Action recognition apparatus, action recognition method and action recognition program

Abstract

To recognize the time series actions of a recognition object with high accuracy.SOLUTION: An action recognition apparatus has: a processor of executing a program; and a storage device of storing the program. The apparatus performs: a frequency detection process of, based on time series action recognition results of a recognition object and time intervals between the time series action recognition results, detecting the frequency of the action recognition results; a resampling process of, based on the detected frequency by the frequency detection process, normalizing the time series action recognition results in a time direction; and a determination process of determining whether reference time series actions which specify actions based on a time series string are present or not in the time series action recognition results resampled by the resampling process to output determination results.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a behavior recognition device, a behavior recognition method, and a behavior recognition program. [Background technology]

[0002] As background art in this technical field, Patent Document 1 discloses an information processing device that estimates recognition posture parameters by removing outliers from feature determination results. This information processing device, which recognizes whether an input video contains a registered action, includes a storage means for storing information on model feature points, which are feature points, and model feature amounts, which are feature amounts at the model feature points, when a model video including a model for recognizing the action is treated as a three-dimensional image plane and time, a first acquisition means for acquiring the input video, a first feature point extraction means for extracting input feature points, which are feature points for recognizing the action, from the input video, when the input video acquired by the first acquisition means is treated as a three-dimensional image plane and time, and an input feature amount, which is a feature amount at the input feature points extracted by the first feature point extraction means. The recognition pose parameter is estimated by comprising a first feature extraction means, a feature comparison means for comparing the input features extracted by the first feature extraction means with the model features stored in the storage means, and generating candidate corresponding feature point pairs as a set of feature points having similar features, a posture estimation means for removing outliers from the candidate corresponding feature point pairs obtained as a result of the comparison by the feature comparison means, estimating the posture of the model on the input video, and obtaining a group of recognition corresponding feature point pairs corresponding to the posture of the model, and a recognition result generation means for generating a recognition result based on the estimation result of the posture of the model obtained by the posture estimation means and the group of recognition corresponding feature point pairs. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-140009 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the above-mentioned Patent Document 1, features are extracted from an input video in three dimensions, that is, an image plane and time, and the number of recognition-corresponding feature point pair groups becomes enormous when considering frequency fluctuations in the time direction of the input video. Therefore, no consideration is given to temporarily increasing or decreasing the limited recognition-corresponding feature point pair groups to improve recognition accuracy.

[0005] An object of the present invention is to recognize the time-series behavior of a recognition target with high accuracy. [Means for solving the problem]

[0006] A behavior recognition device according to one aspect of the invention disclosed in the present application has a processor that executes a program and a storage device that stores the program, and is characterized in that the behavior recognition device executes a frequency detection process that detects a frequency of a behavior recognition result based on a time-series behavior recognition result of a recognition target and a time interval between the time-series behavior recognition results, a resampling process that normalizes the time-series behavior recognition result in the time direction based on the frequency detected by the frequency detection process, and a determination process that determines whether or not a reference time-series behavior that defines behavior by a time-series signal sequence exists in the time-series behavior recognition result resampled by the resampling process, and outputs a determination result. Effect of the Invention

[0007] According to the representative embodiment of the present invention, it is possible to recognize the time-series behavior of a recognition target with high accuracy. Problems, configurations and effects other than those described above will become apparent from the following description of the embodiment. [Brief description of the drawings]

[0008] [Figure 1]FIG. 1 is a diagram illustrating an example of a system configuration of a behavior recognition system according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the hardware configuration of a computer. [Diagram 3] FIG. 3 is an explanatory diagram illustrating an example of learning data. [Figure 4] FIG. 4 is a block diagram of an example of a functional configuration of the behavior recognition system according to the first embodiment. [Diagram 5] FIG. 5 is a block diagram showing an example of a detailed functional configuration of the skeleton information processing unit. [Figure 6] FIG. 6 is an explanatory diagram showing a detailed method of calculating the joint angle executed by the joint angle calculation unit. [Figure 7] FIG. 7 is an explanatory diagram showing an example of a detailed calculation method of the movement amount between frames executed by the movement amount calculation unit. [Figure 8] FIG. 8 is an explanatory diagram showing a detailed method of normalizing skeletal information executed by the normalizing unit. [Figure 9] FIG. 9 is an explanatory diagram showing a detailed example of a teacher signal held in the teacher signal DB. [Figure 10] FIG. 10 is an explanatory diagram illustrating an example of a frame-unit behavior recognition result table. [Figure 11] FIG. 11 is an explanatory diagram illustrating an example of a time-series behavior recognition result graph. [Figure 12] FIG. 12 is an explanatory diagram showing the time-series behavior recognition result before and after the correction. [Figure 13] FIG. 13 is an explanatory diagram illustrating a detailed example of processing performed by the determination unit. [Figure 14] FIG. 14 is a flowchart of a detailed example of a learning process performed by the server (learning device) according to the first embodiment. [Figure 15] FIG. 15 is a flowchart of a detailed processing procedure example of the skeleton information processing according to the first embodiment. [Figure 16] FIG. 16 is a flowchart of an example of a behavior recognition process procedure by the client (behavior recognition device) according to the first embodiment. [Figure 17]FIG. 17 is a block diagram of an example of a functional configuration of the behavior recognition system according to the second embodiment. As illustrated in FIG. [Figure 18] FIG. 18 is a flowchart of an example of a behavior recognition process procedure by a client (behavior recognition device) according to the second embodiment. [Figure 19] FIG. 19 is a block diagram of an example of a functional configuration of the behavior recognition system according to the third embodiment. As illustrated in FIG. [Figure 20] FIG. 20 is a flowchart of a detailed example of a learning process performed by the server (learning device) according to the third embodiment. [Figure 21] FIG. 21 is a flowchart of an example of a behavior recognition process procedure by a client (behavior recognition device) according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all the drawings for explaining the embodiment, the same members are generally given the same reference numerals, and the repeated description thereof will be omitted. In addition, in the following embodiment, the components (including element steps, etc.) are not necessarily essential, unless otherwise specified or considered to be obviously essential in principle. In addition, when it is said that "consists of A", "consists of A", "has A", or "includes A", it is needless to say that other elements are not excluded, except when it is specifically specified that only the element is included. Similarly, in the following embodiment, when referring to the shape, positional relationship, etc. of the components, etc., it is intended to include those that are substantially similar or similar to the shape, etc., except when it is specifically specified or considered to be obviously not essential in principle.

[0010] The designations "first," "second," "third," and the like in this specification are given to identify components, and do not necessarily limit the number, order, or content. Furthermore, numbers for identifying components are used in different contexts, and a number used in one context does not necessarily indicate the same configuration in another context. Furthermore, a component identified by a certain number does not prevent the component from also having the function of a component identified by another number.

[0011] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings, etc. may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings, etc. EXAMPLES

[0012] <Figure 1 Activity recognition system> 1 is an explanatory diagram illustrating an example of a system configuration of a behavior recognition system according to a first embodiment. The behavior recognition system 100 includes a server 101 and one or more clients 102. The server and clients are communicatively connected via a network 105 such as the Internet, a local area network (LAN), or a wide area network (WAN). The server 101 is a computer that manages the clients 102. The client 102 is a computer that is connected to a sensor 103 and acquires data from the sensor 103.

[0013] The sensor 103 detects data to be analyzed from the analysis environment. The sensor 103 is, for example, a camera that captures still images or videos. The sensor 103 may also detect sound or smell. The teacher signal DB 104 is a database that holds combinations of learning data (human skeleton information and joint angles) and behavior information (for example, human postures and movements such as "standing" and "falling") as teacher signals. The teacher signal DB 104 is a database that holds human behavior patterns in the time direction as a reference waveform group. The teacher signal DB 104 may be stored in the server 101, or may be connected to a computer that can communicate with the server 101 or the client 102 via the network 105.

[0014] The behavior recognition system 100 has a learning function using a teacher signal DB 104 and a behavior recognition function using a behavior classification model obtained by the learning function. The behavior classification model is a learning model for classifying the behavior of a recognition target such as a person or an animal. The learning function and the behavior recognition function may be implemented in either the server 101 or the client 102 as long as they are implemented in the behavior recognition system 100. For example, the server 101 may implement the learning function, and the client 102 may implement the behavior recognition function. Alternatively, the server 101 may implement the learning function and the behavior recognition function, and the client 102 may transmit data from the sensor 103 to the server 101 and receive a behavior recognition result by the behavior recognition function from the server 101.

[0015] Also, the client 102 may implement a learning function and a behavior recognition function, and the server 101 may manage the behavior classification model and behavior recognition results from the client 102. A computer that implements the learning function is referred to as a learning device, and a computer that implements at least the behavior recognition function out of the learning function and the behavior recognition function is referred to as a behavior recognition device. Also, while a client-server type behavior recognition system 100 is given as an example in FIG. 1, a standalone type behavior recognition device may also be used. In the first embodiment, for convenience of explanation, a behavior recognition system 100 in which the server 101 implements a learning function (learning device) and the client 102 implements a behavior recognition function (behavior recognition device) will be described as an example.

[0016] <Figure 2 Example of computer hardware configuration> FIG. 2 is a block diagram showing an example of a hardware configuration of a computer (server 101, client 102). The computer 200 has a processor 201, a storage device 202, an input device 203, an output device 204, and a communication interface (communication IF) 205. The processor 201, the storage device 202, the input device 203, the output device 204, and the communication IF 205 are connected by a bus 206. The processor 201 controls the computer 200. The storage device 202 is a working area for the processor 201. The storage device 202 is a non-transient or temporary recording medium that stores various programs and data. Examples of the storage device 202 include a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive), and a flash memory. The input device 203 inputs data. Examples of the input device 203 include a keyboard, a mouse, a touch panel, a numeric keypad, and a scanner. The output device 204 outputs data. The output device 204 may be, for example, a display, a printer, or a speaker. The communication IF 205 is connected to the network 105 and transmits and receives data.

[0017] <Figure 3 Learning data> 3 is an explanatory diagram showing an example of learning data. Learning data 380 is composed of at least skeletal information 320 out of skeletal information 320 and joint angles 370 for each subject. Skeletal information 320 is detected based on analysis target data acquired from sensor 103. Joint angles 370 are calculated based on skeletal information 320. Learning data 380 for one subject is composed of, for example, a combination of skeletal information 320 and joint angles 370 obtained from each of a plurality of time-series frames in which the subject is the subject.

[0018] The skeleton information 320 has a name 321, an x ​​coordinate value 322 on the x axis, and a y coordinate value 323 on the y axis perpendicular to the x axis for each of the multiple (18 in this example) skeleton points 300 to 317. The joint angle 370 also has a name 371 for each of the multiple (18 in this example) skeleton points 300 to 317. In the name 371, ∠abc (a, b, c are the skeleton point names 321) is the joint angle 370 of the skeleton point b formed by the line segments ab and bc. The skeleton information 320 may include, for example, the joints of the fingers. The joint angles 370 may also include other joint angles 370.

[0019] 3, the coordinate values ​​of the skeleton points 300 to 317 are two-dimensional position information (a combination of x and y coordinate values), but they may be three-dimensional position information. Specifically, for example, a z coordinate value on the z axis (for example, the depth direction) perpendicular to the x and y axes may be added.

[0020] <Figure 4: Example of functional configuration of the behavior recognition system 100> 4 is a block diagram showing an example of a functional configuration of the behavior recognition system 100 according to the first embodiment. The server 101 includes a teacher signal acquisition unit 401, a skeleton information processing unit 402, and a behavior learning unit 403. The client 102 includes a skeleton detection unit 410, a skeleton information processing unit 411, a frame-by-frame behavior recognition unit 412, a register 413, a frequency detection unit 414, a noise removal unit 415, a resampling unit 416, and a determination unit 417.

[0021] Specifically, these are realized by, for example, having the processor 201 execute a program stored in the storage device 202 shown in Fig. 2. First, an example of the functional configuration of the server 101 side will be described.

[0022] The teacher signal acquisition unit 401 acquires one or more teacher signals to be used for learning the teacher signal acquired from the teacher signal DB 104, and outputs the selected teacher signal to the behavior learning unit 403. The teacher signal is composed of an image, skeletal information 320, and behavior information. Details of the teacher signal will be described with reference to FIG. 9.

[0023] The skeleton information processing unit 402 processes the skeleton information 320 from the acquired teacher signal. Specifically, for example, the skeleton information processing unit 402 calculates the joint angle 370 and the amount of movement between frames from the skeleton information 320. In addition, the skeleton information processing unit 402 removes absolute position information from the skeleton information 320 and performs normalization so that the size of the skeleton information 320 becomes constant. Then, the skeleton information processing unit 402 outputs the joint angle 370, the amount of movement between frames, and the normalized skeleton information 320 to the behavior learning unit 403.

[0024] <Figure 5 Example of skeletal information processing> 5 is a block diagram showing an example of a detailed functional configuration of the skeleton information processing unit 402, 411. The skeleton information processing unit 402, 411 includes a joint angle calculation unit 501, a movement amount calculation unit 502, and a normalization unit 503.

[0025] The joint angle calculation unit 501 calculates the joint angle 370 from the interpolated skeletal information 320 of the acquired teacher signal, and outputs it to the behavior learning unit 403 and the frame-by-frame behavior recognition unit 412 via the movement amount calculation unit 502 and the normalization unit 503.

[0026] The movement amount calculation unit 502 calculates the amount of movement between frames from the interpolated skeletal information 320 of the acquired teacher signal, and outputs the calculated amount of movement to the action learning unit 403 and the frame-by-frame action recognition unit 412 via the normalization unit 503 .

[0027] The normalization unit 503 removes absolute position information with respect to the interpolated skeletal information 320 from the acquired teacher signal, performs normalization so that the size of the interpolated skeletal information 320 becomes constant, and outputs the result to the behavior learning unit 403 and the frame-by-frame behavior recognition unit 412.

[0028] Returning to FIG. 4, the behavior learning unit 403 learns the joint angle 370, the movement amount, the normalized skeleton information 320 acquired from the skeleton information processing unit 402, and the behavior information in the teacher signal acquired from the teacher signal DB 104 by associating them with each other. Specifically, for example, the behavior learning unit 403 generates a behavior classification model by machine learning using data obtained by linking the joint angle 370, the movement amount, and the normalized skeleton information 320 acquired from the skeleton information processing unit 402 as explanatory variables and the behavior information in the teacher signal acquired from the teacher signal DB 104 as objective variables. The behavior learning unit 403 outputs the behavior classification model generated by learning to the frame-by-frame behavior recognition unit 412.

[0029] Next, an example of a functional configuration on the client 102 side will be described. The skeleton detection unit 410 acquires the analysis target data from the sensor 103. The time when the analysis target data is acquired is also acquired. Hereinafter, this time is referred to as the acquisition time. The skeleton information 320 of the person reflected in the acquired analysis target data is detected and output to the skeleton information processing unit 411. The detection of the skeleton information 320 may use a neural network (NN) capable of estimating the skeleton information 320 of the person generated by machine learning, or a marker may be added to the skeleton points of the person to be detected and the skeleton information 320 may be detected from the marker position reflected in the image, and the method of detecting the skeleton information 320 is not limited.

[0030] The skeleton information processing unit 411 has the same function as the skeleton information processing unit 402. The skeleton information processing unit 411 executes the same processing as the skeleton information processing unit 402 on the skeleton information 320 detected by the skeleton detection unit 410, and outputs the joint angles 370, the movement amount between frames, and the normalized skeleton information 320 to the frame-by-frame action recognition unit 412.

[0031] The frame-by-frame behavior recognition unit 412 recognizes the behavior of the person captured in the analysis target data acquired from the sensor 103 on a frame-by-frame basis based on the behavior classification model generated by the behavior learning unit 403, the joint angle 370, the amount of movement between frames, and the normalized skeletal information 320, and outputs the recognition result to the register 413. Specifically, for example, the frame-by-frame behavior recognition unit 412 inputs the joint angle 370 obtained from the analysis target data, the amount of movement between frames, and the normalized skeletal information 320 into the behavior classification model, and outputs a predicted value indicating the behavior of the person captured in the analysis target data to the register 413 as the behavior recognition result.

[0032] The register 413 holds the behavior recognition result acquired from the frame-unit behavior recognition unit 412 together with the acquisition time, and outputs the held acquisition time and behavior recognition result to the frequency detection unit 414. The register 413 is, for example, a FIFO (First in First out) type register, and may hold all acquired behavior recognition results, or may hold behavior recognition results for a predetermined time period and delete behavior recognition results exceeding the predetermined time period starting from the oldest information, and the type of the register 413 is not limited.

[0033] The frequency detection unit 414 detects the frequency of the behavior recognition result from the multiple acquisition times and behavior recognition results acquired from the register 413, and outputs the detection frequency and the behavior recognition result of the detection source to the noise removal unit 415. Specifically, for example, the frequency detection unit 414 acquires the time interval between successive acquisition times. When multiple time intervals are acquired, the frequency detection unit 414 averages the multiple time intervals. The frequency detection unit 414 calculates the reciprocal of the time interval as the detection frequency. Alternatively, the frequency detection unit 414 may calculate and average the detection frequency for each time interval.

[0034] The noise removal unit 415 corrects the behavior recognition results determined to be erroneous based on the behavior recognition results for multiple time points acquired from the frequency detection unit 414, and outputs the corrected behavior recognition results for multiple time points and the detection frequency to the resampling unit 416. Correcting the behavior recognition results by the noise removal unit 415 improves the resampling accuracy by the resampling unit 416. Note that when priority is given to processing speed, noise removal by the noise removal unit 415 may be omitted.

[0035] The resampling unit 416 normalizes the corrected behavior recognition result acquired from the noise removal unit 415 in the time direction based on the frequency detected by the frequency detection unit 414. Specifically, for example, the resampling unit 416 resamples the corrected behavior recognition result to a predetermined target frequency, and outputs the resampled corrected behavior recognition result to the determination unit 417 as a resampled behavior recognition result.

[0036] More specifically, the resampling unit 416 compares a predetermined target frequency with the detected frequency, and performs downsampling if the detected frequency is equal to or higher than the predetermined target frequency, and performs upsampling if the detected frequency is lower than the predetermined target frequency. Downsampling is a process of resampling a signal sampled at a certain frequency at a lower frequency. Upsampling is a process of resampling a signal sampled at a certain frequency at a higher frequency.

[0037] The determination unit 417 acquires a reference waveform group from the teacher signal DB 104, and acquires a resampled behavior recognition result from the resampling unit 416. The determination unit 417 determines whether or not the acquired resampled behavior recognition result includes a reference waveform group, and calculates a behavior recognition result in the time direction (hereinafter, a time-series behavior recognition result) that is a determination result.

[0038] 12, when the frame groups in which the number of frames before resampling 1210 is "7" and "8" are downsampled, "0" is obtained as the corrected behavior recognition result 1222. The determination unit 417 associates "0" of the corrected behavior recognition result 1222 with the learning data stored in the teacher signal DB 104 as behavior information, and calculates a time-series behavior recognition result.

[0039] 12, for example, when the frame number before resampling 1210 is upsampled to "7", the corrected behavior recognition result 1232 of "0" and "0" for the frame numbers 1230 of "7" and "8" is obtained. The determination unit 417 calculates a time-series behavior recognition result by associating the "0" and "0" of the corrected behavior recognition result 1232 with the learning data stored in the teacher signal DB 104 as behavior information.

[0040] In both downsampling and upsampling, the client 102 performs a process of making the frequency constant as a predetermined target value by resampling, so that the learning data used for learning the behavior classification model only needs to be prepared for one frequency for one time-series behavior. This means that when storing the skeletal information and joint angles of a group of frames as learning data in the teacher signal DB 104, it is not necessary to prepare multiple pieces of frequency information between frames. Specifically, since it is sufficient to prepare the skeletal information and joint angles of one frequency as learning data, the amount of learning data is reduced, and the reference waveform group prepared in the teacher signal DB 104 is also reduced.

[0041] Therefore, there is no need to collect a large amount of training data, such as by preparing a large amount of training data or by performing data augmentation to pad the frequency-shifted training data, and this makes it possible to reduce the cost of collecting training data and the training time.

[0042] Furthermore, the client 102 may use the time-series behavior recognition result to control a device connected to the client 102. For example, if the analysis environment in which the sensor 103 is deployed is a factory, the server 101 applies the time-series behavior recognition result to monitoring the work of workers in the factory, inspecting defects in products, etc. If the analysis environment is a train, the server 101 applies the time-series behavior recognition result to monitoring passengers and on-board equipment in the train, detecting disasters such as fires, etc.

[0043] <Figure 6 Example of joint angle calculation> 6 is an explanatory diagram showing a detailed calculation method of the joint angle 370 executed by the joint angle calculation unit 501. The joint angle calculation unit 501 calculates the joint angle θ at three connected skeleton points 600 to 602. Skeleton information 620 of the skeleton points 600 to 602 is defined as position vectors O, A, and B with an origin 630 as a reference. The joint angle calculation unit 501 calculates a relative vector with the skeleton point 600 as the origin as shown in the following expressions (1) and (2), and calculates the joint angle θ by calculating the arc cosine as shown in the following expression (4) when the following expression (3) is established from the calculated vector.

[0044]

number

[0045] <Figure 7 Example of calculation of movement between frames> 7 is an explanatory diagram showing an example of a detailed calculation method of the movement amount between frames executed by the movement amount calculation unit 502. In calculating the movement amount between frames, the movement amount calculation unit 502 uses skeletal information 701 of the Nth frame and skeletal information 702 of the NMth frame for the same subject. N and M are integers equal to or greater than 1, and N>M. The value of M can be set arbitrarily. As shown in the following formulas (5) to (7), the movement amount calculation unit 502 calculates the distances between the same skeleton points 300 to 317 of the same person shown between the frames. The movement amounts of the 18 skeleton points 300 to 317 between frames become the movement amounts for that person between frames.

[0046]

number

[0047] However, the amount of movement between frames calculated by the movement amount calculation unit 502 is not limited to this. As shown in the following formula (8), the movement amount calculation unit 502 may calculate the distance between the same skeleton points 300 to 317 of the same person shown between the frames, and use the sum of the amounts of movement between frames of all 18 skeleton points 300 to 317 as the amount of movement between frames for that person.

[0048]

number

[0049] Furthermore, the movement amount calculation unit 502 may use the center of gravity skeletal information 711 and the center of gravity skeletal information 712, which are the centers of gravity, from the skeleton information 701 of the nth frame and the skeleton information 702 of the nm-th frame. Specifically, for example, the movement amount calculation unit 502 may calculate the center of gravity for each person as shown in the following formulas (9) to (10), and calculate the amount of movement between frames for the person with respect to the calculated center of gravity as shown in the following formula (11).

[0050]

number

[0051] <Figure 8 Example of normalization> 8 is an explanatory diagram showing a detailed method of normalizing the skeleton information 320 executed by the normalization unit 503. First, the normalization unit 503 (a) calculates the center of gravity from all or part of the skeleton information 320, and (b) converts the center of gravity into relative coordinates with the origin being the origin. After that, the normalization unit 503 (d) divides the position information of each skeleton point of the skeleton information 320 by (c) the length L of the diagonal of the smallest rectangle that encloses the 18 skeleton points 300 to 317. When the skeleton information 320 obtained in (d) is used as a teacher signal, the position information of the skeleton points 300 to 317 after the division is also incorporated.

[0052] For example, if the normalization unit 503 is not executed, when learning is executed for skeleton detection and behavior classification for a behavior such as "a person who is 180 cm tall sits at point A," it may be determined that "the person does not sit at any point other than A" or "people who are not 180 cm tall do not sit." In order to eliminate such limitations and make behavior classification more versatile, the normalization unit 503 executes normalization of the skeleton information 320 to remove absolute position information within the image and information related to the size of the skeleton.

[0053] <Figure 9 Teacher signals held in the teacher signal DB 104> 9 is an explanatory diagram showing a detailed example of a teacher signal held by the teacher signal DB 104. For a person appearing in an image 900 (a) that is the data to be analyzed, a combination of skeletal information 320A (b), joint angles 370 (not shown), and behavioral information 901 (c) associated with the skeletal information 320A ("standing") is a teacher signal. Similarly, for a person appearing in an image 910 (a) that is the data to be analyzed, a combination of skeletal information 320B (b), joint angles 370 (not shown), and behavioral information 911 (c) associated with the skeletal information 320B ("falling") is a teacher signal.

[0054] <Figure 10 Frame-by-frame action recognition result table> FIG. 10 is an explanatory diagram showing an example of a frame-based behavior recognition result table. The frame-based behavior recognition result table 1000 is a table accessible by the frame-based behavior recognition unit 412. The frame-based behavior recognition result table 1000 has, as fields, behavior 1001 and corresponding numerical value 1002. The behavior 1001 is a recognized behavior of a person appearing in the data to be analyzed. In FIG. 10, the behavior 1001 is classified, for example, into "standing", "sitting", "standing and raising hand to the left", "standing and raising hand to the right", "sitting and raising hand to the left", and "sitting and raising hand to the right".

[0055] The corresponding numerical value 1002 is a numerical value assigned to the behavior 1001. In Fig. 10, for example, "0" is assigned to "stand", "1" is assigned to "sit", "2" is assigned to "stand and raise hand to the left", "3" is assigned to "stand and raise hand to the right", "4" is assigned to "sit and raise hand to the left", and "5" is assigned to "sit and raise hand to the right" as the corresponding numerical value 1002.

[0056] The frame-by-frame action recognition unit 412 outputs a corresponding numerical value 1002 of the action 1001 identified on a frame-by-frame basis to the register 413 as an action recognition result.

[0057] <Figure 11 Time series behavior recognition result graph> 11 is an explanatory diagram showing an example of a time-series behavior recognition result graph. The time-series behavior recognition result graph 1100 shows the change over time of the behavior recognition result in units of frames. In the time-series behavior recognition result graph 1100, the horizontal axis is the number of frames in ascending order, and an increase in the number of frames corresponds to the passage of time. The vertical axis is the behavior recognition result, i.e., the corresponding numerical value 1002.

[0058] <Figure 12 Time series behavior recognition results> 12 is an explanatory diagram showing time-series behavior recognition results before and after correction. A pre-resample time-series behavior recognition result 1201 is a time-series behavior recognition result (corresponding numerical value 1002) before and after correction before resampling by the resampling unit 416. Specifically, for example, the pre-resample time-series behavior recognition result 1201 has a frame number 1210, a behavior recognition result 1211, and a corrected behavior recognition result 1212.

[0059] The frame number 1210 is an ascending value starting from 1. The frame number 1210 is the value on the horizontal axis of the time-series behavior recognition result graph 1100. For convenience of explanation, the frame numbers are also ascending values ​​starting from 1. Therefore, the frame number 1210 is assumed to match the frame number.

[0060] The behavior recognition result 1211 is the time-series corresponding numerical value 1002 before correction by the resampling unit 416. The behavior recognition result 1211 becomes the value on the vertical axis of the time-series behavior recognition result graph 1100.

[0061] The corrected behavior recognition result 1212 is a corrected value of the time-series corresponding numerical value 1002 by the resampling unit 416, that is, a value after noise removal by the noise removal unit 415.

[0062] The pre-resample time-series behavior recognition result 1201 indicates behavior recognition results 1211 and corrected behavior recognition results 1212 for frame numbers 1210 from "7" to "17" ("1" to "6" are omitted) in the time-series behavior recognition result graph 1100. The noise removal unit 415 corrects the behavior recognition result to be corrected based on a plurality of behavior recognition results including the behavior recognition result to be corrected among the acquired behavior recognition results 1211.

[0063] For example, the behavior recognition result "4" for the frame number "17" is set as the behavior recognition result to be corrected. In this case, the behavior recognition result "3" of the most frequent value [3,3,4], which is the behavior recognition result 1213 of the sequence of three past frames ([15,16,17]) including the frame number "17", is corrected as the behavior recognition result for the frame number "17". In other words, the behavior recognition result "4" for the frame number "17" is corrected to the corrected behavior recognition result "3" by the noise removal unit 415.

[0064] In the above example, the frame sequence of the behavior recognition result 1213 is the last three consecutive frames [15, 16, 17], but it may be two or more. In addition, since the behavior recognition result to be corrected is corrected by the mode, if there are multiple behavior recognition results with the most frequent value, the noise removal unit 415 extends the frame sequence of the behavior recognition result 1213 in the past direction and corrects it again until there is only one behavior recognition result with the most frequent value.

[0065] In the above example, the frame sequence of the behavior recognition result 1213 is the last three consecutive frames, but in the above example, the frame sequence of the behavior recognition result 1213 may be the last three discrete frames, such as "13", "15", and "17". In addition, the frame sequence is not limited to the mode, and may be other statistical values ​​such as the average value or the median value. The above-mentioned correction method is also applicable to the corrected behavior recognition results 1222 and 1232.

[0066] The downsampled time-series behavior recognition result 1202 is a time-series behavior recognition result before and after correction when downsampling is performed by the resampling unit 416 on the pre-resampled time-series behavior recognition result 1201 .

[0067] In the down-sampling time-series behavior recognition result 1202, the detection frequency by the frequency detection unit 414 is assumed to be 1 / n times the predetermined target frequency (i.e., the sampling period is n times). n is an integer equal to or greater than 2. In FIG. 12, n=2.

[0068] The resampling unit 416 may resample the modified behavior recognition result 1212 before resampling at 1 / n times a predetermined target frequency to obtain the behavior recognition result 1221, or may select a value of the modified behavior recognition result 1212 that is most frequent in the time direction from multiple values ​​of the modified behavior recognition result 1212 before resampling to obtain the behavior recognition result 1221. The resampling method used in downsampling is not limited.

[0069] The downsampled time-series behavior recognition result 1202 has a frame number 1220 , a behavior recognition result 1221 , and a corrected behavior recognition result 1222 .

[0070] Frame number 1220 is the down-sampling result of frame number 1210 from "7" to "17" ("1" to "6" are omitted). In this example, since the detection frequency is n (=2) times the target frequency, frame number 1220 is a value obtained by sampling odd or even numbers of frame number 1210. In the example of FIG. 12, frame number 1220 is "4" to "9" ("1" to "3" are omitted).

[0071] The behavior recognition result 1221 is a downsampling result of the behavior recognition result 1211 in the frame number 1210 from "7" to "17" ("1" to "6" are omitted). Specifically, for example, the behavior recognition result 1221 becomes a corresponding numerical value 1002 in the frame number 1220 downsampled from the frame number 1210.

[0072] The corrected behavior recognition result 1222 is a corrected value of the time-series behavior recognition result 1221 downsampled by the resampling unit 416, that is, a value after noise removal by the noise removal unit 415.

[0073] The upsampled time-series behavior recognition result 1203 is a time-series behavior recognition result before and after correction when upsampling is performed by the resampling unit 416 on the pre-resampled time-series behavior recognition result 1201 .

[0074] In the upsampling time-series behavior recognition result 1203, it is assumed that the detection frequency by the frequency detection unit 414 is n times the predetermined target frequency (that is, the sampling period is 1 / n times). In FIG.

[0075] The upsampling time-series action recognition result 1203 has a frame number 1230 , an action recognition result 1231 , and a corrected action recognition result 1232 .

[0076] Frame number 1230 is the upsampling result of frame number 1210 from "7" to "17" ("1" to "6" are omitted). In this example, since the detection frequency is n (=2) times the target frequency, frame number 1230 is increased to twice frame number 1210. In the example of FIG. 12, frame number 1230 becomes "7" to "28" ("1" to "6" are omitted). That is, "7" of frame number 1210 is corrected to "7" and "8" of frame number 1230, "8" of frame number 1210 is corrected to "9" and "10" of frame number 1230, ..., "17" of frame number 1210 is corrected to "27" and "28" of frame number 1230.

[0077] The behavior recognition result 1231 is an upsampling result of the behavior recognition result 1211 in the frame number 1210 from "7" to "17" ("1" to "6" are omitted). Specifically, for example, the behavior recognition result 1231 becomes a corresponding numerical value 1002 in the frame number 1230 upsampled from the frame number 1210.

[0078] Specifically, for example, since "7" in the frame number 1210 is corrected to "7" and "8" in the frame number 1230, the value of the behavior recognition result 1231 for "7" in the frame number 1230 and the value of the behavior recognition result 1231 for "8" in the frame number 1230 both become "0", the value of the behavior recognition result 1211 for "7" in the frame number 1210. The same process is performed for the values ​​of the behavior recognition result 1231 for the frame numbers 1230 of "9" and onwards.

[0079] The corrected behavior recognition result 1232 is a corrected value of the time-series behavior recognition result 1231 upsampled by the resampling unit 416, that is, a value after noise removal by the noise removal unit 415.

[0080] <Figure 13 Example of similar waveform comparison> Fig. 13 is an explanatory diagram showing a detailed processing example of the determination unit 417. The teacher signal DB 104 has a first reference waveform 1301 and a second reference waveform 1302 as examples of reference waveforms, and is read out by the determination unit 417. In Fig. 13, a process in which the determination unit 417 compares each of the first reference waveform 1301 and the second reference waveform 1302 with the down-sampling time-series action recognition result 1202 will be described.

[0081] The first reference waveform 1301 has a signal sequence for six frames of reference time-series actions 1311 and 1312. A signal sequence 1310 of the reference time-series action 1311 is [2, 2, 2, 3, 3, 3], which matches a signal sequence 1223 of the corrected action recognition result 1222 in which the value of the frame number 1220 in the down-sampling time-series action recognition result 1202 is from “6” to “11”.

[0082] A signal sequence matching the signal sequence [4,4,4,5,5,5] of the reference time-series behavior 1312 does not exist in the modified behavior recognition result 1222 where the frame number 1220 value in the downsampled time-series behavior recognition result 1202 is between "4" and "15".

[0083] Therefore, the determination unit 417 outputs a determination result indicating that the signal sequence 1310 of the reference time-series behavior 1311 and the signal sequence 1223 of the corrected behavior recognition result 1222 match each other.

[0084] The second reference waveform 1302 has a signal sequence for two frames of reference time-series actions 1321 and 1322. A signal sequence 1320 of the reference time-series action 1321 is [2,3], which matches a signal sequence 1224 of a modified action recognition result 1222 in which the value of the frame number 1220 in the down-sampling time-series action recognition result 1202 is from “8” to “9”, and a signal sequence 1225 of a modified action recognition result 1222 in which the value of the frame number 1220 is from “14” to “15”.

[0085] A signal sequence matching the signal sequence [4,5] of the reference time-series behavior 1322 does not exist in the modified behavior recognition result 1222 where the frame number 1220 value in the downsampled time-series behavior recognition result 1202 is between "4" and "15."

[0086] Therefore, the determination unit 417 outputs a determination result indicating that the signal sequence 1320 of the reference time-series behavior 1321 and the signal sequences 1224 and 1225 of the corrected behavior recognition result 1222 match each other.

[0087] In the first reference waveform 1301, the signal length of the reference time-series behaviors 1311 and 1312 is set to six frames, and in the second reference waveform 1302, the signal length of the reference time-series behaviors 1321 and 1322 is set to two frames, but the signal length may be two frames or more.

[0088] In addition, in the first reference waveform 1301 and the second reference waveform 1302, independent reference time-series actions 1311, 1312, 1321, and 1322 are defined, but a connected signal sequence obtained by connecting signal sequences of a plurality of reference time-series actions may be used as a reference waveform. Specifically, for example, a signal sequence [2,2,2,3,3,3,2,3] obtained by connecting signal sequences 1310 and 1320 of the reference time-series actions 1311 and 1321 may be defined as a series of reference time-series actions, such as "stand up and wave 1" -> "stand up and wave 2".

[0089] In addition, in the above-mentioned comparison process, the judgment unit 417 outputs a judgment result indicating that the signal sequences are similar only when the signal sequences match, but the judgment unit 417 may output a judgment result indicating that the signal sequences are similar when more than half of the signal sequence values ​​match in the reference time-series actions.

[0090] For example, the signal sequence 1310 of the reference time-series behavior 1311 of the first reference waveform 1301 is [2, 2, 2, 3, 3, 3], which matches the signal sequence 1226 of the corrected behavior recognition result 1222 in which the value of the frame number 1220 in the downsampled time-series behavior recognition result 1202 is from “7” to “12”.

[0091] Therefore, the determination unit 417 outputs a determination result indicating that the signal sequence 1310 of the reference time-series behavior 1311 and the signal sequence 1226 of the corrected behavior recognition result 1222 are similar to each other.

[0092] In the above example, the determination unit 417 compares the first reference waveform 1301 and the second reference waveform 1302 with the corrected behavior recognition result 1222. However, the determination unit 417 may compare a part of the signal sequence of the first reference waveform 1301 and the second reference waveform 1302 with the corrected behavior recognition result 1222.

[0093] For example, it is assumed that a part of the signal sequence [2,2,3,3] is set as an element of the time-series behavior in the signal sequence [2,2,2,3,3,3] of the reference time-series behavior 1311 of the first reference waveform 1301. In this case, the determination unit 417 determines whether or not there is a signal sequence in the corrected behavior recognition result 1222 that matches the part of the signal sequence [2,2,3,3]. In this way, the determination unit 417 can execute the determination process with the corrected behavior recognition result 1222 by using a part of the existing reference waveform.

[0094] The partial signal sequence is a sequence of two or more signal values. The partial signal sequence may be a signal sequence that matches the beginning of the original signal sequence, a signal sequence that matches the end of the original signal sequence, or a signal sequence that matches the middle part of the original signal sequence. In this way, by applying the partial signal sequence, it becomes possible to identify a behavior that is close to the reference time-series behavior from the corrected behavior recognition result 1222, even if it does not completely match the signal sequence of the reference time-series behavior 1311.

[0095] <Figure 14 Learning process> 14 is a flowchart illustrating a detailed example of a procedure of a learning process by the server 101 (learning device) according to the embodiment 1. The server 101 acquires, by the teacher signal acquisition unit 401, one or more teacher signals to be used for learning from the teacher signal acquired from the teacher signal DB 104 (step S1400).

[0096] The server 101 executes skeleton information processing for each teacher signal by the skeleton information processing unit 402 (step S1401). Specifically, for example, the server 101 executes processing by the joint angle calculation unit 501, the movement amount calculation unit 502, and the normalization unit 503.

[0097] <Figure 15 Skeleton information processing> 15 is a flowchart showing a detailed example of a processing procedure of the skeleton information processing according to the first embodiment. The server 101 calculates the joint angle 370 from the skeleton information 320 in each teacher signal by the joint angle calculation unit 501 (step S1501). Next, the server 101 calculates the movement amount between frames from the skeleton information 320 in each teacher signal by the movement amount calculation unit 502 (step S1502).

[0098] Then, the server 101 causes the normalization unit to remove absolute position information for each teacher signal with respect to the skeleton information 320, and executes normalization so that the size of the skeleton information 320 becomes constant (step S1503). As a result, for the teacher signal, the joint angle 370, the amount of movement between frames, and the normalized skeleton information 320 are obtained. Then, the process proceeds to step S1402 in FIG. 14.

[0099] Returning to FIG. 14, the server 101 performs learning using the behavior learning unit 403, with the normalized skeletal information 320, the joint angles 370, and the amount of movement between frames as explanatory variables, and the behavior information in the teacher signal as a target variable, and generates a behavior classification model as a result of the learning (step S1402).

[0100] <Figure 16 Action recognition processing> 16 is a flowchart illustrating an example of a behavior recognition process procedure by the client 102 (behavior recognition device) according to the embodiment 1. The client 102 detects, by the skeleton detection unit 410, skeleton information 320 of a person appearing in the analysis target data acquired from the sensor 103 (step S1600).

[0101] Next, the client 102 executes skeleton information processing (step S1601) on the skeleton information 320 detected in step S1600 by the skeleton information processing unit 411 in the same manner as the processing in step S1601. Specifically, for example, the client 102 executes processing (steps S1501 to S1503) by the joint angle calculation unit 501, the movement amount calculation unit 502, and the normalization unit 503 as shown in FIG.

[0102] Next, the client 102 uses the frame-by-frame behavior recognition unit 412 to recognize the behavior of the person captured in the analysis target data acquired from the sensor 103 on a frame-by-frame basis, based on the behavior classification model learned in step S1402, and the skeletal information 320, joint angles 370, and amount of movement between frames normalized in step S1601 (step S1602).

[0103] Next, the client 102 stores the human behavior recognized on a frame-by-frame basis in step S1602 as a behavior recognition result together with the acquisition time in the register 413 (step S1603).

[0104] Next, the client 102 detects the frequency of the behavior recognition result from the multiple acquisition times and behavior recognition results stored in step S1603 by the frequency detection unit 414 (step S1604).

[0105] Next, the client 102 corrects the behavior recognition result determined to be erroneous from the multiple acquisition times and behavior recognition results held in step S1603 using the noise removal unit 415 (step S1605).

[0106] Next, the client 102 resamples the behavior recognition result corrected in step S1605 by the resampling unit 416 so as to have a predetermined target frequency (step S1606).

[0107] Next, the client 102 compares the behavior recognition result resampled in step S1606 with the reference waveform group stored in the teacher signal DB 104 by the determination unit 417, and calculates the behavior recognition result in the time direction, which is the determination result, as a time-series behavior recognition result (step S1607). This ends the processing of the client 102.

[0108] In this way, according to the first embodiment, it is possible to capture the movement of the recognition target in the time direction and recognize a plurality of types of behavior with high accuracy. In this case, the results of the behavior recognition in units of frames are stored in the time direction, noise is removed, and then resampled to a predetermined target frequency, making it possible to recognize the behavior in consideration of the retention time of each result of the frame-by-frame behavior recognition.

[0109] In addition, since the behavior recognition result in the time direction is obtained by resampling the frame-by-frame behavior recognition result to a predetermined target frequency and comparing it with a reference waveform, it is not necessary to prepare teacher signals for different behavior frequencies for each person. Therefore, the amount of teacher signals and reference waveforms to be prepared in the teacher signal DB 104 can be reduced. EXAMPLES

[0110] The second embodiment will be described with a focus on the differences from the first embodiment. The same reference numerals are used to designate the same parts as the first embodiment, and the description thereof will be omitted.

[0111] <Figure 17 Example of functional configuration of the behavior recognition system 100> 17 is a block diagram showing an example of a functional configuration of a behavior recognition system 100 according to Example 2. In Example 2, an output result of the sensor 103 is output to a frequency detection unit 414, and either a downsampling process by a downsampling unit 1700 or an upsampling process by an upsampling unit 1701 is performed depending on the magnitude relationship between the frequency detected by the frequency detection unit 414 and a predetermined target frequency.

[0112] The frequency detection unit 414 acquires analysis target data from the sensor 103. It detects the frequency from the acquisition interval of this analysis target data, and outputs a predetermined target frequency and the detected frequency to the downsampling unit 1700 and the upsampling unit 1701.

[0113] Downsampling unit 1700 compares the frequency detected by frequency detection unit 414 with a predetermined target frequency, and if the detected frequency is equal to or higher than the predetermined target frequency, downsampling is performed on the analysis target data acquired from sensor 103 and outputs the analysis target data to skeleton detection unit 410. On the other hand, if the detected frequency is lower than the predetermined target frequency, downsampling unit 1700 outputs the analysis target data to skeleton detection unit 410 without downsampling.

[0114] The downsampling unit 1700 performs downsampling by, for example, holding the analysis target data for a necessary time and generating a delay so that the reciprocal of the acquisition time interval of the analysis target data becomes a predetermined target frequency. There are other downsampling methods. For example, it is assumed that the multiple analysis target data acquired in the time direction change linearly in the time direction. In this case, the downsampling unit 1700 may generate new analysis target data based on the acquired multiple analysis target data and the acquisition time of each analysis target data, and downsample the newly generated analysis target data so that the reciprocal of each acquisition time interval becomes a predetermined target frequency.

[0115] The upsampling unit 1701 compares the magnitude relationship between the frequency detected by the frequency detection unit 414 and a predetermined target frequency, and if the detection frequency is lower than the predetermined target frequency, performs upsampling of the corrected behavior recognition result acquired from the noise removal unit 415 and outputs the corrected behavior recognition result to the determination unit 417. If the detection frequency is equal to or higher than the predetermined target frequency, the upsampling unit 1701 outputs the corrected behavior recognition result to the determination unit 417 without performing upsampling.

[0116] <Figure 18 Action recognition processing> Fig. 18 is a flowchart showing an example of a behavior recognition processing procedure by the client 102 (behavior recognition device) according to the embodiment 2. In the embodiment 2, in Fig. 16, prior to skeleton detection (step S1600), a frequency detection process (step S1604), a comparison between the detected frequency and the target frequency (step S1801), and downsampling of the analysis target data (step S1802) are executed if the detected frequency is equal to or higher than the target frequency in the determination result in step S1801.

[0117] Furthermore, after noise removal (step S1605), the detected frequency in step S1604 is compared with the target frequency (step S1803), and if the detected frequency is less than the target frequency in the determination result in step S1803, upsampling (step S1804) is performed.

[0118] As described above, according to the second embodiment, the client 102 (behavior recognition device) first performs frequency detection using the analysis target data from the sensor 103, and if the detected frequency is equal to or higher than the predetermined target frequency, performs downsampling before each processing, and if the detected frequency is lower than the predetermined target frequency, performs upsampling after each processing.

[0119] This makes it possible to reduce the processing load of the skeleton detection unit 410, the skeleton information processing unit 411, the frame-unit action recognition unit 412, the register 413, the frequency detection unit 414, the noise removal unit 415, and the judgment unit 417. EXAMPLES

[0120] The third embodiment will be described with a focus on the differences between the first and second embodiments. The same reference numerals are used to designate the same parts as the first and second embodiments, and the description thereof will be omitted.

[0121] <Figure 19 Example of functional configuration of behavior recognition system 100> 19 is a block diagram showing an example of a functional configuration of the behavior recognition system 100 according to Example 3. In Example 3, dimension reduction units 1900 and 1901 are newly added, which perform dimension reduction on information output from skeleton information processing units 402 and 411, and output the results to the behavior learning unit 403 and the frame-by-frame behavior recognition unit 412, respectively.

[0122] Dimension reduction methods include principal component analysis, independent component analysis, SNE (Stochastic Neighbor Embedding), t-SNE (t-Distributed Stochastic Neighbor Embedding), UMAP (Uniform Manifold Approximation and Projection), Isomap, LLE (Locally Linear Embedding), Laplacian Eigenmap, LargeVis, and Diffusion Map. Any of these dimensionality reduction methods may be used, or these dimensionality reduction methods may be combined, and the dimensionality reduction method is not limited.

[0123] <Figure 20 Learning process> Fig. 20 is a flowchart illustrating a detailed example of a procedure of the learning process by the server 101 (learning device) according to Example 3. In Example 3, in Fig. 12, after the skeleton information process (step S1401), dimension reduction (step S2000) is executed.

[0124] <Figure 21 Action recognition processing> 21 is a flowchart illustrating an example of a behavior recognition processing procedure by the client 102 (behavior recognition device) according to Example 1. In Example 3, in FIG. 14, after skeleton information processing (step S1401), dimension reduction (step S2100) is executed.

[0125] As described above, according to the third embodiment, the client 102 (behavior recognition device) reduces the number of variables by performing dimensionality reduction on the skeleton detection output using the analysis target data from the sensor 103. This can reduce the processing load of the behavior learning unit 403, the frame-by-frame behavior recognition unit 412, the register 413, the frequency detection unit 414, the noise removal unit 415, and the determination unit 417.

[0126] In the above-mentioned first to third embodiments, if there is a missing part in the skeleton information 320, the skeleton information processing units 402, 411 may interpolate the missing part. A skeleton point in the skeleton information 320 where the x coordinate value 322 and the y coordinate value 323 are not detected and are indefinite is referred to as missing information. A skeleton point in the skeleton information 320 excluding the missing information is referred to as non-missing information. For example, if only the x coordinate value 322 and the y coordinate value 323 of the skeleton point 310 among the skeleton points 300 to 317 in FIG. 3 are indefinite, the skeleton point 310 is missing information, and the skeleton points 300 to 309 and 311 to 317 are non-missing information.

[0127] The skeleton information processing units 402 and 411 interpolate missing information (skeleton point 310 in the above example) from skeleton information 320 from the teacher signal DB 104 or skeleton detection unit 410 using non-missing information (skeleton points 300 to 309, 311 to 317 in the above example).

[0128] The skeleton information processing units 402, 411 determine whether there are any skeleton points that cannot be acquired due to occlusion or the like among the skeleton information 320 from the teacher signal DB 104 or the skeleton detection unit 410, and if there are any skeleton points that could not be acquired, generate missing position information indicating the position of the skeleton point as a missing part and output it to the frame-by-frame action recognition unit 412.

[0129] As described above, the behavior recognition devices according to Examples 1 to 3 can perform highly accurate recognition while suppressing the calculation load when recognizing an object by taking into account changes in the time direction in two-dimensional information such as a video, even if there is a fluctuation in the frequency of the object.

[0130] Moreover, the behavior recognition devices according to the above-mentioned first to third embodiments can also be configured as follows (1) to (14).

[0131] (1) A behavior recognition device (client 102) having a processor 201 that executes a program and a storage device 202 that stores the program executes a frequency detection process that detects a frequency of the behavior recognition result based on a time-series behavior recognition result of a recognition target and a time interval between the time-series behavior recognition results, a resampling process that normalizes the time-series behavior recognition result in the time direction based on the frequency detected by the frequency detection process, and a determination process that determines whether or not a reference time-series behavior that defines behavior by a time-series signal sequence exists in the time-series behavior recognition result resampled by the resampling process, and outputs a determination result.

[0132] This allows the granularity in the time direction of the behavior recognition result of the recognition target to be changed, making it possible to recognize the time-series behavior of the recognition target with high accuracy.

[0133] (2) In the behavior recognition device of (1) above, the processor executes a noise removal process to remove noise from the time-series behavior recognition result, and in the resampling process, the processor normalizes the time-series behavior recognition result from which noise has been removed by the noise removal process in a time direction, based on the detection frequency.

[0134] This makes it possible to improve the accuracy of the determination process.

[0135] (3) In the behavior recognition device of (1) above, in the noise removal process, the processor removes noise from the time-series behavior recognition result by using a nonlinear filter, and the nonlinear filter can be a mode filter.

[0136] (4) In the behavior recognition device of (1) above, in the resampling process, the processor performs either upsampling or downsampling of the time-series behavior recognition results by comparing the detected frequency with a target frequency.

[0137] This makes it possible to increase or decrease the time-series behavior recognition results.

[0138] (5) In the behavior recognition device of (4) above, in the resampling process, the processor executes the downsampling of the time-series behavior recognition results when the detection frequency is equal to or higher than the target frequency.

[0139] This makes it possible to reduce time-series action recognition results.

[0140] (6) In the behavior recognition device of (4) above, in the resampling process, the processor performs the upsampling of the time-series behavior recognition results when the detection frequency is not equal to or higher than the target frequency.

[0141] This makes it possible to increase the number of time-series action recognition results.

[0142] (7) In the behavior recognition device of (1) above, in the determination process, the processor determines whether or not the resampled time-series behavior recognition result contains a behavior that matches the reference time-series behavior, and outputs a determination result.

[0143] This allows identifying behaviors that match the reference time-series behavior.

[0144] (8) In the behavior recognition device of (1) above, in the determination process, the processor determines whether or not the resampled time-series behavior recognition result contains a behavior that matches a part of the reference time-series behavior, and outputs a determination result.

[0145] This makes it possible to identify behavior that is similar to the reference time-series behavior but does not exactly match the reference time-series behavior.

[0146] (9) In the behavior recognition device of (1) above, a behavior classification model trained to classify the behavior of the learning target from skeletal information indicating a skeletal shape of the learning target can be accessed, and the processor executes a skeletal detection process to detect skeletal information of the recognition target from analysis target data, and a behavior recognition process to output the behavior recognition result indicating the behavior of the recognition target by inputting the skeletal information of the recognition target detected by the skeletal detection process to the behavior classification model, and in the frequency detection process, the processor detects a frequency of the behavior recognition result based on the time-series behavior recognition result of the recognition target by the behavior recognition process and the time interval between the time-series analysis target data.

[0147] This makes it possible to suppress an increase in the amount of skeletal information to be learned, and also to reduce the cost of collecting learning data and the learning time.

[0148] (10) In the behavior recognition device of (9) above, the behavior classification model is a model trained to classify the behavior of the learning target from skeletal information indicating a skeletal shape of the learning target and angles of the joints of the learning target, and in the skeleton detection process, the processor normalizes a size of the skeletal information and calculates angles of the joints of the recognition target based on the normalized skeletal information, and in the behavior recognition process, the processor inputs the skeletal information of the recognition target and the angles of the joints of the recognition target to the behavior classification model, thereby outputting the behavior recognition result indicating the behavior of the recognition target.

[0149] This makes it possible to suppress increases in the skeletal information and joint angles that are the learning targets, and also to reduce the cost of collecting learning data and the learning time.

[0150] (11) In the behavior recognition device of (9) above, in the frequency detection process, the processor detects a frequency between the time-series analysis target data, and the processor executes a downsampling process to downsample the time-series analysis target data based on the frequency detected by the frequency detection process, and in the skeleton detection process, detects skeleton information of the recognition target from the analysis target data based on a result of the downsampling process.

[0151] This can reduce the load of the skeleton detection process.

[0152] (12) In the behavior recognition device of (9) above, the processor executes a noise removal process to remove noise from the time-series behavior recognition result, and in the resampling process, the processor up-samples the time-series behavior recognition result from which noise has been removed by the noise removal process, based on the detection frequency.

[0153] This can reduce the load of the noise removal process.

[0154] (13) In the behavior recognition device of (9) above, the processor executes a dimensionality reduction process to reduce the dimensions of skeletal information of the recognition target, and in the behavior recognition process, the processor inputs the skeletal information of the recognition target, the dimensionality of which has been reduced by the dimensionality reduction process, into the behavior classification model, thereby outputting the behavior recognition result indicating the behavior of the recognition target.

[0155] This can reduce the load of the behavior recognition process.

[0156] The present invention is not limited to the above-described embodiments, and includes various modified examples and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. Also, a part of the configuration of one embodiment may be replaced with a configuration of another embodiment. Also, a configuration of another embodiment may be added to a configuration of one embodiment. Also, a part of the configuration of each embodiment may be added to, deleted from, or replaced with another configuration.

[0157] In addition, each of the configurations, functions, processing units, processing means, etc. described above may be realized in hardware, for example by designing some or all of them as an integrated circuit, or may be realized in software by processor 201 interpreting and executing a program that realizes each function.

[0158] Information such as programs, tables, files, etc. that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC (Integrated Circuit) card, an SD card, or a DVD (Digital Versatile Disc).

[0159] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines that are necessary for implementation. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0160] 100 Behavior Recognition System 101 Server 102 Clients 103 Sensors 104 Teacher signal DB 201 Processor 202 Storage Devices 320 Skeletal Information 401 Teacher signal acquisition section 402, 411 Skeleton information processing unit 403 Behavioral Learning Department 410 Skeleton detection unit 412 Frame-based Action Recognition Unit 413 Register 414 Frequency detection section 415 Noise Removal Section 416 Resampling section 417 Judgment section 1700 Downsampling section 1701 Upsampling section 1900, 1901 Dimension reduction part

Claims

1. A behavior recognition device having a processor that executes a program and a storage device that stores the program, a frequency detection process for detecting a frequency of the behavior recognition result based on a time-series behavior recognition result of a recognition target and a time interval between the time-series behavior recognition results; a resampling process of normalizing the time-series behavior recognition result in a time direction based on a frequency detected by the frequency detection process; a determination process for determining whether or not a reference time-series action that defines an action by a time-series signal sequence exists in the resampled time-series action recognition result obtained by the resampling process, and outputting the determination result; The behavior recognition device is characterized by executing the above.

2. The behavior recognition device according to claim 1 , The processor, execute a noise removal process to remove noise from the time-series behavior recognition results; In the resampling process, the processor normalizes the time-series action recognition result from which noise has been removed by the noise removal process in a time direction based on the detection frequency. The behavior recognition device according to the present invention is characterized in that

3. The behavior recognition device according to claim 2, In the noise removal process, the processor uses a nonlinear filter to removing noise from the time-series behavior recognition results; The behavior recognition device according to the present invention is characterized in that:

4. The behavior recognition device according to claim 1 , In the resampling process, the processor performs either upsampling or downsampling of the time-series behavior recognition result by comparing the detection frequency with a target frequency. The behavior recognition device according to the present invention is characterized in that:

5. The behavior recognition device according to claim 4, In the resampling process, when the detection frequency is equal to or higher than the target frequency, the processor performs the downsampling of the time-series behavior recognition result. The behavior recognition device according to the present invention is characterized in that:

6. The behavior recognition device according to claim 4, In the resampling process, when the detection frequency is not equal to or greater than the target frequency, the processor performs the upsampling of the time-series behavior recognition result. The behavior recognition device according to the present invention is characterized in that:

7. The behavior recognition device according to claim 1 , In the determination process, the processor determines whether or not a behavior that matches the reference time-series behavior is present in the resampled time-series behavior recognition result, and outputs a determination result. The behavior recognition device according to the present invention is characterized in that:

8. The behavior recognition device according to claim 1 , In the determination process, the processor determines whether or not a behavior that matches a part of the reference time-series behavior is present in the resampled time-series behavior recognition result, and outputs a determination result. The behavior recognition device according to the present invention is characterized in that:

9. The behavior recognition device according to claim 1 , A behavior classification model that is trained to classify the behavior of a learning subject from skeletal information indicating the skeletal shape of the learning subject is accessible; The processor, a skeleton detection process for detecting skeleton information of the recognition target from the analysis target data; a behavior recognition process for inputting skeletal information of the recognition target detected by the skeleton detection process into the behavior classification model, thereby outputting the behavior recognition result indicating the behavior of the recognition target; In the frequency detection process, the processor detects a frequency of the behavior recognition result based on the time series behavior recognition result of the recognition target by the behavior recognition process and a time interval between the time series analysis target data. The behavior recognition device according to the present invention is characterized in that:

10. The behavior recognition device according to claim 9, the behavior classification model is a model trained to classify the behavior of the learning subject based on skeletal information indicating a skeletal shape of the learning subject and angles of the joints of the learning subject; In the skeleton detection process, the processor normalizes a size of the skeleton information, and calculates angles of the joints of the recognition target based on the normalized skeleton information; In the behavior recognition process, the processor inputs skeletal information of the recognition target and angles of the joints of the recognition target into the behavior classification model, thereby outputting the behavior recognition result indicating the behavior of the recognition target. The behavior recognition device according to the present invention is characterized in that:

11. The behavior recognition device according to claim 9, In the frequency detection process, the processor detects a frequency between the analysis target data in a time series; The processor, executing a downsampling process for downsampling the time-series analysis target data based on the detected frequency by the frequency detection process; In the skeleton detection process, skeleton information of the recognition target is detected from the analysis target data based on a result of the downsampling process. The behavior recognition device according to the present invention is characterized in that:

12. The behavior recognition device according to claim 9, The processor, execute a noise removal process to remove noise from the time-series behavior recognition results; In the resampling process, the processor upsamples the time-series action recognition result from which noise has been removed by the noise removal process, based on the detection frequency. The behavior recognition device according to the present invention is characterized in that:

13. The behavior recognition device according to claim 9, The processor, Executing a dimension reduction process for reducing the dimension of the skeleton information of the recognition target; In the behavior recognition process, the processor inputs skeletal information of the recognition target, the dimension of which has been reduced by the dimension reduction process, into the behavior classification model, thereby outputting the behavior recognition result indicating the behavior of the recognition target. The behavior recognition device according to the present invention is characterized in that:

14. A behavior recognition method executed by a behavior recognition device having a processor that executes a program and a storage device that stores the program, a frequency detection process for detecting a frequency of the behavior recognition result based on a time-series behavior recognition result of a recognition target and a time interval between the time-series behavior recognition results; a resampling process of normalizing the time-series behavior recognition result in a time direction based on a frequency detected by the frequency detection process; a determination process for determining whether or not a reference time-series action that defines an action by a time-series signal sequence exists in the resampled time-series action recognition result obtained by the resampling process, and outputting the determination result; The behavior recognition method is characterized by executing the above.

15. The processor: a frequency detection process for detecting a frequency of the behavior recognition result based on a time-series behavior recognition result of a recognition target and a time interval between the time-series behavior recognition results; a resampling process of normalizing the time-series behavior recognition result in a time direction based on a frequency detected by the frequency detection process; a determination process for determining whether or not a reference time-series action that defines an action by a time-series signal sequence exists in the resampled time-series action recognition result obtained by the resampling process, and outputting the determination result; A behavior recognition program that causes the program to execute the above steps.

Images