Estimation program, estimation method, and information processing device

Abstract

This information processing device acquires point cloud information generated on the basis of a reflection signal from radar emitted toward a plurality of persons. On the basis of the spatial distance of the acquired point cloud information and the speed of the point cloud information, the information processing device performs clustering of the point cloud information. The information processing device estimates the orientation of each of the plurality of persons on the basis of the clustered point cloud information.

Description

Therefore, when the cluster 17a is input to the AI 12, posture data is output and is not correct posture data.

[0013] As described with reference to FIG. 17, in Related Art 2, when the distances between a plurality of persons are short, the posture of each person is not accurately estimated.

[0014] In one aspect, an object of the present invention is to provide an estimation program, an estimation method, and an information processing apparatus capable of accurately estimating postures of a plurality of persons. Solution to Problem

[0015] According to an aspect of the embodiment of the invention, an estimation program causes a computer to execute acquiring point cloud information generated based on a reflection signal of a radar wave emitted toward a plurality of persons clustering the point cloud information based on a spatial distance of the acquired point cloud information and a velocity of the point cloud information, and estimating a posture of each of the plurality of persons based on the clustered point cloud information. Advantageous Effects of Invention

[0016] It is possible to accurately estimate the postures of a plurality of persons. Brief Description of Drawings

[0017] FIG. 1 is a diagram illustrating a system according to the present embodiment. FIG. 2 is a diagram illustrating a Doppler effect. FIG. 3 is a diagram illustrating an example of processing of an information processing apparatus according to the present embodiment. FIG. 4 is a functional block diagram illustrating a configuration of the information processing apparatus according to the present embodiment. FIG. 5 is a view illustrating an example of a skeleton model of a human body. FIG. 6 is a diagram illustrating an example of joint names. FIG. 7 is a diagram illustrating an example of a data structure of a point cloud data table. FIG. 8 is a diagram illustrating an example of a data structure of a cluster table. FIG. 9 is a diagram illustrating an example of a data structure of a posture data table. FIG. 10 is a diagram (1) illustrating an abnormality detection process to be performed by the information processing apparatus. FIG. 11 is a diagram (2) illustrating an abnormality detection process to be performed by the information processing apparatus. FIG. 12 is a diagram (3) illustrating an abnormality detection process to be performed by the information processing apparatus. FIG. 13 is a flowchart illustrating a processing procedure of the information processing apparatus according to the present embodiment. FIG. 14 is a diagram illustrating an example of a hardware configuration of a computer that implements functions similar to those of the information processing apparatus according to the embodiment. FIG. 15 is a diagram illustrating Related Art 1. FIG. 16 is a diagram illustrating Related Art 2. FIG. 17 is a diagram illustrating a problem of Related Art 2. Description of Embodiments

[0018] Hereinafter, embodiments of an estimation program, an estimation method, and an information processing apparatus disclosed in the present application will be described in detail with reference to the drawings. Note that the present invention is not limited by the embodiments. (Embodiments)

[0019] An example of a system according to the present embodiment will be described. FIG. 1 is a diagram illustrating a system according to the present embodiment. As illustrated in FIG. 1, the system includes a millimeter wave sensor 20 and an information processing apparatus 100. The millimeter wave sensor 20 and the information processing apparatus 100 are connected to each other via a network 25. In the present embodiment, it is assumed that the persons h1 and h2 are included in the irradiation range of the radio wave (millimeter wave) of the millimeter wave sensor 20.

[0020] The millimeter wave sensor 20 emits radio waves, detects reflected waves from the persons h1 and h2, and transmits a detection result to the information processing apparatus 100. In the following description, for convenience of description, a detection result obtained by the millimeter wave sensor 20 at a certain time t is referred to as a "frame ft". The frame ft includes the detected time, a time difference, a phase difference, an increase / decrease in frequency, and the like from irradiation of the radio waves to arrival of the reflected waves arrive. The millimeter wave sensor 20 repeatedly executes the above processing at predetermined time intervals (for example, 0.1 seconds).

[0021] The information processing apparatus 100 estimates the postures of the persons h1 and h2 on the basis of the point cloud data generated on the basis of the frame ft. Note that the point cloud data generated by the information processing apparatus 100 includes position information (three-dimensional coordinates) and velocity information of each point.

[0022] For example, the information processing apparatus 100 obtains the velocity information for each point using the Doppler effect. FIG. 2 is a diagram illustrating a Doppler effect. By using the Doppler effect, the velocity of a moving object can be calculated. For example, the information processing apparatus 100 calculates the velocity of the point cloud information by using the Doppler effect for the radar waves irradiated toward the object. The object may be a plurality of persons.

[0023] When the radio wave transmitted by the millimeter wave sensor 20 is reflected on a moving object, the frequency increases or decreases in proportion to the speed of the object. For example, for a stationary car, the frequency does not change, whereas for a car moving at 100 km / h (an approaching car), the frequency increases by approximately 4 kHz. The information processing apparatus 100 obtains velocity information from a difference in frequency between the transmitted radio wave and the received radio wave (reflected wave).

[0024] Next, an example of processing of the information processing apparatus 100 will be described. FIG. 3 is a diagram illustrating an example of processing of an information processing apparatus according to the present embodiment. With reference to FIG. 3, the processing of the information processing apparatus 100 will be described step by step. Note that, in the present embodiment, an image of a person is illustrated in a frame for ease of viewing, but an actual frame is information acquired from the millimeter wave sensor 20, and is information including time, a time difference, a phase difference, an increase / decrease in frequency, and the like from irradiation of a radio wave to arrival of a reflected wave.

[0025] Step S10 will be described.  The information processing apparatus 100 generates the point cloud data 30t based on the frame ft. The information processing apparatus 100 classifies the point cloud data 30t into the cluster 40ta by executing clustering on the basis of the position information of each point included in the point cloud data 30t. For example, since the distance between the person h1 and the person h2 is short at the time t, the point clouds corresponding to the person h1 and the person h2 are classified into one cluster 40ta.

[0026] Step S11 will be described. The information processing apparatus 100 classifies the point cloud data 30t into the clusters 40tb, 40tc again on the basis of the velocity information of each point included in the point cloud data 30t.  In FIG. 3, as an example, a point cloud classified into the cluster 40tb is indicated by a square, and a point cloud classified into the cluster 40tc is indicated by a triangle.

[0027] Step S12 will be described. The information processing apparatus 100 corrects the clustering result in step S11 on the basis of the cluster table 142 that stores past clustering results. For example, the final clustering result for the point cloud data 30t-1 of the immediately preceding (time t-1) frame ft-1 is registered in the cluster table 142. The information processing apparatus 100 corrects the clustering result in step S11 on the basis of the clustering result (point cloud distribution) for the point cloud data 30t-1.

[0028] An example of processing in which the information processing apparatus 100 corrects the point cloud classified into the cluster 40tb corresponding to the person h1 will be described. Note that the cluster corresponding to the person h1 at the time t-1 is referred to as "cluster 40tb-1". The information processing apparatus 100 estimates the distribution of the cluster 40tb-1 at the time t from the cluster 40tb-1 and the velocity of the person h1. The distribution of the cluster 40tb-1 at the estimation time t is referred to as a "distribution Bb".

[0029] The information processing apparatus 100 may specify the velocity of the person h1 in any manner. For example, the information processing apparatus 100 uses an average value, a median value, or the like of the velocity information of each point included in the cluster 40tb-1 as the velocity of the person h1.

[0030] The information processing apparatus 100 compares the region of the distribution Bb with the position of each point of the cluster 40tb, and specifies points that do not exist within a predetermined distance based on the region of the distribution Bb among the points of the cluster 40tb as points that are not suitable for the cluster 40tb. The information processing apparatus 100 corrects the classification destination of the specified point from the cluster 40tb to the cluster 40tc.

[0031] An example of processing in which the information processing apparatus 100 corrects the point cloud classified into the cluster 40tc corresponding to the person h2 will be described. Note that the cluster corresponding to the person h2 at the time t-1 is referred to as "cluster 40tc-1". The information processing apparatus 100 estimates the distribution of the cluster 40tc-1 at the time t from the cluster 40tc-1 and the velocity of the person h2. The distribution of the cluster 40tc-1 at the estimation time t is referred to as a "distribution Bc".

[0032] The information processing apparatus 100 may specify the velocity of the person h2 in any manner. For example, the information processing apparatus 100 uses an average value, a median value, or the like of the velocity information of each point included in the cluster 40tc-1 as the velocity of the person h2.

[0033] The information processing apparatus 100 compares the region of the distribution Bc with the position of each point of the cluster 40tc, and specifies points that do not exist within a predetermined distance based on the region of the distribution Bc among the points of the cluster 40tc as points that are not suitable for the cluster 40tc. The information processing apparatus 100 corrects the classification destination of the specified point from the cluster 40tc to the cluster 40tb.

[0034] The information processing apparatus 100 registers the information on the cluster 40tb after the correction and the information on the cluster 40tc after the correction in the cluster table 142.

[0035] The information processing apparatus 100 inputs the cluster 40tb after the correction to the AI 50 to estimate posture data 45tb of the person h1. Further, the information processing apparatus 100 inputs the cluster 40tc after the correction to the AI 50 to estimate posture data 45tc of the person h2.

[0036] As described above, the information processing apparatus 100 according to the present embodiment executes clustering based on spatial distances and clustering based on velocity information on the basis of the point cloud data obtained on the basis of the millimeter wave sensor 20. With such clustering, even in a case where a plurality of persons are close in distance, the point cloud data can be clustered in association with each person. Therefore, the information processing apparatus 100 can accurately estimate the postures of the plurality of persons using the clustering result.

[0037] Next, a configuration example of the information processing apparatus 100 will be described. FIG. 4 is a functional block diagram illustrating a configuration of the information processing apparatus according to the present embodiment. As illustrated in FIG. 4, the information processing apparatus 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0038] The communication unit 110 executes data communication with the millimeter wave sensor 20 and the like via the network 25. For example, the communication unit 110 receives the frame ft from the millimeter wave sensor 20.

[0039] The input unit 120 inputs various types of information to the control unit 150 of the information processing apparatus 100.

[0040] The display unit 130 displays information output from the control unit 150.

[0041] The storage unit 140 includes an AI 50, a point cloud data table 141, a cluster table 142, and a posture data table 143. The storage unit 140 is a memory or the like.

[0042] The AI 50 is a machine learning model having a point cloud as an input and posture data of a person as an output. The AI 50 is a neural network (NN) or the like.

[0043] For example, the posture data of the person is data corresponding to skeleton data. Here, the skeleton data is data in which two-dimensional or three-dimensional coordinates are set for a plurality of joints defined in the skeleton model of the human body. Here, the coordinates of each joint of the skeleton data are three dimensional coordinates. FIG. 5 is a view illustrating an example of a skeleton model of a human body. For example, as illustrated in FIG. 5, a skeleton model of a human body is defined by 21 joints ar0 to ar20.

[0044] The relationship between each of the joints ar0 to ar20 illustrated in FIG. 5 and the joint name is as illustrated in FIG. 6. FIG. 6 is a diagram illustrating an example of joint names. For example, the joint name of the joint ar0 is "SPINE_BASE". The joint names of the joints ar1 to ar20 are as illustrated in FIG. 6, and the description thereof will be omitted.

[0045] The point cloud data table 141 is a table that stores point cloud data generated from a frame at each time. FIG. 7 is a diagram illustrating an example of a data structure of a point cloud data table. As illustrated in FIG. 7, the point cloud data table 141 associates time with point cloud data. The time is a time at which the millimeter wave sensor 20 generates a frame (detection result). The point cloud data is point cloud data generated from a frame at a corresponding time. For example, the point cloud data is generated by a preprocessing unit 151 described later. As described above, in the point cloud data, the position information (three-dimensional coordinates) and the velocity information are associated with each point. In addition, it is assumed that point identification information for uniquely identifying a point is assigned to each point.

[0046] The cluster table 142 stores cluster information at each time. FIG. 8 is a diagram illustrating an example of a data structure of the cluster table. As illustrated in FIG. 8, the cluster table 142 associates time, first cluster data, second cluster data, and third cluster data.

[0047] The first cluster data includes clustering-result data obtained by performing clustering based on a spatial distance on point cloud data at a certain time. The clustering based on the spatial distance is the clustering described in step S10 of FIG. 3. For example, in the first cluster data, cluster identification information for identifying a cluster is associated with point identification information of each point (point cloud) belonging to the corresponding cluster. At each time, the same cluster identification information is assigned to each cluster assumed to be the same person.

[0048] The second cluster data includes clusteringresult data obtained by performing clustering based on velocity information on point cloud data at a certain time. The clustering based on the velocity information is the clustering described in step S11 of FIG. 3. For example, in the second cluster data, cluster identification information for identifying a cluster is associated with point identification information of each point (point cloud) belonging to the corresponding cluster. At each time, the same cluster identification information is assigned to each cluster assumed to be the same person.

[0049] The third cluster data includes data of a corrected clustering result for the second cluster data. The correction for the cluster is the correction described in step S12 of FIG. 3.

[0050] The posture data table 143 stores posture data estimated at each time. FIG. 9 is a diagram illustrating an example of a data structure of a posture data table. As illustrated in FIG. 9, the posture data table 143 associates time with posture data. The posture data is data corresponding to the skeleton data described with reference to FIG. 5. Person identification information for identifying a person corresponding to the posture data is assigned to the posture data.

[0051] The description will be made again with reference to FIG. 4. The control unit 150 includes a preprocessing unit 151, a first classification unit 152, a second classification unit 153, a correction unit 154, an estimation unit 155, and an abnormal action detection unit 156. The control unit 150 is a central processing unit (CPU), a graphics processing unit (GPU), or the like.

[0052] The preprocessing unit 151 receives the frame f from the millimeter wave sensor 20 and generates the point cloud data based on the frame f. The preprocessing unit 151 registers the time and the point cloud data in the point cloud data table 141 in association with each other. The preprocessing unit 151 repeatedly executes the above processing every time the frame f is received.

[0053] For example, the preprocessing unit 151 analyzes the time difference, the phase difference, the increase / decrease in frequency, and the like from the irradiation of the radio wave to the arrival of the reflected wave, extracts the distance information to the person and the velocity information of the person, and specifies the position information and the velocity information for each point on the surface of the person based on the extraction result. Note that the preprocessing unit 151 may generate the point cloud data from the frame f using any known technique.

[0054] The first classification unit 152 acquires the point cloud data from the point cloud data table 141, and executes clustering on the basis of the position information of each point included in the point cloud data. By such clustering, points having similar position information are classified into the same cluster. The first classification unit 152 registers the first cluster data as a clustering result in the cluster table 142 in association with time.

[0055] The clustering by the first classification unit 152 corresponds to the clustering described in step S10 in FIG. 3. The first classification unit 152 repeatedly executes the above processing each time new point cloud data is registered in the point cloud data table 141.

[0056] The second classification unit 153 acquires the point cloud data from the point cloud data table 141, and executes clustering on the basis of the velocity information of each point included in the point cloud data. By such clustering, points having similar velocity information are classified into the same cluster. The second classification unit 153 registers the second cluster data as a clustering result in the cluster table 142 in association with time.

[0057] The clustering by the second classification unit 153 corresponds to the clustering described in step S11 in FIG. 3. The second classification unit 153 repeatedly executes the above processing each time new point cloud data is registered in the point cloud data table 141.

[0058] The correction unit 154 corrects the second correction result in the cluster table 142 in association with time.

[0059] For example, the correction unit 154 acquires, from the cluster table 142, the third cluster data corresponding to the time t-1 immediately before the current time t. The correction unit 154 generates the third cluster data at the time t1 by correcting the second cluster data based on the third cluster data corresponding to the time t-1 and the second cluster data corresponding to the time t1, and registers the third cluster data at the time t1 in the cluster table 142.

[0060] The correction performed by the correction unit 154 corresponds to the correction described in step S12 of FIG. 3. The correction unit 154 repeatedly executes the above processing each time new second cluster data is registered in the cluster table 142.

[0061] When the first cluster data and the second cluster data at the same time satisfy the similar condition, the correction unit 154 registers the first cluster data in the cluster table 142 as the third cluster data. For example, when a plurality of persons are separated, the number of clusters and the distribution of each cluster included in the first cluster data are similar to the number of clusters and the distribution of clusters included in the second cluster data. Usually, when a plurality of persons are separated from each other, since the first cluster data has high reliability, the first cluster data can be adopted as a correction result of the second cluster data. The similar condition may be set in any manner. For example, the similar condition is a condition that "the number of clusters is the same and the difference between the distributions of the clusters is less than the threshold".

[0062] The estimation unit 155 acquires the third cluster data registered in the cluster table 142 and inputs the third cluster data to the AI 50 to estimate the posture data of the person. The estimation unit 155 registers the posture data in the posture data table 143 in association with the time.

[0063] When the third cluster data includes a cluster for each person, the estimation unit 155 inputs a point cloud belonging to the cluster for each person to the AI 50, and estimates posture data of each person. The estimation of the posture data performed by the estimation unit 155 corresponds to the estimation of the posture data described in step S12 of FIG. 3.

[0064] The estimation unit 155 repeatedly executes the above processing for each time, whereby time-series posture data is registered in the posture data table 143 for each person (for each person identification information).

[0065] The abnormal action detection unit 156 detects an abnormal action of a person on the basis of the posture data table 143. The abnormal action detection unit 156 detects an abnormal action using a rule table or AI for abnormality detection. The rule table is a table in which the transition of each joint position of posture data is associated with the type of abnormal action.  The AI for abnormality detection is a machine learning model in which posture data in time series is input and a type of abnormal action is output. The AI for abnormality detection is an NN or the like.

[0066] When detecting an abnormality, the abnormal action detection unit 156 may transmit abnormal action information to a predetermined external terminal. The abnormal action information is associated with a frame of a predetermined period based on the time at which the abnormal action is detected, point cloud data, the content of the abnormal action, the time, and the like.

[0067] Here, a notification example of issuing an alert to a terminal of a surveillance staff who monitors a predetermined room of a building will be described. The information processing apparatus 100 monitors a plurality of persons present in a predetermined room. Note that the plurality of persons exist in a room in which privacy is to be maintained. The predetermined room is, for example, a room among spaces in which privacy is to be maintained, and is a private space such as a private room of a hospital or a hotel, a dressing room, a nursing room, or a toilet. At this time, for example, the information processing apparatus 100 specifies the point cloud information of the first person among the plurality of persons and the point cloud information of the second person among the plurality of persons, based on the clustered point cloud information. Next, for example, the information processing apparatus 100 detects an abnormal action of a plurality of persons by separately analyzing the point cloud information of the specified first person and the point cloud information of the second person. Then, for example, the information processing apparatus 100 notifies a terminal of a surveillance staff who monitors a room in which privacy is to be maintained of an alert indicating the occurrence of the abnormal action. More specifically, for example, the information processing apparatus 100 causes the terminal to display a place where an abnormality has occurred and an alert indicating the occurrence of the abnormal action.

[0068] Here, an example of abnormality detection processing by the information processing apparatus 100 will be described for each case with reference to FIGS. 10, 11, and 12. FIGS. 10 to 12 are diagrams illustrating abnormality detection processing by the information processing apparatus.

[0069] First, a description will be made with reference to FIG. 10. FIG. 10 illustrates an example of abnormality detection in a case where the nurse h4 performs a rescue response for the patient h3.  The information processing apparatus 100 acquires the frame ft1 at time t1 and executes the above processing to specify the cluster 61t1b corresponding to the patient h3. The information processing apparatus 100 obtains posture data 71t1b by inputting the point cloud of the cluster 61t1b to the AI 50. The abnormal action detection unit 156 detects an abnormal action "falling" based on the posture data 71t1b.

[0070] The information processing apparatus 100 acquires the frame ft2 at time t2 and executes the above processing to separately specify the cluster 61t2b corresponding to the patient h3 and the cluster 61t2c corresponding to the nurse h4. The information processing apparatus 100 obtains posture data 71t2b by inputting the point cloud of the cluster 61t2b to the AI 50. The information processing apparatus 100 obtains posture data 71t2c by inputting the point cloud of the cluster 61t2c to the AI 50.  The abnormal action detection unit 156 detects an abnormal action "falling" based on the posture data 71t2b.

[0071] As described with reference to FIG. 10, even in a case where the distance between the nurse h4 and the patient h3 is short, the abnormal action "falling" of the patient h3 can be detected. On the other hand, in Related Art 2, in a case where the distance between the nurse h4 and the patient h3 is short, the cluster 61t2b corresponding to the patient h3 and the cluster 61t2c corresponding to the nurse h4 are not specified separately, and it is difficult to detect an abnormality.

[0072] Next, a description will be made with reference to FIG. 11. FIG. 11 illustrates an example of abnormality detection in the private space in which the persons h5 and h6 are present. The information processing apparatus 100 acquires the frame ft1 at time t1 and executes the above processing to separately specify the cluster 62t1b corresponding to the person h5 and the cluster 62t1c corresponding to the person h6. The information processing apparatus 100 obtains posture data 72t1b by inputting the point cloud of the cluster 62t1b to the AI 50. The information processing apparatus 100 obtains posture data 72t1c by inputting the point cloud of the cluster 62t1c to the AI 50.

[0073] The information processing apparatus 100 acquires the frame ft2 at time t2 and executes the above processing to separately specify the cluster 62t2b corresponding to the person h5 and the cluster 62t2c corresponding to the person h6. The information processing apparatus 100 obtains posture data 72t2b by inputting the point cloud of the cluster 62t2b to the AI 50. The information processing apparatus 100 obtains posture data 72t2c by inputting the point cloud of the cluster 62t2c to the AI 50.  The abnormal action detection unit 156 detects an abnormal action "falling" based on the posture data 72t2b.

[0074] As described with reference to FIG. 11, even in a case where the distance between the person h5 and the person h6 is short, the abnormal action "falling" of the person h5 can be detected. In addition, according to the processing result of FIG. 11, the information processing apparatus 100 can specify that the person h5 and the person h6 collided while passing each other, and the person h5 fell.

[0075] Next, a description will be made with reference to FIG. 12. FIG. 12 illustrates an example of abnormality detection in the private space in which the persons h7 and h8 are present. The information processing apparatus 100 acquires the frame ft1 at time t1 and executes the above processing to separately specify the cluster 63t1b corresponding to the person h7 and the cluster 63t1c corresponding to the person h8. The information processing apparatus 100 obtains posture data 73t1b by inputting the point cloud of the cluster 63t1b to the AI 50. The information processing apparatus 100 obtains posture data 73t1c by inputting the point cloud of the cluster 63t1c to the AI 50.

[0076] The information processing apparatus 100 acquires the frame ft2 at time t2 and executes the above processing to separately specify the cluster 63t2b corresponding to the person h7 and the cluster 63t2c corresponding to the person h8. The information processing apparatus 100 obtains posture data 73t2b by inputting the point cloud of the cluster 63t2b to the AI 50. The information processing apparatus 100 obtains posture data 73t2c by inputting the point cloud of the cluster 63t2c to the AI 50. For example, the abnormal action detection unit 156 detects an abnormal action "approaching another person at an abnormal velocity and raising a right knee" of the person h7 based on the posture data 73t2b.

[0077] The information processing apparatus 100 acquires the frame ft3 at time t3 and executes the above processing to specify the cluster 63t3b corresponding to the person h7. Note that the person h8 has fallen (out of the range of the millimeter wave sensor 20), and the point cloud data is not generated. The information processing apparatus 100 obtains posture data 73t3b by inputting the point cloud of the cluster 63t3b to the AI 50.

[0078] As described with reference to FIG. 12, even when the distance between the person h7 and the person h8 is short, the abnormal action "approaching another person at an abnormal velocity and raising a right knee" of the person h7 can be detected. For example, "approaching another person at an abnormal velocity and raising a right knee" corresponds to a violent act on another person.

[0079] Next, an example of a processing procedure of the information processing apparatus 100 according to the present embodiment will be described. FIG. 13 is a flowchart illustrating a processing procedure of the information processing apparatus according to the present embodiment. As illustrated in FIG. 13, the preprocessing unit 151 of the information processing apparatus 100 acquires a frame from the millimeter wave sensor 20 and generates point cloud data (step S101).

[0080] The first classification unit 152 of the information processing apparatus 100 performs clustering on the basis of the position information of the point cloud data (step S102). The second classification unit 153 of the information processing apparatus 100 performs clustering on the basis of the velocity information of the point cloud data (step S103).

[0081] The correction unit 154 of the information processing apparatus 100 acquires the third cluster data at the time t-1 from the cluster table 142 (step S104). The correction unit 154 determines whether or not to correct the second cluster data based on the third cluster data corresponding to the time t-1 and the second cluster data corresponding to the time t1 (step S105).

[0082] In the case of correction (in step S106, Yes), the correction unit 154 corrects the second cluster data at (step S107). On the other hand, when no correction is to be made (step S106, No), the correction unit 154 sets the second cluster data at time t as the third cluster data at time t (step S108).

[0083] The correction unit 154 registers the third cluster data at the time t in the cluster table 142 (step S109).

[0084] The estimation unit 155 of the information processing apparatus 100 inputs the third cluster data at the time t to the AI 50 and estimates posture data (step S110).  The abnormal action detection unit 156 of the information processing apparatus 100 executes abnormal action detection on the basis of the posture data (step S111).  The abnormal action detection unit 156 outputs a result of the abnormal action detection (step S112).

[0085] The processing procedure of the information processing apparatus 100 has been described above with reference to FIG. 13. The information processing apparatus 100 repeatedly executes the processing described in FIG. 13 for each time.

[0086] Next, effects of the information processing apparatus 100 according to the present embodiment will be described. The information processing apparatus 100 clusters the point cloud data on the basis of the spatial distance of the point cloud data and the velocity of the point cloud information, and estimates the posture of each of the plurality of persons on the basis of the clustered point cloud data. This makes it possible to accurately estimate the postures of a plurality of persons.

[0087] The information processing apparatus 100 sequentially executes clustering on the time-series point cloud data, and adjusts the result of current clustering on point cloud data and the result of current clustering based on the point cloud data. Thus, the accuracy of the clustering result of each person can be improved.

[0088] When a distance between a certain point included in a cluster of a specific person included in a result of current clustering and a cluster corresponding to the specific person included in a result of previous clustering is equal to or more than a threshold value specified based on a velocity of the point cloud data, the information processing apparatus 100 classifies the certain point into a cluster of a person other than the specific person. Thus, the accuracy of the clustering result of each person can be improved.

[0089] The information processing apparatus 100 specifies an abnormal action of each of a plurality of persons based on skeleton information of each of the plurality of persons. Thus, even when the millimeter wave sensor 20 is used, abnormal action of a person can be detected.

[0090] Next, an example of a hardware configuration of a computer that implements functions similar to those of the information processing apparatus 100 described above will be described. FIG. 14 is a diagram illustrating an example of a hardware configuration of a computer that implements functions similar to those of the information processing apparatus according to the embodiment.

[0091] As illustrated in FIG. 14, a computer 200 includes a CPU 201 that executes various types of arithmetic processing, an input device 202 that receives an input of data from a user, and a display 203. Furthermore, the computer 200 includes a communication device 204 that exchanges data with an external device or the like via a wired or wireless network, and an interface device 205. In addition, the computer 200 includes a RAM 206 that temporarily stores various types of information and a hard disk drive 207. Each of the devices 201 to 207 is connected to a bus 208.

[0092] The hard disk drive 207 includes a preprocessing program 207a, a first classification program 207b, a second classification program 207c, a correction program 207d, an estimation program 207e, and an abnormal action detection program 207f. The CPU 201 reads the programs 207a to 207f and develops the programs in the RAM 206.

[0093] The preprocessing program 207a functions as a preprocessing process 206a. The first classification program 207b functions as a first classification process 206b. The second classification program 207c functions as a second classification process 206c. The correction program 207d functions as a correction process 206d. The estimation program 207e functions as an estimation process 206e. The abnormal action detection program 207f functions as an abnormal action detection process 206f.

[0094] The processing of the preprocessing process 206a corresponds to the processing of the preprocessing unit 151. The processing of the first classification process 206b corresponds to the processing of the first classification unit 152. The processing of the second classification process 206c corresponds to the processing of the second classification unit 153. The processing of the correction process 206d corresponds to the processing of the correction unit 154. The processing of the estimation process 206e corresponds to the processing of the estimation unit 155. The processing of the abnormal action detection process 206f corresponds to the processing of the abnormal action detection unit 156.

[0095] The programs 207a to 207f do not need to be stored in the hard disk drive 207 from the beginning. For example, each program is stored in a "portable physical medium" such as a flexible disk (FD), a CD-ROM, a DVD, a magneto-optical disk, or an IC card inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207f.

[0096] Regarding the embodiments including the above embodiments, the following supplementary notes are further disclosed.

[0097] (Supplementary note 1) An estimation program causing a computer to execute: acquiring point cloud information generated based on a reflection signal of a radar wave emitted toward a plurality of persons; clustering the point cloud information based on a spatial distance of the acquired point cloud information and a velocity of the point cloud information, and estimating a posture of each of the plurality of persons based on the clustered point cloud information.

[0098] (Supplementary note 2) The estimation program according to Supplementary note 1, in which the acquiring includes acquiring time-series point cloud information, the clustering includes sequentially executing clustering on the time-series point cloud information, and the estimation program causes the computer to further execute: correcting a result of current clustering, based on a result of previous clustering based on the point cloud information and a result of current clustering based on the point cloud information.

[0099] (Supplementary note 3) The estimation program according to Supplementary note 2, in which in the correcting, when a distance between a certain point included in a cluster of a specific person included in a result of current clustering and a cluster corresponding to the specific person included in a result of previous clustering is equal to or more than a threshold value specified based on a velocity of the point cloud information, the certain point is classified into a cluster of a person other than the specific person.

[0100] (Supplementary note 4) The estimation program according to Supplementary note 1, in which the estimating includes estimating skeleton information of each of the plurality of persons based on the point cloud information, and the estimation program causes the computer to further execute: specifying an abnormal action of each of the plurality of persons based on the skeleton information of each of the plurality of persons.

[0101] (Supplementary note 5) The estimation program according to Supplementary note 1, in which the point cloud information includes positions and velocities of a plurality of points on surfaces of the plurality of persons, and the clustering includes first clustering based on positions of the plurality of points included in the point cloud information, and second clustering based on velocities of the plurality of points included in the point cloud information.

[0102] (Supplementary note 6) The estimation program according to Supplementary note 1, in which the velocity of the point cloud information is calculated based on a Doppler effect of a radar wave emitted toward an object.

[0103] (Supplementary note 7) The estimation program according to Supplementary note 1, in which each of the plurality of persons is present in a room where privacy is to be maintained, and the estimating includes specifying point cloud information of a first person among the plurality of persons and point cloud information of a second person among the plurality of persons, based on the clustered point cloud information, detecting an occurrence of an abnormal action of the plurality of persons by separately analyzing the point cloud information of the first person and the point cloud information of the second person, and notifying a terminal of a surveillance staff who monitors a room where privacy is to be maintained of an alert indicating the occurrence of the abnormal action.

[0104] (Supplementary note 8) An estimation method executed by a computer, the method comprising: acquiring point cloud information generated based on a reflection signal of a radar wave emitted toward a plurality of persons; clustering the point cloud information based on a spatial distance of the acquired point cloud information and a velocity of the point cloud information, and estimating a posture of each of the plurality of persons based on the clustered point cloud information.

[0105] (Supplementary note 9) The estimation method according to Supplementary note 8, in which the acquiring includes acquiring time-series point cloud information, the clustering includes sequentially executing clustering on the time-series point cloud information, and the estimation method further comprising: correcting a result of current clustering, based on a result of previous clustering based on the point cloud information and a result of current clustering based on the point cloud information.

[0106] (Supplementary note 10) The estimation method according to Supplementary note 9, in which in the correcting, when a distance between a certain point included in a cluster of a specific person included in a result of current clustering and a cluster corresponding to the specific person included in a result of previous clustering is equal to or more than a threshold value specified based on a velocity of the point cloud information, the certain point is classified into a cluster of a person other than the specific person.

[0107] (Supplementary note 11) The estimation method according to Supplementary note 8, in which the estimating includes estimating skeleton information of each of the plurality of persons based on the point cloud information, and the estimation method further comprising: specifying an abnormal action of each of the plurality of persons based on the skeleton information of each of the plurality of persons.

[0108] (Supplementary note 12) The estimation method according to Supplementary note 8, in which the point cloud information includes positions and velocities of a plurality of points on surfaces of the plurality of persons, and the clustering includes first clustering based on positions of the plurality of points included in the point cloud information, and second clustering based on velocities of the plurality of points included in the point cloud information.

[0109] (Supplementary note 13) The estimation method according to Supplementary note 8, in which the velocity of the point cloud information is calculated based on a Doppler effect of radar waves emitted toward an object.

[0110] (Supplementary note 14) The estimation method according to Supplementary note 8, in which each of the to be maintained, and the estimating includes specifying point cloud information of a first person among the plurality of persons and point cloud information of a second person among the plurality of persons, based on the clustered point cloud information, detecting an occurrence of an abnormal action of the plurality of persons by separately analyzing the point cloud information of the first person and the point cloud information of the second person, and notifying a terminal of a surveillance staff who monitors a room where privacy is to be maintained of an alert indicating the occurrence of the abnormal action.

[0111] (Supplementary note 15) An information processing apparatus comprising a control unit that executes: acquiring point cloud information generated based on a reflection signal of a radar wave emitted toward a plurality of persons; clustering the point cloud information based on a spatial distance of the acquired point cloud information and a velocity of the point cloud information, and estimating a posture of each of the plurality of persons based on the clustered point cloud information.

[0112] (Supplementary note 16) The information processing apparatus according to Supplementary note 15, in which the control unit acquires time-series point cloud information, sequentially executes clustering on the timeseries point cloud information, and further executes correction of a result of current clustering, based on a result of previous clustering based on the point cloud information and a result of current clustering based on the point cloud information.

[0113] (Supplementary note 17) The information processing apparatus according to Supplementary note 16, in which when a distance between a certain point included in a cluster of a specific person included in a result of current clustering and a cluster corresponding to the specific person included in a result of previous clustering is equal to or more than a threshold value specified based on a velocity of the point cloud information, the control unit classifies the certain point into a cluster of a person other than the specific person.

[0114] (Supplementary note 18) The information processing apparatus according to Supplementary note 15, in which the control unit estimates skeleton information of each of the plurality of persons based on the point cloud information, and further executes specifying an abnormal action of each of the plurality of persons based on the skeleton information of each of the plurality of persons.

[0115] (Supplementary note 19) The information processing apparatus according to Supplementary note 15, in which the point cloud information includes positions and velocities of a plurality of points on surfaces of the plurality of persons, and the control unit executes first clustering based on positions of the plurality of points included in the point cloud information, and second clustering based on velocities of the plurality of points included in the point cloud information.

[0116] (Supplementary note 20) The information processing apparatus according to Supplementary note 15, in which the velocity of the point cloud information is calculated based on a Doppler effect for a radar wave emitted toward an object.

[0117] (Supplementary note 21) The information processing apparatus according to Supplementary note 15, in where privacy is to be maintained, and the estimating includes specifying point cloud information of a first person among the plurality of persons and point cloud information of a second person among the plurality of persons, based on the clustered point cloud information, detecting an occurrence of an abnormal action of the plurality of persons by separately analyzing the point cloud information of the first person and the point cloud information of the second person, and notifying a terminal of a surveillance staff who monitors room where privacy is to be maintained of an alert indicating the occurrence of the abnormal action. Reference Signs List

[0118] 50 AI 100  INFORMATION PROCESSING APPARATUS 110 COMMUNICATION UNIT 120  INPUT UNIT 130  DISPLAY UNIT 140  STORAGE UNIT 141  POINT CLOUD DATA TABLE 142 CLUSTER TABLE 143  POSTURE DATA TABLE 150  CONTROL UNIT 151  PREPROCESSING UNIT 152 FIRST CLASSIFICATION UNIT 153 SECOND CLASSIFICATION UNIT 154 CORRECTION UNIT 155 ESTIMATION UNIT 156 ABNORMAL ACTION DETECTION UNIT

Claims

1. An estimation program causing a computer to execute: acquiring point cloud information generated based on areflection signal of a radar wave emitted toward aplurality of persons;clustering the point cloud information based on a spatial distance of the acquired point cloud information and a velocity of the point cloud information; andestimating a posture of each of the plurality ofpersons based on the clustered point cloud information.

2. The estimation program according to claim 1, wherein the acquiring includes acquiring time-series point cloud information, the clustering includes sequentially executing clustering on the time-series point cloud information, and the estimation program causes the computer to furtherexecute:correcting a result of current clustering, based on aresult of previous clustering based on the point cloudinformation and a result of current clustering based on thepoint cloud information.

3. The estimation program according to claim 2, wherein in the correcting, when a distance between a certain pointincluded in a cluster of a specific person included in a result of current clustering and a cluster corresponding tothe specific person included in a result of previous clustering is equal to or more than a threshold valuespecified based on a velocity of the point cloud information, the certain point is classified into a clusterof a person other than the specific person.

4. The estimation program according to claim 1, wherein the estimating includes estimating skeletoninformation of each of the plurality of persons based onthe point cloud information, andthe estimation program causes the computer to furtherexecute: specifying an abnormal action of each of theplurality of persons based on the skeleton information of each of the plurality of persons.

5. The estimation program according to claim 1, whereinthe point cloud information includes positions andvelocities of a plurality of points on surfaces of theplurality of persons, and the clustering includes firstclustering based on positions of the plurality of pointsincluded in the point cloud information, and secondclustering based on velocities of the plurality of points included in the point cloud information.

6. The estimation program according to claim 1, wherein the velocity of the point cloud information is calculated based on the Doppler effect of radar waves emitted toward an object.

7. The estimation program according to claim 1, whereineach of the plurality of persons is present in a room where privacy is to be maintained, andthe estimating includes specifying point cloud information of a first person among the plurality ofpersons and point cloud information of a second personamong the plurality of persons, based on the clusteredpoint cloud information, detecting an occurrence of anabnormal action of the plurality of persons by separately analyzing the point cloud information of the first personand the point cloud information of the second person, andnotifying a terminal of a surveillance staff who monitors aroom where privacy is to be maintained of an alertindicating the occurrence of the abnormal action.

8. An estimation method executed by a computer, the method comprising:acquiring point cloud information generated based on areflection signal of a radar wave emitted toward aplurality of persons;clustering the point cloud information based on a spatial distance of the acquired point cloud information and a velocity of the point cloud information, andestimating a posture of each of the plurality ofpersons based on the clustered point cloud information.

9. An information processing apparatus comprising a control unit that executes:acquiring point cloud information generated based on areflection signal of a radar wave emitted toward aplurality of persons;clustering the point cloud information based on a spatial distance of the acquired point cloud information and a velocity of the point cloud information, andestimating a posture of each of the plurality ofpersons based on the clustered point cloud information.

Images