Estimation program, estimation method, and information processing device

Abstract

This information processing device acquires point cloud information generated on the basis of a reflection signal from radar emitted toward a plurality of persons. On the basis of the spatial distance of the acquired point cloud information and the speed of the point cloud information, the information processing device performs clustering of the point cloud information. The information processing device estimates the orientation of each of the plurality of persons on the basis of the clustered point cloud information.

Description

Estimation program, estimation method, and information processing device

[0001] The present invention relates to an estimation program and the like.

[0002] Prior Art 1 exists for analyzing images of people with the aim of early detection of accidents caused by falls, etc. Fig. 15 is a diagram for explaining Prior Art 1.

[0003] 15, in prior art 1, video data of person 4 captured by camera 5 is input to AI (Artificial Intelligence), which determines whether person 4 has had an accident such as a fall based on the video data. In prior art 1, if AI 6 determines that person 4 has had an accident, it notifies manager 3 of an abnormality in person 4.

[0004] In the prior art 1 described with reference to FIG. 15, it is difficult to protect the privacy of the person 4 because video data is constantly acquired.

[0005] In contrast to this, there is a conventional technique 2 that estimates a person's posture using a millimeter wave sensor that can acquire information equivalent to that of a camera while protecting privacy.

[0006] FIG. 16 is a diagram for explaining Prior Art 2. In the following description, an apparatus that executes Prior Art 2 will be referred to as a "conventional apparatus." The conventional apparatus generates point cloud data 11 using a millimeter wave sensor 10. The conventional apparatus inputs the point cloud data 11 into an AI 12, thereby identifying posture data 13 of a person.

[0007] JP-T-2021-536068 A JP-A-2018-13999 WO 2021 / 220365

[0008] However, the above-mentioned prior art 2 has a problem in that it is not possible to accurately estimate the postures of multiple people.

[0009] 17 is a diagram illustrating the problem of conventional technique 2. For example, in real space, the real space at time t1 is frame f1, and the real space at time t2 is frame f2. Frames f1 and f2 contain people h1 and h2. Person h1 is moving to the right, and person h2 is moving to the left.

[0010] At time t1, the conventional device generates point cloud data 15 using the millimeter wave sensor 10. The conventional device performs distance-based clustering on the point cloud data 15 to classify the point cloud data 15 into clusters 15a and 15b. At time t1, the distance between person h1 and person h2 is large, so the point cloud is classified into cluster 15a corresponding to person h1 and cluster 15b corresponding to person h2.

[0011] The conventional device generates posture data 16a for person h1 by inputting cluster 15a to AI 12. The conventional device generates posture data 16b for person h2 by inputting cluster 15b to AI 12.

[0012] On the other hand, the conventional device generates point cloud data 17 at time t2 using the millimeter wave sensor 10. The conventional device performs distance-based clustering on the point cloud data 15 to classify the point cloud data 17 into cluster 17a. At time t2, the distance between person h1 and person h2 is short, so the point clouds of person h1 and person h2 are classified into a single cluster 17a. Therefore, when cluster 17a is input to the AI ​​12, posture data is output, which is not correct posture data.

[0013] As described with reference to FIG. 17, in the conventional technique 2, when multiple people are close to each other, the posture of each person cannot be estimated with high accuracy.

[0014] In one aspect, the present invention aims to provide an estimation program, an estimation method, and an information processing device that can accurately estimate the postures of multiple people.

[0015] In the first proposal, a computer executes the following process: the computer acquires point cloud information generated based on reflected signals from a radar irradiated toward a plurality of people; the computer clusters the acquired point cloud information based on the spatial distance and velocity of the point cloud information; and the computer estimates the posture of each of the plurality of people based on the clustered point cloud information.

[0016] The postures of multiple people can be estimated with high accuracy.

[0017] FIG. 1 is a diagram illustrating a system according to the present embodiment. FIG. 2 is a diagram illustrating the Doppler effect. FIG. 3 is a diagram illustrating an example of processing by an information processing device according to the present embodiment. FIG. 4 is a functional block diagram illustrating a configuration of an information processing device according to the present embodiment. FIG. 5 is a diagram illustrating an example of a skeletal model of a human body. FIG. 6 is a diagram illustrating an example of joint names. FIG. 7 is a diagram illustrating an example of the data structure of a point cloud data table. FIG. 8 is a diagram illustrating an example of the data structure of a cluster table. FIG. 9 is a diagram illustrating an example of the data structure of a posture data table. FIG. 10 is a diagram illustrating an anomaly detection process performed by an information processing device (1). FIG. 11 is a diagram illustrating an anomaly detection process performed by an information processing device (2). FIG. 12 is a diagram illustrating an anomaly detection process performed by an information processing device (3). FIG. 13 is a flowchart illustrating a processing procedure of an information processing device according to the present embodiment. FIG. 14 is a diagram illustrating an example of the hardware configuration of a computer that achieves the same functions as the information processing device of the embodiment. FIG. 15 is a diagram illustrating Prior Art 1. FIG. 16 is a diagram illustrating Prior Art 2. FIG. 17 is a diagram illustrating problems associated with Prior Art 2.

[0018] Hereinafter, an estimation program, an estimation method, and an information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments.

[0019] An example of a system according to the present embodiment will be described. FIG. 1 is a diagram showing the system according to the present embodiment. As shown in FIG. 1, the system includes a millimeter wave sensor 20 and an information processing device 100. The millimeter wave sensor 20 and the information processing device 100 are connected to each other via a network 25. In this embodiment, it is assumed that people h1 and h2 are included in the irradiation range of radio waves (millimeter waves) of the millimeter wave sensor 20.

[0020] The millimeter wave sensor 20 emits radio waves, detects reflected waves from the persons h1 and h2, and transmits the detection results to the information processing device 100. In the following description, for convenience of explanation, the detection result detected by the millimeter wave sensor 20 at a certain time t is referred to as "frame f ｔ " Frame f ｔ The millimeter wave sensor 20 repeatedly executes the above process at predetermined time intervals (for example, every 0.1 seconds).

[0021] The information processing device 100 receives frame f ｔ The postures of the persons h1 and h2 are estimated based on the point cloud data generated based on the above. Note that the point cloud data generated by the information processing device 100 includes position information (three-dimensional coordinates) and velocity information of each point.

[0022] For example, the information processing device 100 calculates velocity information of each point by utilizing the Doppler effect. FIG. 2 is a diagram for explaining the Doppler effect. By utilizing the Doppler effect, the velocity of a moving object can be calculated. For example, the information processing device 100 calculates the velocity of point cloud information by utilizing the Doppler effect of a radar irradiated toward an object. The object is, for example, multiple people.

[0023] When the frequency of the radio waves transmitted by the millimeter wave sensor 20 is reflected by a moving object, the frequency increases or decreases in proportion to the speed of the object. For example, for a stationary car, the frequency is static, but for a car moving at 100 km / h (approaching car), the frequency increases by approximately 4 kHz. The information processing device 100 calculates speed information from the difference in frequency between the transmitted radio waves and the received radio waves (reflected waves).

[0024] Next, an example of processing by the information processing device 100 will be described. Fig. 3 describes an example of processing by the information processing device according to this embodiment. In Fig. 3, the processing by the information processing device 100 will be described by dividing it into steps. Note that in this embodiment, for ease of viewing, an image of a person is shown in a frame, but the actual frame is information acquired from the millimeter wave sensor 20, and is information including the time, the time difference between when a radio wave is emitted and when a reflected wave arrives, a phase difference, an increase or decrease in frequency, etc.

[0025] Step S10 will now be described. ｔ Based on this, the point cloud data 30 ｔ The information processing device 100 generates the point cloud data 30 ｔ By performing clustering based on the position information of each point included in the point cloud data 30 ｔ Cluster 40 ｔａ For example, at time t, the distance between person h1 and person h2 is short, so the point groups corresponding to person h1 and person h2 are classified into one cluster 40. ｔａ It is classified as follows.

[0026] Step S11 will be described. The information processing device 100 processes the point cloud data 30 ｔ Based on the velocity information of each point included in the point cloud data 30 ｔ Cluster 40 ｔｂ , 40 ｔｃ In FIG. 3, as an example, cluster 40 ｔｂ The points classified into cluster 40 are shown by squares. ｔｃ The points classified as are shown by triangles.

[0027] Step S12 will now be described. The information processing device 100 corrects the clustering result of step S11 based on the cluster table 142 that stores past clustering results. For example, the cluster table 142 stores the clustering results of the previous frame f ｔ－１ Point cloud data 30 ｔ－１ The information processing device 100 registers the final clustering result for the point cloud data 30 ｔ－１ The clustering result of step S11 is corrected based on the clustering result (distribution of point cloud) for

[0028] The information processing device 100 detects the cluster 40 corresponding to the person h1. ｔｂ An example of a process for correcting a group of points classified into the cluster 40 will be described. The cluster corresponding to person h1 at time t-1 is called "cluster 40". ｔｂ－１ The information processing device 100 is a cluster 40 ｔｂ－１ and the velocity of person h1, the cluster 40 at time t is calculated. ｔｂ－１ The distribution of the estimated cluster 40 at time t is estimated. ｔｂ－１ The distribution of is denoted as "distribution Bb".

[0029] The information processing device 100 may specify the velocity of the person h1 in any way. For example, the information processing device 100 may specify the velocity of the person h1 in the cluster 40 ｔｂ－１ The average value, median value, etc. of the velocity information of each point included in is used as the velocity of person h1.

[0030] The information processing device 100 is configured to divide the area of ​​distribution Bb into clusters 40 ｔｂ The positions of the points in the cluster 40 are compared. ｔｂ Among the points of the distribution Bb, points that do not exist within a predetermined distance based on the area of ​​the distribution Bb are included in the cluster 40 ｔｂ The information processing device 100 classifies the identified points into the cluster 40. ｔｂ From cluster 40 ｔｃ Correct to.

[0031] The information processing device 100 selects the cluster 40 corresponding to the person h2. ｔｃAn example of a process for correcting a group of points classified into the cluster 40 will be described. The cluster corresponding to person h2 at time t-1 is called "cluster 40". ｔｃ－１ The information processing device 100 is a cluster 40 ｔｃ－１ and the velocity of person h2, cluster 40 at time t ｔｃ－１ The distribution of the estimated cluster 40 at time t is estimated. ｔｃ－１ The distribution of is denoted as "distribution Bc".

[0032] The information processing device 100 may specify the velocity of the person h2 in any way. For example, the information processing device 100 may specify the velocity of the person h2 in the cluster 40 ｔｃ－１ The average value, median value, etc. of the velocity information of each point included in is used as the velocity of person h2.

[0033] The information processing device 100 is configured to divide the area of ​​the distribution Bc and the cluster 40 ｔｃ The positions of the points in the cluster 40 are compared. ｔｃ Among the points, points that do not exist within a predetermined distance based on the area of ​​distribution Bc are included in cluster 40 ｔｃ The information processing device 100 classifies the identified points into the cluster 40. ｔｃ From cluster 40 ｔｄ Correct to.

[0034] The information processing device 100 calculates the corrected cluster 40 ｔｂ and the corrected cluster 40 ｔｃ The information is registered in the cluster table 142.

[0035] The information processing device 100 calculates the corrected cluster 40 ｔｂ By inputting this to the AI ​​50, the posture data 45 of the person h1 is obtained. ｔｂ Furthermore, the information processing device 100 estimates the corrected cluster 40 ｔｃ By inputting this to the AI ​​50, the posture data 45 of the person h2 is obtained. ｔｃ Estimate.

[0036] As described above, the information processing device 100 according to this embodiment performs clustering based on spatial distance and clustering based on velocity information, based on point cloud data obtained from the millimeter wave sensor 20. By performing such clustering, even when multiple people are close to each other, the point cloud data can be clustered to correspond to each person. Therefore, the information processing device 100 can accurately estimate the postures of multiple people using the clustering results.

[0037] Next, an example of the configuration of the information processing device 100 will be described. Fig. 4 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Fig. 4, the information processing device 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0038] The communication unit 110 performs data communication with the millimeter wave sensor 20 and the like via the network 25. For example, the communication unit 110 receives a frame f ｔ Receive.

[0039] The input unit 120 inputs various types of information to the control unit 150 of the information processing device 100 .

[0040] The display unit 130 displays information output from the control unit 150 .

[0041] The storage unit 140 includes an AI 50, a point cloud data table 141, a cluster table 142, and an orientation data table 143. The storage unit 140 is a memory or the like.

[0042] AI50 is a machine learning model that takes a point cloud as input and outputs posture data of a person. AI50 is, for example, a neural network (NN).

[0043] For example, posture data of a person is data corresponding to skeletal data. Here, skeletal data is data in which two-dimensional or three-dimensional coordinates are set for multiple joints defined in a skeletal model of the human body. Here, the coordinates of each joint in the skeletal data are three-dimensional coordinates. FIG. 5 is a diagram showing an example of a skeletal model of the human body. For example, as shown in FIG. 5, the skeletal model of the human body is defined by 21 joints ar0 to ar20.

[0044] The relationship between the joints ar0 to ar20 shown in Fig. 5 and the joint names is as shown in Fig. 6. Fig. 6 is a diagram showing examples of joint names. For example, the joint name of joint ar0 is "SPINE_BASE". The joint names of joints ar1 to ar20 are as shown in Fig. 6, and explanations will be omitted.

[0045] The point cloud data table 141 is a table that holds point cloud data generated from frames at each time. FIG. 7 is a diagram showing an example of the data structure of the point cloud data table. As shown in FIG. 7, the point cloud data table 141 associates time with point cloud data. The time is the time when the millimeter wave sensor 20 generates a frame (detection result). The point cloud data is point cloud data generated from the frame at the corresponding time. For example, the point cloud data is generated by the preprocessing unit 151, which will be described later. As described above, in the point cloud data, position information (three-dimensional coordinates) and velocity information are associated with each point. Furthermore, point identification information that uniquely identifies the point is assigned to each point.

[0046] The cluster table 142 holds information about clusters at each time. Fig. 8 is a diagram showing an example of the data structure of the cluster table. As shown in Fig. 8, the cluster table 142 associates time with first cluster data, second cluster data, and third cluster data.

[0047] The first cluster data includes data of the clustering results based on spatial distances performed on point cloud data at a certain time. The clustering based on spatial distances is the clustering described in step S10 of FIG. 3. For example, the first cluster data associates cluster identification information that identifies a cluster with point identification information of each point (point cloud) belonging to the corresponding cluster. At each time, each cluster that is assumed to represent the same person is assigned the same cluster identification information.

[0048] The second cluster data includes data of the clustering result based on velocity information, which is performed on the point cloud data at a certain time. The clustering based on velocity information is the clustering described in step S11 of FIG. 3. For example, the second cluster data associates cluster identification information for identifying a cluster with point identification information of each point (point cloud) belonging to the corresponding cluster. At each time, each cluster assumed to represent the same person is assigned the same cluster identification information.

[0049] The third cluster data includes data of the clustering result after correction of the second cluster data. The correction of the cluster is the correction described in step S12 of FIG.

[0050] The posture data table 143 holds posture data estimated at each time. Fig. 9 is a diagram showing an example of the data structure of the posture data table. As shown in Fig. 9, the posture data table 143 associates time with posture data. The posture data is data corresponding to the skeleton data described in Fig. 5. The posture data is assigned person identification information that identifies the person corresponding to the posture data.

[0051] Returning to the description of Fig. 4, the control unit 150 includes a preprocessing unit 151, a first classification unit 152, a second classification unit 153, a correction unit 154, an estimation unit 155, and an abnormal behavior detection unit 156. The control unit 150 is a central processing unit (CPU), a graphics processing unit (GPU), or the like.

[0052] The preprocessing unit 151 receives frame f from the millimeter wave sensor 20 and generates point cloud data based on frame f. The preprocessing unit 151 associates time with the point cloud data and registers the data in the point cloud data table 141. The preprocessing unit 151 repeatedly executes the above process each time it receives frame f.

[0053] For example, the preprocessing unit 151 analyzes the time difference, phase difference, frequency increase / decrease, etc., between the emission of radio waves and the arrival of reflected waves, to extract information on the distance to the person and information on the person's speed, and specifies position information and speed information for each point on the person's surface from the extraction results. Note that the preprocessing unit 151 may generate point cloud data from the frame f using any well-known technology.

[0054] The first classification unit 152 acquires point cloud data from the point cloud data table 141 and performs clustering based on the position information of each point included in the point cloud data. As a result of this clustering, points with similar position information are classified into the same cluster. The first classification unit 152 associates the first cluster data, which is the clustering result, with a time and registers it in the cluster table 142.

[0055] The clustering by the first classification unit 152 corresponds to the clustering described in step S10 of Fig. 3. The first classification unit 152 repeatedly executes the above process every time new point cloud data is registered in the point cloud data table 141.

[0056] The second classification unit 153 acquires point cloud data from the point cloud data table 141 and performs clustering based on the velocity information of each point included in the point cloud data. By this clustering, points having similar velocity information are classified into the same cluster. The second classification unit 153 registers the second cluster data, which is the clustering result, in the cluster table 142 in association with time.

[0057] The clustering by the second classification unit 153 corresponds to the clustering described in step S11 of Fig. 3. The second classification unit 153 repeatedly executes the above process every time new point cloud data is registered in the point cloud data table 141.

[0058] The correction unit 154 corrects the second cluster data, and registers the third cluster data, which is the result of the correction, in the cluster table 142 in association with the time.

[0059] For example, the correction unit 154 obtains the third cluster data corresponding to time t−1, which is one time before the current time t, from the cluster table 142. The correction unit 154 corrects the second cluster data based on the third cluster data corresponding to time t−1 and the second cluster data corresponding to time t1, thereby generating the third cluster data for time t1 and registering the third cluster data for time t1 in the cluster table 142.

[0060] The correction performed by the correction unit 154 corresponds to the correction described in step 12 of Fig. 3. The correction unit 154 repeatedly executes the above process every time new second cluster data is registered in the cluster table 142.

[0061] If the first cluster data and the second cluster data at the same time satisfy the similarity condition, the correction unit 154 registers the first cluster data as the third cluster data in the cluster table 142. For example, if multiple people are far apart, the number of clusters included in the first cluster data and the distribution of each cluster will be similar to the number of clusters and the distribution of each cluster included in the second cluster data. Typically, if multiple people are far apart, the first cluster data has a high reliability, and therefore the first cluster data can be used as the correction result for the second cluster data. The similarity condition may be set in any way, but for example, the similarity condition is a condition that "the number of clusters is the same and the difference in the distribution of each cluster is less than a threshold."

[0062] The estimation unit 155 acquires the third cluster data registered in the cluster table 142 and estimates posture data of the person by inputting the third cluster data to the AI ​​50. The estimation unit 155 registers the posture data in the posture data table 143 in association with time.

[0063] When the third cluster data includes a cluster for each person, the estimation unit 155 inputs the point clouds belonging to the cluster for each person to the AI ​​50, and estimates the posture data of each person. The posture data estimation performed by the estimation unit 155 corresponds to the posture data estimation described in step 12 of FIG. 3 .

[0064] The estimation unit 155 repeatedly executes the above process at each time, whereby time-series posture data is registered in the posture data table 143 for each person (each piece of personal identification information).

[0065] The abnormal behavior detection unit 156 detects abnormal behavior of a person based on the posture data table 143. The abnormal behavior detection unit 156 detects abnormal behavior using a rule table or AI for abnormal detection. The rule table is a table that associates the transition of each joint position in posture data with the type of abnormal behavior. The AI ​​for abnormal detection is a machine learning model that receives time-series posture data as input and outputs the type of abnormal behavior. The AI ​​for abnormal detection is an NN or the like.

[0066] When the abnormal behavior detection unit 156 detects an abnormality, it may transmit abnormal behavior information to a predetermined external terminal. The abnormal behavior information is associated with frames of a predetermined period based on the time when the abnormal behavior was detected, point cloud data, the content of the abnormal behavior, the time, etc.

[0067] Here, an example of a notification in which an alert is sent to the terminal of a monitor monitoring a specific room in a building will be described. The information processing device 100 monitors multiple people present in the specific room. The multiple people are present in a room where privacy must be maintained. The specific room may be, for example, a room constituting a space where privacy must be maintained, such as a private room in a hospital or hotel, a changing room, a nursing room, or a restroom. In this case, the information processing device 100 identifies, for example, point cloud information of a first person constituting the multiple people and point cloud information of a second person constituting the multiple people based on the clustered point cloud information. Next, the information processing device 100 detects abnormal behavior among the multiple people by, for example, separately analyzing the point cloud information of the identified first person and the point cloud information of the second person. The information processing device 100 then notifies, for example, an alert indicating the occurrence of abnormal behavior to the terminal of the monitor monitoring the room where privacy must be maintained. More specifically, for example, the information processing device 100 displays, on the terminal, the location of the abnormality and an alert indicating the occurrence of abnormal behavior.

[0068] Here, an example of the anomaly detection process by the information processing device 100 will be described for each case with reference to Fig. 10, Fig. 11, and Fig. 12. Fig. 10 to Fig. 12 are diagrams illustrating the anomaly detection process by the information processing device.

[0069] First, FIG. 10 will be described. FIG. 10 shows an example of abnormality detection when a nurse h4 is providing rescue care to a patient h3. The information processing device 100 detects an abnormality in a frame f at a time t1. ｔ１ and executes the above process to obtain the cluster 61 corresponding to patient h3. ｔ１ｂ The information processing device 100 identifies the cluster 61 ｔ１ｂ By inputting the point cloud into AI 50, posture data 71 ｔ１ｂ The abnormal behavior detection unit 156 obtains the posture data 71 ｔ１ｂ Based on this, the abnormal behavior of "falling" is detected.

[0070] At time t2, the information processing device 100 ｔ２ and executes the above process to obtain the cluster 61 corresponding to patient h3.ｔ２ｂ and cluster 61 corresponding to nurse h4. ｔ２ｃ The information processing device 100 separately identifies the cluster 61 ｔ２ｂ By inputting the point cloud into AI 50, posture data 71 ｔ２ｂ The information processing device 100 obtains the cluster 61 ｔ２ｃ By inputting the point cloud into AI 50, posture data 71 ｔ２ｃ The abnormal behavior detection unit 156 obtains the posture data 71 ｔ２ｂ Based on this, the abnormal behavior of "falling" is detected.

[0071] 10, even if the distance between the nurse h4 and the patient h3 is short, the abnormal behavior of the patient h3, "falling," can be detected. In contrast, in the conventional technique 2, when the distance between the nurse h4 and the patient h3 is short, the cluster 61 corresponding to the patient h3 ｔ２ｂ and cluster 61 corresponding to nurse h4. ｔ２ｃ It is not possible to identify these separately, making it difficult to detect anomalies.

[0072] Next, a description will be given of FIG. 11. FIG. 11 shows an example of anomaly detection in the private spaces of persons h5 and h6. At time t1, the information processing device 100 detects anomalies in a frame f ｔ１ and executes the above process to obtain the cluster 62 corresponding to person h5. ｔ１ｂ and cluster 62 corresponding to person h6 ｔ１ｃ The information processing device 100 separately identifies the cluster 62 ｔ１ｂ By inputting the point cloud into the AI ​​50, the posture data 72 ｔ１ｂ The information processing device 100 obtains the cluster 62 ｔ１ｃ By inputting the point cloud into the AI ​​50, the posture data 72 ｔ１ｃ get.

[0073] At time t2, the information processing device 100 ｔ２ and executes the above process to obtain the cluster 62 corresponding to person h5. ｔ２ｂ and cluster 62 corresponding to person h6 ｔ２ｃ The information processing device 100 separately identifies the cluster 62 ｔ２ｂBy inputting the point cloud into the AI ​​50, the posture data 72 ｔ２ｂ The information processing device 100 obtains the cluster 62 ｔ２ｃ By inputting the point cloud into the AI ​​50, the posture data 72 ｔ２ｃ The abnormal behavior detection unit 156 obtains the posture data 72 ｔ２ｂ Based on this, the abnormal behavior of "falling" is detected.

[0074] 11, even when the distance between person h5 and person h6 is short, the abnormal behavior of person h5, "falling," can be detected. Furthermore, based on the processing result of FIG. 11, the information processing device 100 can identify that person h5 and person h6 collided with each other while passing each other, causing person h5 to fall.

[0075] Next, FIG. 12 will be described. FIG. 12 shows an example of anomaly detection in the private spaces of persons h7 and h8. At time t1, the information processing device 100 detects anomalies in a frame f ｔ１ and executes the above process to obtain the cluster 63 corresponding to person h7. ｔ１ｂ and cluster 63 corresponding to person h8 ｔ１ｃ The information processing device 100 separately identifies the cluster 63 ｔ１ｂ By inputting the point cloud into AI 50, posture data 73 ｔ１ｂ The information processing device 100 obtains the cluster 63 ｔ１ｃ By inputting the point cloud into AI 50, posture data 73 ｔ１ｃ get.

[0076] At time t2, the information processing device 100 ｔ２ and executes the above process to obtain the cluster 63 corresponding to person h7. ｔ２ｂ and cluster 63 corresponding to person h8 ｔ２ｃ The information processing device 100 separately identifies the cluster 63 ｔ２ｂ By inputting the point cloud into AI 50, posture data 73 ｔ２ｂ The information processing device 100 obtains the cluster 63 ｔ２ｃ By inputting the point cloud into AI 50, posture data 73 ｔ２ｃ For example, the abnormal behavior detection unit 156 obtains the posture data 73 ｔ２ｂBased on this, the abnormal behavior of person h7, "approaching the direction of another person at an abnormal speed and raising his right knee", is detected.

[0077] At time t3, the information processing device 100 ｔ３ and executes the above process to obtain the cluster 63 corresponding to person h7. ｔ２ｂ The person h8 has fallen down (outside the range of the millimeter wave sensor 20), and point cloud data is not generated for the person h8. ｔ３ｂ By inputting the point cloud into AI 50, posture data 73 ｔ３ｂ get.

[0078] 12, even when the distance between person h7 and person h8 is short, the abnormal behavior of person h7, "approaching another person at an abnormal speed and raising his right knee," can be detected. For example, "approaching another person at an abnormal speed and raising his right knee" corresponds to a violent act against another person.

[0079] Next, an example of a processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 13 is a flowchart showing the processing procedure of the information processing device according to this embodiment. As shown in Fig. 13, the pre-processing unit 151 of the information processing device 100 acquires frames from the millimeter wave sensor 20 and generates point cloud data (step S101).

[0080] The first classification unit 152 of the information processing device 100 performs clustering based on the position information of the point cloud data (step S102), and the second classification unit 153 of the information processing device 100 performs clustering based on the velocity information of the point cloud data (step S103).

[0081] The correction unit 154 of the information processing device 100 acquires the third cluster data at time t-1 from the cluster table 142 (step S104). The correction unit 154 determines whether to correct the second cluster data based on the third cluster data corresponding to time t-1 and the second cluster data corresponding to time t1 (step S105).

[0082] If the correction unit 154 determines to correct the second cluster data at time t (Yes at step S106), it generates third cluster data at time t (step S107). On the other hand, if the correction unit 154 determines not to correct the second cluster data at time t (No at step S106), it sets the second cluster data at time t to the third cluster data at time t (step S108).

[0083] The correction unit 154 registers the third cluster data at time t in the cluster table 142 (step S109).

[0084] The estimation unit 155 of the information processing device 100 inputs the third cluster data at time t to the AI ​​50 and estimates posture data (step S110). The abnormal behavior detection unit 156 of the information processing device 100 performs abnormal behavior detection based on the posture data (step S111). The abnormal behavior detection unit 156 outputs the result of the abnormal behavior detection (step S112).

[0085] The processing procedure of the information processing device 100 has been described above with reference to Fig. 13. The information processing device 100 repeatedly executes the processing described with reference to Fig. 13 at each time point.

[0086] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 clusters the point cloud data based on the spatial distance of the point cloud data and the velocity of the point cloud data, and estimates the postures of each of multiple people based on the clustered point cloud data. This makes it possible to accurately estimate the postures of multiple people.

[0087] The information processing device 100 sequentially performs clustering on the time-series point cloud data, and adjusts the current clustering result based on the previous clustering result based on the point cloud data and the current clustering result based on the point cloud data, thereby improving the accuracy of the clustering result for each person.

[0088] When the distance between a point included in a cluster of a specific person included in the current clustering result and a cluster corresponding to the specific person included in the previous clustering result is equal to or greater than a threshold value determined based on the speed of the point cloud data, the information processing device 100 classifies the point into a cluster of a person other than the specific person. This makes it possible to improve the accuracy of the clustering result for each person.

[0089] The information processing device 100 identifies abnormal behavior of each of the multiple people based on the skeletal information of each of the multiple people. This makes it possible to detect abnormal behavior of a person even when the millimeter wave sensor 20 is used.

[0090] Next, a description will be given of an example of the hardware configuration of a computer that realizes the same functions as the above-described information processing device 100. Fig. 14 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0091] 14, computer 200 has a CPU 201 that executes various types of arithmetic processing, an input device 202 that accepts data input from a user, and a display 203. Computer 200 also has a communication device 204 that exchanges data with external devices via a wired or wireless network, and an interface device 205. Computer 200 also has a RAM 206 that temporarily stores various types of information, and a hard disk drive 207. Each of devices 201 to 207 is connected to a bus 208.

[0092] The hard disk drive 207 stores a preprocessing program 207 a, a first classification program 207 b, a second classification program 207 c, a correction program 207 d, an estimation program 207 e, and an abnormal behavior detection program 207 f. The CPU 201 reads out each of the programs 207 a to 207 f and loads them into the RAM 206.

[0093] The preprocessing program 207a functions as a preprocessing process 206a. The first classification program 207b functions as a first classification process 206b. The second classification program 207c functions as a second classification process 206c. The correction program 207d functions as a correction process 206d. The estimation program 207e functions as an estimation process 206e. The abnormal behavior detection program 207f functions as an abnormal behavior detection process 206f.

[0094] The processing of the preprocessing process 206a corresponds to the processing of the preprocessing unit 151. The processing of the first classification process 206b corresponds to the processing of the first classification unit 152. The processing of the second classification process 206c corresponds to the processing of the second classification unit 153. The processing of the correction process 206d corresponds to the processing of the correction unit 154. The processing of the estimation process 206e corresponds to the processing of the estimation unit 155. The processing of the abnormal behavior detection process 206f corresponds to the processing of the abnormal behavior detection unit 156.

[0095] It should be noted that each of the programs 207a to 207f does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207f.

[0096] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0097] (Supplementary Note 1) An estimation program that causes a computer to execute the following processes: acquire point cloud information generated based on reflected signals of a radar irradiated toward a plurality of people; cluster the acquired point cloud information based on spatial distances and velocities of the point cloud information; and estimate the postures of each of the plurality of people based on the clustered point cloud information.

[0098] (Supplementary Note 2) The estimation program according to Supplementary Note 1, wherein the acquiring process acquires time-series point cloud information, the clustering process sequentially executes clustering on the time-series point cloud information, and the computer is further made to execute a process of correcting the current clustering result based on the previous clustering result based on the point cloud information and the current clustering result based on the point cloud information.

[0099] (Appendix 3) The estimation program described in Appendix 2, characterized in that the correction process classifies a point included in a cluster of a specific person included in the result of the current clustering into a cluster of a person other than the specific person if the distance between the point included in the result of the previous clustering and the cluster corresponding to the specific person is equal to or greater than a threshold determined based on the speed of the point cloud information.

[0100] (Appendix 4) The estimation program described in Appendix 1, characterized in that the estimation process estimates skeletal information of each of the plurality of persons based on the point cloud information, and further causes the computer to execute a process of identifying abnormal behavior of each of the plurality of persons based on the skeletal information of each of the plurality of persons.

[0101] (Appendix 5) The point cloud information includes positions and velocities of multiple points on the surfaces of the multiple people, and the clustering process performs a first clustering based on the positions of the multiple points included in the point cloud information, and a second clustering based on the velocities of the multiple points included in the point cloud information.

[0102] (Appendix 6) The estimation program according to appendix 1, characterized in that the velocity of the point cloud information is calculated based on the Doppler effect of a radar irradiated toward an object.

[0103] (Supplementary Note 7) The estimation program according to Supplementary Note 1, wherein each of the plurality of persons is present in a room where privacy is to be maintained, and the estimation process identifies point cloud information of a first person constituting the plurality of persons and point cloud information of a second person constituting the plurality of persons based on the clustered point cloud information, and detects the occurrence of abnormal behavior in the plurality of persons by separately analyzing the point cloud information of the first person and the point cloud information of the second person, and notifies an alert indicating the occurrence of the abnormal behavior to a terminal of a monitor monitoring the room where privacy is to be maintained.

[0104] (Supplementary Note 8) An estimation method characterized by having a computer execute the following processes: acquiring point cloud information generated based on reflected signals of a radar irradiated toward a plurality of people; clustering the acquired point cloud information based on spatial distances and velocities of the point cloud information; and estimating the postures of each of the plurality of people based on the clustered point cloud information.

[0105] (Supplementary Note 9) The estimation method according to Supplementary Note 8, characterized in that the acquiring process acquires time-series point cloud information, the clustering process sequentially performs clustering on the time-series point cloud information, and further performs a process of correcting the current clustering result based on the previous clustering result based on the point cloud information and the current clustering result based on the point cloud information.

[0106] (Appendix 10) The estimation method described in Appendix 9, characterized in that the correction process classifies a point included in a cluster of a specific person included in the result of the current clustering into a cluster of a person other than the specific person when the distance between the point included in the result of the previous clustering and the cluster corresponding to the specific person is equal to or greater than a threshold determined based on the speed of the point cloud information.

[0107] (Appendix 11) The estimation method described in Appendix 8, characterized in that the estimation process estimates skeletal information of each of the plurality of persons based on the point cloud information, and further executes a process of identifying abnormal behavior of each of the plurality of persons based on the skeletal information of each of the plurality of persons.

[0108] (Appendix 12) The estimation method described in Appendix 8, wherein the point cloud information includes positions and velocities of multiple points on the surfaces of the multiple people, and the clustering process performs a first clustering based on the positions of the multiple points included in the point cloud information, and a second clustering based on the velocities of the multiple points included in the point cloud information.

[0109] (Supplementary Note 13) The estimation method according to Supplementary Note 8, characterized in that the velocity of the point cloud information is calculated based on the Doppler effect of a radar irradiated toward an object.

[0110] (Supplementary Note 14) The estimation method according to Supplementary Note 8, wherein each of the plurality of persons is present in a room where privacy is to be maintained, and the estimation process identifies point cloud information of a first person constituting the plurality of persons and point cloud information of a second person constituting the plurality of persons based on the clustered point cloud information, and detects the occurrence of abnormal behavior in the plurality of persons by separately analyzing the point cloud information of the first person and the point cloud information of the second person, and notifies an alert indicating the occurrence of the abnormal behavior to a terminal of a monitor monitoring the room where privacy is to be maintained.

[0111] (Supplementary Note 15) An information processing device having a control unit that executes a process of acquiring point cloud information generated based on reflected signals of a radar irradiated toward a plurality of people, clustering the acquired point cloud information based on spatial distances and velocities of the point cloud information, and estimating the postures of each of the plurality of people based on the clustered point cloud information.

[0112] (Appendix 16) The information processing device described in Appendix 15 is characterized in that the control unit acquires time-series point cloud information, performs clustering on the time-series point cloud information in sequence, and further performs a process of correcting the result of the current clustering based on the result of the previous clustering based on the point cloud information and the result of the current clustering based on the point cloud information.

[0113] (Appendix 17) The information processing device described in Appendix 16 is characterized in that the control unit classifies a point included in a cluster of a specific person included in the result of the current clustering and a cluster corresponding to the specific person included in the result of the previous clustering, when the distance between the point included in the cluster of a specific person included in the result of the current clustering is greater than or equal to a threshold determined based on the speed of the point cloud information.

[0114] (Appendix 18) The information processing device described in Appendix 15, characterized in that the control unit estimates skeletal information of each of the multiple people based on the point cloud information, and further performs processing to identify abnormal behavior of each of the multiple people based on the skeletal information of each of the multiple people.

[0115] (Appendix 19) The information processing device described in Appendix 15, characterized in that the point cloud information includes positions and velocities of multiple points on the surface of the multiple people, and the control unit performs first clustering based on the positions of the multiple points included in the point cloud information, and performs second clustering based on the velocities of the multiple points included in the point cloud information.

[0116] (Supplementary Note 20) The information processing device according to Supplementary Note 15, characterized in that the velocity of the point cloud information is calculated based on the Doppler effect of a radar irradiated toward an object.

[0117] (Supplementary Note 21) The information processing device according to Supplementary Note 15, wherein each of the plurality of persons is present in a room where privacy is to be maintained, and the estimation process identifies point cloud information of a first person constituting the plurality of persons and point cloud information of a second person constituting the plurality of persons based on the clustered point cloud information, and detects the occurrence of abnormal behavior in the plurality of persons by separately analyzing the point cloud information of the first person and the point cloud information of the second person, and notifies an alert indicating the occurrence of the abnormal behavior to a terminal of a monitor monitoring the room where privacy is to be maintained.

[0118] 50 AI 100 Information processing device 110 Communication unit 120 Input unit 130 Display unit 140 Storage unit 141 Point cloud data table 142 Cluster table 143 Posture data table 150 Control unit 151 Preprocessing unit 152 First classification unit 153 Second classification unit 154 Correction unit 155 Estimation unit 156 Abnormal behavior detection unit

Claims

1. An estimation program that causes a computer to execute the following processes: acquire point cloud information generated based on reflected signals from radar irradiated toward multiple people; cluster the acquired point cloud information based on the spatial distance and velocity of the point cloud information; and estimate the posture of each of the multiple people based on the clustered point cloud information.

2. The estimation program according to claim 1, characterized in that the acquiring process acquires time-series point cloud information, the clustering process sequentially performs clustering on the time-series point cloud information, and the computer is further made to execute a process of correcting the current clustering result based on the previous clustering result based on the point cloud information and the current clustering result based on the point cloud information.

3. The estimation program according to claim 2, characterized in that the correction process classifies a point included in a cluster of a specific person included in the results of the current clustering into a cluster corresponding to the specific person included in the results of the previous clustering, if the distance between the point included in the cluster of the specific person included in the results of the previous clustering is equal to or greater than a threshold determined based on the speed of the point cloud information.

4. The estimation program according to claim 1, characterized in that the estimation process further causes the computer to execute a process of estimating skeletal information for each of the plurality of persons based on the point cloud information, and identifying abnormal behavior of each of the plurality of persons based on the skeletal information for each of the plurality of persons.

5. The estimation program described in claim 1, characterized in that the point cloud information includes positions and velocities of multiple points on the surfaces of the multiple people, and the clustering process performs a first clustering based on the positions of the multiple points included in the point cloud information, and a second clustering based on the velocities of the multiple points included in the point cloud information.

6. The estimation program according to claim 1, wherein the velocity of the point cloud information is calculated based on the Doppler effect of a radar irradiated toward an object.

7. The estimation program of claim 1, wherein each of the plurality of persons is present in a room where privacy should be maintained, and the estimation process identifies point cloud information of a first person constituting the plurality of persons and point cloud information of a second person constituting the plurality of persons based on the clustered point cloud information, and detects the occurrence of abnormal behavior in the plurality of persons by separately analyzing the point cloud information of the first person and the point cloud information of the second person, and notifies an alert indicating the occurrence of the abnormal behavior to a terminal of a monitor monitoring the room where privacy should be maintained.

8. An estimation method characterized by the steps of: acquiring point cloud information generated based on reflected signals from radar irradiated toward multiple people; clustering the acquired point cloud information based on spatial distances and velocities of the point cloud information; and estimating the postures of each of the multiple people based on the clustered point cloud information.

9. An information processing device having a control unit that executes the following processes: acquiring point cloud information generated based on reflected signals from radar irradiated toward multiple people; clustering the acquired point cloud information based on spatial distances and velocities of the point cloud information; and estimating the postures of each of the multiple people based on the clustered point cloud information.

Images