Estimation program, estimation method and information processing device

Abstract

To accurately estimate postures of a plurality of persons.SOLUTION: An information processing device acquires point cloud information generated based on a reflection signal of a radar emitted toward a plurality of persons. The information processing device clusters the acquired point cloud information based on a spatial distance of and a velocity of the point cloud information. The information processing device estimates a posture of each of the plurality of persons based on the clustered point cloud information.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an estimation program and the like. [Background technology]

[0002] Prior Art 1 exists for analyzing video images of people with the aim of early detection of accidents caused by falls, etc. Fig. 15 is a diagram for explaining Prior Art 1.

[0003] As shown in Fig. 15, in conventional technology 1, video data of person 4 captured by camera 5 is input to AI (Artificial Intelligence), which determines whether person 4 has had an accident such as a fall based on the video data. In conventional technology 1, if AI 6 determines that person 4 has had an accident, it notifies manager 3 of an abnormality in person 4.

[0004] In the prior art 1 described with reference to FIG. 15, it is difficult to protect the privacy of the person 4 because video data is constantly acquired.

[0005] In contrast to this, there is prior art 2, which estimates a person's posture using a millimeter wave sensor that can acquire information equivalent to that of a camera while protecting privacy.

[0006] FIG. 16 is a diagram for explaining Prior Art 2. In the following description, an apparatus that executes Prior Art 2 will be referred to as a "conventional apparatus." The conventional apparatus generates point cloud data 11 using a millimeter wave sensor 10. The conventional apparatus inputs the point cloud data 11 into an AI 12, thereby identifying posture data 13 of a person. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Special Publication No. 2021-536068 [Patent Document 2] Japanese Patent Application Publication No. 2018-13999 [Patent Document 3] International Publication No. 2021 / 220365 Summary of the Invention [Problem to be solved by the invention]

[0008] However, the above-mentioned prior art 2 has a problem in that it is not possible to accurately estimate the postures of multiple people.

[0009] 17 is a diagram illustrating the problem of conventional technique 2. For example, in real space, the real space at time t1 is frame f1, and the real space at time t2 is frame f2. Frames f1 and f2 contain people h1 and h2. Person h1 is moving to the right, and person h2 is moving to the left.

[0010] At time t1, the conventional device generates point cloud data 15 using a millimeter wave sensor 10. The conventional device performs distance-based clustering on the point cloud data 15 to classify the point cloud data 15 into clusters 15a and 15b. At time t1, the distance between person h1 and person h2 is large, so the point cloud is classified into cluster 15a corresponding to person h1 and cluster 15b corresponding to person h2.

[0011] The conventional device generates posture data 16a for person h1 by inputting cluster 15a to AI 12. The conventional device generates posture data 16b for person h2 by inputting cluster 15b to AI 12.

[0012] On the other hand, at time t2, the conventional device generates point cloud data 17 using the millimeter wave sensor 10. The conventional device performs distance-based clustering on the point cloud data 15 to classify the point cloud data 17 into cluster 17a. At time t2, the distance between person h1 and person h2 is short, so the point clouds of person h1 and person h2 are classified into one cluster 17a. Therefore, when cluster 17a is input to AI 12, posture data is output, which is not correct posture data.

[0013] As explained in FIG. 17, in the conventional technique 2, when multiple people are close to each other, it is not possible to accurately estimate the posture of each person.

[0014] In one aspect, the present invention aims to provide an estimation program, an estimation method, and an information processing device that can accurately estimate the postures of multiple people. [Means for solving the problem]

[0015] In the first proposal, a computer executes the following process: The computer acquires point cloud information generated based on reflected signals from a radar beam directed at multiple people; The computer clusters the acquired point cloud information based on the spatial distance and velocity of the point cloud information; The computer estimates the posture of each of the multiple people based on the clustered point cloud information. [Effects of the Invention]

[0016] The postures of multiple people can be estimated with high accuracy. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram showing a system according to this embodiment. [Figure 2] FIG. 2 is a diagram for explaining the Doppler effect. [Figure 3] FIG. 3 is a diagram illustrating an example of processing performed by the information processing device according to the present embodiment. [Figure 4] FIG. 4 is a functional block diagram showing the configuration of the information processing device according to this embodiment. [Figure 5] FIG. 5 is a diagram showing an example of a skeletal model of a human body. [Figure 6] FIG. 6 is a diagram showing an example of joint names. [Figure 7] FIG. 7 is a diagram illustrating an example of the data structure of the point cloud data table. [Figure 8]FIG. 8 is a diagram illustrating an example of the data structure of the cluster table. [Figure 9] FIG. 9 is a diagram illustrating an example of the data structure of the posture data table. [Figure 10] FIG. 10 is a diagram (1) illustrating the process of detecting an anomaly by an information processing device. [Figure 11] FIG. 11 is a diagram (2) illustrating the process of detecting an anomaly by the information processing device. [Figure 12] FIG. 12 is a diagram (3) for explaining the process of detecting anomalies by an information processing device. [Figure 13] FIG. 13 is a flowchart illustrating a processing procedure of the information processing device according to the present embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. [Figure 15] FIG. 15 is a diagram for explaining the prior art 1. In FIG. [Figure 16] FIG. 16 is a diagram for explaining the prior art 2. In FIG. [Figure 17] FIG. 17 is a diagram for explaining the problem of the conventional technique 2. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, an estimation program, an estimation method, and an information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. [Example]

[0019] An example of a system according to this embodiment will be described. FIG. 1 is a diagram showing the system according to this embodiment. As shown in FIG. 1, this system includes a millimeter wave sensor 20 and an information processing device 100. The millimeter wave sensor 20 and the information processing device 100 are connected to each other via a network 25. In this embodiment, it is assumed that people h1 and h2 are included in the irradiation range of the radio waves (millimeter waves) of the millimeter wave sensor 20.

[0020] The millimeter wave sensor 20 emits radio waves, detects reflected waves from the persons h1 and h2, and transmits the detection results to the information processing device 100. In the following description, for convenience of explanation, the detection result detected by the millimeter wave sensor 20 at a certain time t is referred to as "frame f t " Frame f t The time includes the time of detection, the time difference between when the radio wave is emitted and when the reflected wave arrives, the phase difference, the increase or decrease in frequency, etc. The millimeter wave sensor 20 repeatedly executes the above process every predetermined time (for example, every 0.1 seconds).

[0021] The information processing device 100 receives a frame f t The postures of the persons h1 and h2 are estimated based on the point cloud data generated based on the above. Note that the point cloud data generated by the information processing device 100 includes position information (three-dimensional coordinates) and velocity information of each point.

[0022] For example, the information processing device 100 calculates the velocity information of each point by utilizing the Doppler effect. FIG. 2 is a diagram for explaining the Doppler effect. By utilizing the Doppler effect, the velocity of a moving object can be calculated. For example, the information processing device 100 calculates the velocity of point cloud information by utilizing the Doppler effect of a radar irradiated toward an object. The object is, for example, multiple people.

[0023] When the frequency of the radio waves transmitted by the millimeter wave sensor 20 is reflected by a moving object, the frequency increases or decreases in proportion to the speed of the object. For example, for a stationary car, the frequency is static, but for a car moving at 100 km / h (approaching car), the frequency increases by approximately 4 kHz. The information processing device 100 calculates speed information from the difference in frequency between the transmitted radio waves and the received radio waves (reflected waves).

[0024] Next, an example of processing by the information processing device 100 will be described. Fig. 3 describes an example of processing by the information processing device according to this embodiment. In Fig. 3, the processing by the information processing device 100 will be explained by dividing it into steps. Note that in this embodiment, for ease of viewing, an image of a person is shown in a frame, but the actual frame is information acquired from the millimeter wave sensor 20, and is information including time, the time difference between when a radio wave is emitted and when a reflected wave arrives, a phase difference, an increase or decrease in frequency, etc.

[0025] Step S10 will now be described. The information processing device 100 t Based on this, point cloud data 30 t The information processing device 100 generates the point cloud data 30 t By performing clustering based on the position information of each point contained in the point cloud data, t Cluster 40 ta For example, at time t, the distance between person h1 and person h2 is short, so the point groups corresponding to person h1 and person h2 are classified into one cluster 40. ta It is classified as follows.

[0026] Step S11 will be described. The information processing device 100 processes the point cloud data 30 t Based on the velocity information of each point contained in the point cloud data 30 t Cluster 40 tb ,40 tc As an example, in Figure 3, cluster 40 tb The points classified into cluster 40 are shown as squares. tc The points classified as are shown by triangles.

[0027] Step S12 will now be described. The information processing device 100 corrects the clustering result of step S11 based on the cluster table 142 that stores past clustering results. For example, the cluster table 142 stores the previous (time t-1) frame f t-1 Point cloud data 30 t-1 The information processing device 100 registers the final clustering result for the point cloud data 30. t-1The clustering result of step S11 is corrected based on the clustering result (distribution of point cloud) for

[0028] The information processing device 100 detects the cluster 40 corresponding to the person h1. tb An example of a process for correcting a group of points classified into the above will be described. The cluster corresponding to person h1 at time t-1 is referred to as "cluster 40". tb-1 The information processing device 100 is a cluster 40. tb-1 From the velocity of person h1, cluster 40 at time t tb-1 The distribution of cluster 40 at estimated time t is estimated. tb-1 The distribution of is denoted as "distribution Bb".

[0029] The information processing device 100 may specify the velocity of the person h1 in any way. For example, the information processing device 100 may specify the velocity of the person h1 in the cluster 40. tb-1 The average value, median value, etc. of the velocity information of each point included in is used as the velocity of person h1.

[0030] The information processing device 100 is configured to divide the area of ​​distribution Bb into clusters 40 tb Compare the positions of each point in the cluster 40 tb Among the points, points that do not exist within a predetermined distance based on the area of ​​distribution Bb are classified into cluster 40 tb The information processing device 100 classifies the identified points into the cluster 40. tb From Cluster 40 tc Correct to.

[0031] The information processing device 100 detects the cluster 40 corresponding to the person h2. tc An example of a process for correcting a group of points classified into the cluster 40 will be described. tc-1 The information processing device 100 is a cluster 40. tc-1 From the velocity of person h2, cluster 40 at time t tc-1 The distribution of cluster 40 at estimated time t is estimated. tc-1 The distribution of is denoted as "distribution Bc".

[0032] The information processing device 100 may specify the velocity of the person h2 in any way. For example, the information processing device 100 may specify the velocity of the person h2 in the cluster 40. tc-1 The average value, median value, etc. of the velocity information of each point included in is used as the velocity of person h2.

[0033] The information processing device 100 is configured to divide the area of ​​the distribution Bc and the cluster 40 tc Compare the positions of each point in the cluster 40 tc Among the points, points that do not exist within a predetermined distance based on the area of ​​distribution Bc are classified into cluster 40 tc The information processing device 100 classifies the identified points into the cluster 40. tc From Cluster 40 td Correct to.

[0034] The information processing device 100 calculates the corrected cluster 40 tb Information on the corrected cluster 40 tc The information is registered in the cluster table 142.

[0035] The information processing device 100 calculates the corrected cluster 40 tb By inputting this to AI50, the posture data of person h1 is 45 tb Furthermore, the information processing device 100 estimates the corrected cluster 40 tc By inputting this to AI50, posture data 45 of person h2 is obtained. tc Estimate.

[0036] As described above, the information processing device 100 according to the present embodiment performs clustering based on spatial distance and clustering based on velocity information, based on point cloud data obtained from the millimeter wave sensor 20. By performing such clustering, even when multiple people are close to each other, the point cloud data can be clustered in association with each person. Therefore, the information processing device 100 can accurately estimate the postures of multiple people using the clustering results.

[0037] Next, a configuration example of the information processing device 100 will be described. Fig. 4 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Fig. 4, the information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0038] The communication unit 110 performs data communication with the millimeter wave sensor 20 and the like via the network 25. For example, the communication unit 110 receives a frame f t Receive.

[0039] The input unit 120 inputs various types of information to the control unit 150 of the information processing device 100 .

[0040] The display unit 130 displays information output from the control unit 150 .

[0041] The storage unit 140 includes an AI50, a point cloud data table 141, a cluster table 142, and an orientation data table 143. The storage unit 140 is a memory or the like.

[0042] AI50 is a machine learning model that takes a point cloud as input and outputs a person's posture data. AI50 is a neural network (NN), etc.

[0043] For example, posture data of a person is data corresponding to skeletal data. Here, the skeletal data is data in which two-dimensional or three-dimensional coordinates are set for multiple joints defined in a skeletal model of the human body. Here, the coordinates of each joint in the skeletal data are three-dimensional coordinates. FIG. 5 is a diagram showing an example of a skeletal model of the human body. For example, as shown in FIG. 5, the skeletal model of the human body is defined by 21 joints ar0 to ar20.

[0044] The relationship between the joints ar0 to ar20 shown in FIG. 5 and the joint names is as shown in FIG. 6. FIG. 6 is a diagram showing an example of joint names. For example, the joint name of joint ar0 is "SPINE_BASE." The joint names of joints ar1 to ar20 are as shown in FIG. 6, and explanations thereof will be omitted.

[0045] The point cloud data table 141 is a table that holds point cloud data generated from frames at each time. FIG. 7 is a diagram showing an example of the data structure of the point cloud data table. As shown in FIG. 7, the point cloud data table 141 associates time with point cloud data. The time is the time when the millimeter wave sensor 20 generates a frame (detection result). The point cloud data is point cloud data generated from frames at the corresponding time. For example, the point cloud data is generated by the preprocessing unit 151, which will be described later. As described above, in the point cloud data, position information (three-dimensional coordinates) and velocity information are associated with each point. Furthermore, it is assumed that each point is assigned point identification information that uniquely identifies the point.

[0046] The cluster table 142 holds information about clusters at each time. Fig. 8 is a diagram showing an example of the data structure of the cluster table. As shown in Fig. 8, the cluster table 142 associates time with first cluster data, second cluster data, and third cluster data.

[0047] The first cluster data includes data resulting from clustering based on spatial distances performed on point cloud data at a certain time. The clustering based on spatial distances is the clustering described in step S10 of FIG. 3. For example, the first cluster data associates cluster identification information that identifies a cluster with point identification information of each point (point cloud) that belongs to the cluster. At each time, each cluster that is assumed to represent the same person is assigned the same cluster identification information.

[0048] The second cluster data includes data of the clustering result based on speed information, which is performed on point cloud data at a certain time. The clustering based on speed information is the clustering described in step S11 of FIG. 3. For example, the second cluster data associates cluster identification information that identifies a cluster with point identification information of each point (point cloud) belonging to the corresponding cluster. At each time, each cluster that is assumed to represent the same person is assigned the same cluster identification information.

[0049] The third cluster data includes data of the clustering result after correction for the second cluster data. The correction for the cluster is the correction described in step S12 of FIG.

[0050] Posture data table 143 holds posture data estimated at each time. FIG. 9 is a diagram showing an example of the data structure of the posture data table. As shown in FIG. 9, posture data table 143 associates time with posture data. The posture data is data corresponding to the skeleton data described in FIG. 5. Person identification information that identifies the person corresponding to the posture data is attached to the posture data.

[0051] Returning to the description of Fig. 4, the control unit 150 has a preprocessing unit 151, a first classification unit 152, a second classification unit 153, a correction unit 154, an estimation unit 155, and an abnormal behavior detection unit 156. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0052] The preprocessing unit 151 receives frame f from the millimeter wave sensor 20 and generates point cloud data based on frame f. The preprocessing unit 151 associates time with the point cloud data and registers the data in the point cloud data table 141. The preprocessing unit 151 repeatedly executes the above process every time it receives frame f.

[0053] For example, the preprocessing unit 151 analyzes the time difference, phase difference, frequency increase / decrease, etc., between the emission of radio waves and the arrival of reflected waves, extracts information on the distance to the person and information on the person's speed, and specifies position information and speed information for each point on the person's surface from the extraction results. Note that the preprocessing unit 151 may generate point cloud data from the frame f using any well-known technology.

[0054] The first classification unit 152 acquires point cloud data from the point cloud data table 141 and performs clustering based on the position information of each point included in the point cloud data. By this clustering, points having similar position information are classified into the same cluster. The first classification unit 152 registers the first cluster data, which is the clustering result, in the cluster table 142 in association with a time.

[0055] The clustering by the first classification unit 152 corresponds to the clustering described in step S10 of Fig. 3. Every time new point cloud data is registered in the point cloud data table 141, the first classification unit 152 repeatedly executes the above process.

[0056] The second classification unit 153 acquires point cloud data from the point cloud data table 141 and performs clustering based on the velocity information of each point included in the point cloud data. By this clustering, points having similar velocity information are classified into the same cluster. The second classification unit 153 registers the second cluster data, which is the clustering result, in the cluster table 142 in association with time.

[0057] The clustering by the second classification unit 153 corresponds to the clustering described in step S11 of Fig. 3. Every time new point cloud data is registered in the point cloud data table 141, the second classification unit 153 repeatedly executes the above process.

[0058] The correction unit 154 corrects the second cluster data, and registers the third cluster data, which is the result of the correction, in the cluster table 142 in association with the time.

[0059] For example, the correction unit 154 obtains the third cluster data corresponding to time t−1, which is one time before the current time t, from the cluster table 142. The correction unit 154 corrects the second cluster data based on the third cluster data corresponding to time t−1 and the second cluster data corresponding to time t1, thereby generating the third cluster data for time t1 and registering the third cluster data for time t1 in the cluster table 142.

[0060] The correction performed by the correction unit 154 corresponds to the correction described in step 12 of Fig. 3. The correction unit 154 repeatedly executes the above process every time new second cluster data is registered in the cluster table 142.

[0061] If the first cluster data and the second cluster data at the same time satisfy the similarity condition, the correction unit 154 registers the first cluster data as the third cluster data in the cluster table 142. For example, if multiple people are far apart, the number of clusters included in the first cluster data and the distribution of each cluster will be similar to the number of clusters and the distribution of each cluster included in the second cluster data. Normally, if multiple people are far apart, the first cluster data has a high reliability, so the first cluster data can be used as the correction result for the second cluster data. The similarity condition may be set in any way, but for example, the similarity condition is a condition that "the number of clusters is the same and the difference in the distribution of each cluster is less than a threshold."

[0062] Estimation unit 155 acquires the third cluster data registered in cluster table 142 and inputs the third cluster data to AI 50, thereby estimating posture data of the person. Estimation unit 155 registers the posture data in posture data table 143 in association with time.

[0063] When the third cluster data includes a cluster for each person, estimation unit 155 inputs the point clouds belonging to the cluster for each person to AI 50, and estimates the posture data of each person. The posture data estimation performed by estimation unit 155 corresponds to the posture data estimation described in step 12 of FIG. 3.

[0064] Estimation unit 155 repeatedly executes the above process at each time, whereby time-series posture data is registered in posture data table 143 for each person (each piece of personal identification information).

[0065] The abnormal behavior detection unit 156 detects abnormal behavior of a person based on the posture data table 143. The abnormal behavior detection unit 156 detects abnormal behavior using a rule table or an AI for abnormal behavior detection. The rule table is a table that associates the transition of each joint position in posture data with the type of abnormal behavior. The AI ​​for abnormal behavior detection is a machine learning model that receives time-series posture data as input and outputs the type of abnormal behavior. The AI ​​for abnormal behavior detection is, for example, a neural network.

[0066] When the abnormal behavior detection unit 156 detects an abnormality, it may transmit abnormal behavior information to a predetermined external terminal. The abnormal behavior information is associated with frames of a predetermined period based on the time when the abnormal behavior was detected, point cloud data, the content of the abnormal behavior, the time, etc.

[0067] Here, an example of a notification in which an alert is sent to a terminal of a monitor monitoring a specific room in a building will be described. The information processing device 100 monitors multiple people in the specific room. The multiple people are in a room where privacy must be maintained. The specific room may be, for example, a room constituting a space where privacy must be maintained, such as a private room in a hospital or hotel, a changing room, a nursing room, or a restroom. In this case, the information processing device 100 identifies point cloud information of a first person constituting the multiple people and point cloud information of a second person constituting the multiple people, for example, based on the clustered point cloud information. Next, the information processing device 100 detects abnormal behavior among the multiple people by, for example, separately analyzing the point cloud information of the identified first person and the point cloud information of the second person. The information processing device 100 then notifies, for example, an alert indicating the occurrence of abnormal behavior to the terminal of the monitor monitoring the room where privacy must be maintained. More specifically, for example, the information processing device 100 displays, on the terminal, the location of the abnormality and an alert indicating the occurrence of abnormal behavior.

[0068] Here, an example of the process of anomaly detection by the information processing device 100 will be described for each case with reference to Fig. 10, Fig. 11, and Fig. 12. Fig. 10 to Fig. 12 are diagrams for explaining the process of anomaly detection by the information processing device.

[0069] First, Fig. 10 will be described. Fig. 10 shows an example of abnormality detection when a nurse h4 is providing rescue care to a patient h3. At time t1, the information processing device 100 detects an abnormality in a frame f t1 By performing the above process, cluster 61 corresponding to patient h3 is obtained. t1b The information processing device 100 identifies the cluster 61 t1b By inputting the point cloud into AI50, posture data 71 t1b The abnormal behavior detection unit 156 obtains the posture data 71 t1b Based on this, the abnormal behavior of "falling" is detected.

[0070] At time t2, the information processing device 100 t2By performing the above process, cluster 61 corresponding to patient h3 is obtained. t2b and cluster 61 corresponding to nurse h4. t2c The information processing device 100 separately identifies the cluster 61. t2b By inputting the point cloud into AI50, posture data 71 t2b The information processing device 100 obtains the cluster 61 t2c By inputting the point cloud into AI50, posture data 71 t2c The abnormal behavior detection unit 156 obtains the posture data 71 t2b Based on this, the abnormal behavior of "falling" is detected.

[0071] As explained in Fig. 10, even if the distance between nurse h4 and patient h3 is short, the abnormal behavior of patient h3, "falling," can be detected. In contrast, in the conventional technique 2, when the distance between nurse h4 and patient h3 is short, the cluster 61 corresponding to patient h3 t2b and cluster 61 corresponding to nurse h4. t2c It is not possible to identify these separately, making it difficult to detect anomalies.

[0072] Next, FIG. 11 will be described. FIG. 11 shows an example of anomaly detection in the private spaces of persons h5 and h6. At time t1, the information processing device 100 detects anomalies in a frame f t1 By performing the above process, cluster 62 corresponding to person h5 is obtained. t1b and cluster 62 corresponding to person h6. t1c The information processing device 100 separately identifies the cluster 62. t1b By inputting the point cloud into AI50, posture data 72 t1b The information processing device 100 obtains the cluster 62 t1c By inputting the point cloud into AI50, posture data 72 t1c get.

[0073] At time t2, the information processing device 100 t2 By performing the above process, cluster 62 corresponding to person h5 is obtained. t2band cluster 62 corresponding to person h6. t2c The information processing device 100 separately identifies the cluster 62. t2b By inputting the point cloud into AI50, posture data 72 t2b The information processing device 100 obtains the cluster 62 t2c By inputting the point cloud into AI50, posture data 72 t2c The abnormal behavior detection unit 156 obtains the posture data 72 t2b Based on this, the abnormal behavior of "falling" is detected.

[0074] As explained in Fig. 11, even if the distance between person h5 and person h6 is short, the abnormal behavior of person h5, "falling," can be detected. Furthermore, based on the processing result of Fig. 11, the information processing device 100 can identify that person h5 bumped into person h6 while passing each other, causing person h5 to fall.

[0075] Next, FIG. 12 will be described. FIG. 12 shows an example of anomaly detection in the private spaces of persons h7 and h8. At time t1, the information processing device 100 detects anomalies in a frame f t1 By performing the above process, cluster 63 corresponding to person h7 is obtained. t1b and cluster 63 corresponding to person h8. t1c The information processing device 100 separately identifies the cluster 63 t1b By inputting the point cloud into AI50, the posture data 73 t1b The information processing device 100 obtains the cluster 63 t1c By inputting the point cloud into AI50, the posture data 73 t1c get.

[0076] At time t2, the information processing device 100 t2 By performing the above process, cluster 63 corresponding to person h7 is obtained. t2b and cluster 63 corresponding to person h8. t2c The information processing device 100 separately identifies the cluster 63 t2b By inputting the point cloud into AI50, the posture data 73 t2bThe information processing device 100 obtains the cluster 63 t2c By inputting the point cloud into AI50, the posture data 73 t2c For example, the abnormal behavior detection unit 156 obtains the posture data 73 t2b Based on this, the abnormal behavior of person h7, "approaching the direction of another person at an abnormal speed and raising his right knee," is detected.

[0077] At time t3, the information processing device 100 t3 By performing the above process, cluster 63 corresponding to person h7 is obtained. t2b The person h8 has fallen down (outside the range of the millimeter wave sensor 20), and point cloud data is not generated for the person h8. t3b By inputting the point cloud into AI50, the posture data 73 t3b get.

[0078] As explained in Fig. 12, even if the distance between person h7 and person h8 is short, the abnormal behavior of person h7, "approaching another person at an abnormal speed and raising his right knee," can be detected. For example, "approaching another person at an abnormal speed and raising his right knee" corresponds to a violent act against another person.

[0079] Next, an example of a processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 13 is a flowchart showing the processing procedure of the information processing device according to this embodiment. As shown in Fig. 13, the pre-processing unit 151 of the information processing device 100 acquires frames from the millimeter wave sensor 20 and generates point cloud data (step S101).

[0080] The first classification unit 152 of the information processing device 100 performs clustering based on the position information of the point cloud data (step S102). The second classification unit 153 of the information processing device 100 performs clustering based on the velocity information of the point cloud data (step S103).

[0081] The correction unit 154 of the information processing device 100 acquires the third cluster data at time t-1 from the cluster table 142 (step S104). The correction unit 154 determines whether to correct the second cluster data based on the third cluster data corresponding to time t-1 and the second cluster data corresponding to time t1 (step S105).

[0082] If correction is to be performed (step S106, Yes), correction unit 154 corrects the second cluster data at time t to generate third cluster data at time t (step S107). On the other hand, if correction is not to be performed (step S106, No), correction unit 154 sets the second cluster data at time t to the third cluster data at time t (step S108).

[0083] The correction unit 154 registers the third cluster data at time t in the cluster table 142 (step S109).

[0084] The estimation unit 155 of the information processing device 100 inputs the third cluster data at time t to the AI50 and estimates posture data (step S110). The abnormal behavior detection unit 156 of the information processing device 100 performs abnormal behavior detection based on the posture data (step S111). The abnormal behavior detection unit 156 outputs the result of the abnormal behavior detection (step S112).

[0085] The processing procedure of the information processing device 100 has been described above with reference to Fig. 13. The information processing device 100 repeatedly executes the processing described with reference to Fig. 13 at each time point.

[0086] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 clusters the point cloud data based on the spatial distance of the point cloud data and the velocity of the point cloud data, and estimates the postures of each of the multiple people based on the clustered point cloud data. This makes it possible to accurately estimate the postures of the multiple people.

[0087] The information processing device 100 sequentially performs clustering on the time-series point cloud data, and adjusts the current clustering result based on the previous clustering result based on the point cloud data and the current clustering result based on the point cloud data, thereby improving the accuracy of the clustering result for each person.

[0088] When the distance between a point included in a cluster of a specific person included in the current clustering result and a cluster corresponding to the specific person included in the previous clustering result is equal to or greater than a threshold value determined based on the speed of the point cloud data, the information processing device 100 classifies the point into a cluster of a person other than the specific person. This makes it possible to improve the accuracy of the clustering result for each person.

[0089] The information processing device 100 identifies abnormal behavior of each of the multiple people based on the skeletal information of each of the multiple people. This makes it possible to detect abnormal behavior of a person even when the millimeter wave sensor 20 is used.

[0090] Next, a description will be given of an example of a hardware configuration of a computer that realizes the same functions as the above-described information processing device 100. Fig. 14 is a diagram showing an example of a hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0091] 14, computer 200 has CPU 201 that executes various types of arithmetic processing, input device 202 that accepts data input from a user, and display 203. Computer 200 also has communication device 204 that exchanges data with external devices via a wired or wireless network, and interface device 205. Computer 200 also has RAM 206 that temporarily stores various types of information, and hard disk drive 207. Each of devices 201 to 207 is connected to bus 208.

[0092] The hard disk drive 207 stores a preprocessing program 207a, a first classification program 207b, a second classification program 207c, a correction program 207d, an estimation program 207e, and an abnormal behavior detection program 207f. The CPU 201 reads out each of the programs 207a to 207f and loads them into the RAM 206.

[0093] The pre-processing program 207a functions as the pre-processing process 206a. The first classification program 207b functions as the first classification process 206b. The second classification program 207c functions as the second classification process 206c. The correction program 207d functions as the correction process 206d. The estimation program 207e functions as the estimation process 206e. The abnormal behavior detection program 207f functions as the abnormal behavior detection process 206f.

[0094] The processing of the preprocessing process 206a corresponds to the processing of the preprocessing unit 151. The processing of the first classification process 206b corresponds to the processing of the first classification unit 152. The processing of the second classification process 206c corresponds to the processing of the second classification unit 153. The processing of the correction process 206d corresponds to the processing of the correction unit 154. The processing of the estimation process 206e corresponds to the processing of the estimation unit 155. The processing of the abnormal behavior detection process 206f corresponds to the processing of the abnormal behavior detection unit 156.

[0095] It should be noted that each of the programs 207a to 207f does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207f.

[0096] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0097] (Appendix 1) Obtain point cloud information generated based on the reflected signals of radar irradiated toward multiple people, clustering the point cloud information based on the spatial distance of the acquired point cloud information and the velocity of the point cloud information; Estimating the posture of each of the plurality of people based on the clustered point cloud information. An estimation program that causes a computer to execute processing.

[0098] (Supplementary Note 2) The acquiring process acquires time-series point cloud information, The clustering process includes sequentially performing clustering on time-series point cloud information, 2. The estimation program according to claim 1, further causing the computer to execute a process of correcting a result of the current clustering based on the result of the previous clustering based on the point cloud information and the result of the current clustering based on the point cloud information.

[0099] (Appendix 3) The estimation program according to Appendix 2, characterized in that the correction process classifies a point included in a cluster of a specific person included in the current clustering result and a cluster corresponding to the specific person included in the previous clustering result, into a cluster of a person other than the specific person, if the distance between the point included in the current clustering result and the cluster corresponding to the specific person included in the previous clustering result is equal to or greater than a threshold determined based on the speed of the point cloud information.

[0100] (Supplementary Note 4) The estimating process estimates skeletal information of each of the plurality of people based on the point cloud information; The estimation program according to claim 1, further comprising causing a computer to execute a process of identifying abnormal behavior of each of the plurality of persons based on skeletal information of each of the plurality of persons.

[0101] (Appendix 5) The estimation program according to Appendix 1, wherein the point cloud information includes positions and velocities of multiple points on the surfaces of the multiple people, and the clustering process performs a first clustering based on the positions of the multiple points included in the point cloud information, and a second clustering based on the velocities of the multiple points included in the point cloud information.

[0102] (Appendix 6) The estimation program according to appendix 1, characterized in that the velocity of the point cloud information is calculated based on the Doppler effect of a radar irradiated toward an object.

[0103] (Supplementary Note 7) Each of the plurality of people is present in a room where privacy should be maintained, The estimation program according to claim 1, characterized in that the estimation process identifies point cloud information of a first person constituting the plurality of people and point cloud information of a second person constituting the plurality of people based on the clustered point cloud information, detects the occurrence of abnormal behavior among the plurality of people by separately analyzing the point cloud information of the first person and the point cloud information of the second person, and notifies an alert indicating the occurrence of the abnormal behavior to a terminal of a monitor monitoring the room where privacy should be maintained.

[0104] (Appendix 8) Obtain point cloud information generated based on the reflected signals of radar irradiated toward multiple people, clustering the point cloud information based on the spatial distance of the acquired point cloud information and the velocity of the point cloud information; Estimating the posture of each of the plurality of people based on the clustered point cloud information. An estimation method characterized in that the processing is performed by a computer.

[0105] (Supplementary Note 9) The acquiring process acquires time-series point cloud information, The clustering process includes sequentially performing clustering on time-series point cloud information, 9. The estimation method according to claim 8, further comprising: correcting the result of the current clustering based on the previous result of clustering based on the point cloud information and the result of the current clustering based on the point cloud information.

[0106] (Appendix 10) The estimation method according to Appendix 9, characterized in that the correction process classifies a point included in a cluster of a specific person included in the current clustering result into a cluster of a person other than the specific person when the distance between the point included in the previous clustering result and the cluster corresponding to the specific person is equal to or greater than a threshold determined based on the speed of the point cloud information.

[0107] (Supplementary Note 11) The estimating process estimates skeletal information of each of the plurality of people based on the point cloud information; The estimation method according to claim 8, further comprising the step of identifying abnormal behavior of each of the plurality of persons based on skeletal information of each of the plurality of persons.

[0108] (Appendix 12) The estimation method described in Appendix 8, wherein the point cloud information includes positions and velocities of multiple points on the surfaces of the multiple people, and the clustering process performs a first clustering based on the positions of the multiple points included in the point cloud information, and a second clustering based on the velocities of the multiple points included in the point cloud information.

[0109] (Appendix 13) The estimation method according to appendix 8, characterized in that the velocity of the point cloud information is calculated based on the Doppler effect of a radar irradiated toward an object.

[0110] (Supplementary Note 14) Each of the plurality of people is present in a room where privacy should be maintained, The estimation method according to Supplementary Note 8, characterized in that the estimation process identifies point cloud information of a first person constituting the plurality of persons and point cloud information of a second person constituting the plurality of persons based on the clustered point cloud information, detects the occurrence of abnormal behavior among the plurality of persons by separately analyzing the point cloud information of the first person and the point cloud information of the second person, and notifies an alert indicating the occurrence of the abnormal behavior to a terminal of a monitor monitoring the room where privacy should be maintained.

[0111] (Appendix 15) Obtain point cloud information generated based on the reflected signals of radar irradiated toward multiple people, clustering the point cloud information based on the spatial distance of the acquired point cloud information and the velocity of the point cloud information; Estimating the posture of each of the plurality of people based on the clustered point cloud information. An information processing device having a control unit that executes processing.

[0112] (Appendix 16) The information processing device described in Appendix 15, characterized in that the control unit acquires time-series point cloud information, sequentially performs clustering on the time-series point cloud information, and further performs a process of correcting the result of the current clustering based on the result of the previous clustering based on the point cloud information and the result of the current clustering based on the point cloud information.

[0113] (Appendix 17) The information processing device described in Appendix 16, characterized in that the control unit classifies a point included in a cluster of a specific person included in the result of the current clustering and a cluster corresponding to the specific person included in the result of the previous clustering, if the distance between the point included in the cluster of a specific person included in the result of the current clustering is equal to or greater than a threshold determined based on the speed of the point cloud information.

[0114] (Appendix 18) The information processing device described in Appendix 15, characterized in that the control unit estimates skeletal information of each of the multiple people based on the point cloud information, and further executes a process of identifying abnormal behavior of each of the multiple people based on the skeletal information of each of the multiple people.

[0115] (Appendix 19) The information processing device described in Appendix 15, characterized in that the point cloud information includes positions and velocities of multiple points on the surfaces of the multiple people, and the control unit performs first clustering based on the positions of the multiple points included in the point cloud information, and performs second clustering based on the velocities of the multiple points included in the point cloud information.

[0116] (Supplementary Note 20) The information processing device according to Supplementary Note 15, characterized in that the speed of the point cloud information is calculated based on the Doppler effect of a radar irradiated toward an object.

[0117] (Supplementary Note 21) Each of the plurality of people is present in a room where privacy should be maintained, The information processing device according to claim 15, wherein the estimation process identifies point cloud information of a first person constituting the plurality of persons and point cloud information of a second person constituting the plurality of persons based on the clustered point cloud information, and detects the occurrence of abnormal behavior among the plurality of persons by separately analyzing the point cloud information of the first person and the point cloud information of the second person, and notifies an alert indicating the occurrence of the abnormal behavior to a terminal of a monitor monitoring the room where privacy should be maintained. [Explanation of symbols]

[0118] 50 AI 100 Information processing device 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 Point Cloud Data Table 142 Cluster Table 143 Attitude Data Table 150 control section 151 Pretreatment section 152 1st classification section 153 Second Classification Division 154 Correction Unit 155 Estimation Department 156 Abnormal Behavior Detection Unit

Claims

1. Obtain point cloud information generated based on the reflected signals of radar irradiated toward multiple people, clustering the point cloud information based on the spatial distance of the acquired point cloud information and the velocity of the point cloud information; Estimating the posture of each of the plurality of people based on the clustered point cloud information. An estimation program that causes a computer to execute processing.

2. The acquiring process acquires time-series point cloud information, The clustering process includes sequentially performing clustering on time-series point cloud information, 2. The estimation program according to claim 1, further causing the computer to execute a process of correcting a current clustering result based on a previous clustering result based on the point cloud information and a current clustering result based on the point cloud information.

3. The estimation program according to claim 2, characterized in that the correction process classifies a point included in a cluster of a specific person included in the result of the current clustering into a cluster of a person other than the specific person when a distance between the point included in the result of the previous clustering and a cluster corresponding to the specific person is equal to or greater than a threshold value determined based on the speed of the point cloud information.

4. the estimating process includes estimating skeletal information of each of the plurality of people based on the point cloud information; 2. The estimation program according to claim 1, further causing a computer to execute a process of identifying abnormal behavior of each of the plurality of people based on skeletal information of each of the plurality of people.

5. 2. The estimation program according to claim 1, wherein the point cloud information includes positions and velocities of a plurality of points on the surfaces of the plurality of people, and the clustering process performs a first clustering based on the positions of the plurality of points included in the point cloud information, and a second clustering based on the velocities of the plurality of points included in the point cloud information.

6. 2. The estimation program according to claim 1, wherein the velocity of the point cloud information is calculated based on the Doppler effect of a radar irradiated toward an object.

7. Each of the plurality of people is present in a room where privacy should be maintained, The estimation program according to claim 1, characterized in that the estimation process identifies point cloud information of a first person constituting the plurality of people and point cloud information of a second person constituting the plurality of people based on the clustered point cloud information, detects the occurrence of abnormal behavior among the plurality of people by separately analyzing the point cloud information of the first person and the point cloud information of the second person, and notifies an alert indicating the occurrence of the abnormal behavior to a terminal of a monitor monitoring the room where privacy should be maintained.

8. Obtain point cloud information generated based on the reflected signals of radar irradiated toward multiple people, clustering the point cloud information based on the spatial distance of the acquired point cloud information and the velocity of the point cloud information; Estimating the posture of each of the plurality of people based on the clustered point cloud information. An estimation method characterized in that the processing is performed by a computer.

9. Obtain point cloud information generated based on the reflected signals of radar irradiated toward multiple people, clustering the point cloud information based on the spatial distance of the acquired point cloud information and the velocity of the point cloud information; Estimating the posture of each of the plurality of people based on the clustered point cloud information. An information processing device having a control unit that executes processing.

Images