Wearable monitoring devices

Abstract

This wearable monitoring device (10) comprises: a three-dimensional distance sensor (12) for periodically emitting laser light to a forward region and receiving reflected light from a target; a control module (15) for performing information processing relating to a person among the target on the basis of three-dimensional point group data indicating the distance to each reflection point on the target acquired by the three-dimensional distance sensor (12); and a cuboid case (10a) in which the three-dimensional distance sensor (12) and the control module (15) are internally provided. The control module (15) performs a person extraction determination and determination of a person's action on the basis of the three-dimensional point group data. The surrounding situation is thereby monitored in real time in a stable manner with minimal invasion of privacy, while being of a power-saving type and being sufficiently small to be portable.

Description

Technical Field

[0001] The present invention relates to a wearable monitoring device for monitoring the surrounding situation.

Background Art

[0002] Today, in daily and social life, there is an increasing demand for checking the proximity to others, assisting the movement of disabled people, and reforming work styles (visualization). In response to such demands, various sensing technologies have been proposed.

[0003] For example, Non-Patent Document 1 describes a crowd detection technique using LiDAR (Light Detection and Ranging) that scans a laser beam within a measurement area and captures a reflected signal from the ankle part of the human body as a three-dimensional point cloud signal. Also, a person tracking system that dispersedly arranges a plurality of LiDARs in space to measure the position and behavior of people within the measurement target area and estimates the position of people using a statistical method based on such measurement data has been commercialized. Furthermore, Non-Patent Document 2 describes that many companies are working on development with the issue of AI processing (edge AI) on the mobile device side.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

[0005] However, the human flow detection technology described in Non-Patent Document 1, and conventional human tracking systems, are based on the premise of implementing a computer with sufficient processing power, and no attention has been paid to miniaturization or weight reduction. Furthermore, Non-Patent Document 2 is based on the premise of powerful processing power and image processing by dedicated hardware equivalent to a GPU (Graphics Processing Unit), such as most mobile devices being smartphones, and does not aim to develop simpler wearable monitoring devices.

[0006] The present invention has been made in view of the above, and provides a small, low-power wearable monitoring device that monitors the surrounding environment in real time using a 3D distance sensor designed to minimize privacy infringement. [Means for solving the problem]

[0007] The wearable monitoring device according to the present invention comprises a three-dimensional distance sensor having a light-emitting unit that periodically emits light for distance measurement to a forward area and a light-receiving unit that receives reflected light from a target among the emitted light; a control module that performs information processing on a person in the target based on three-dimensional point cloud data indicating the distance to each reflection point of the target acquired by the three-dimensional distance sensor; and a small case that houses the three-dimensional distance sensor and the control module, wherein the control module includes determination means for determining the actions of the person based on the three-dimensional point cloud data.

[0008] According to the present invention, a three-dimensional distance sensor and a control module for information processing are housed in a small case, making it highly portable and allowing for easy acquisition of information about people in the surrounding area in a desired location. Furthermore, by using a three-dimensional distance sensor with laser light instead of acquiring information about people in the surrounding area using images from an imaging device, privacy is minimized. In addition, since the three-dimensional distance sensor is operated periodically, power consumption is reduced compared to simply operating it continuously. Moreover, by performing the distance measurement operation periodically, it is possible to compensate for the shaking and orientation of the case while carrying it, thereby enabling more stable distance measurement. The light used for distance measurement may be pulsed, sinusoidal, or scanning beam light, or light such as laser light or infrared light having a predetermined dot pattern. Distance measurement may be performed using time difference, phase difference, or the difference in dot patterns between transmission and reception. [Effects of the Invention]

[0009] According to the present invention, it is possible to create a power-saving, portable, and compact device that can stably monitor the surrounding environment in real time with minimal infringement of privacy. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows one embodiment of the wearable monitoring device according to the present invention, where (A) is a diagram showing an example of how each component is housed inside the case, and (B) is a diagram showing how it is worn around the neck. [Figure 2] This diagram illustrates a method of target detection using a 3D distance sensor. (A) is a plan view illustrating laser scanning, (B) is a side view illustrating raster scanning, and (C) is a plan view illustrating target surface detection using laser scanning. [Figure 3] This is a configuration diagram showing the functional blocks of a wearable monitoring device and some of the functional blocks of a mobile communication terminal. [Figure 4] This is a functional configuration diagram, primarily showing the control unit of a wearable monitoring device. [Figure 5] This diagram illustrates the correspondence between y-axis acceleration (gravity-direction acceleration) during walking and frontal orientation detection. (A) is a single frame image showing a street alley, illustrating the vibration state of the mobile communication terminal 100. (B) is the detection data from the y-axis acceleration sensor of the inertial sensor 11 corresponding to the direction of gravity during walking. (C) is a single frame image (showing an alley) and detection data for three steps during a period Tf that includes the timing when the mobile communication terminal 100 is facing the direction in front of the owner. (D) is a single frame image (not showing an alley) and detection data for three steps during a period Ts when the mobile communication terminal 100 is not facing the direction in front of the owner. [Figure 6] This is an explanatory diagram showing the relationship between the vanishing point and the determination of the front view. [Figure 7] This is an explanatory diagram illustrating the creation of a trained model based on feature vectors extracted from inertial data and judgment values. [Figure 8] This is a flowchart showing the pre-training and data collection process. [Figure 9] This is a flowchart showing the pre-training and feature vector creation process. [Figure 10] This is a flowchart showing an example of front-facing detection processing. [Figure 11] This is a flowchart illustrating an example of behavioral classification processing. [Figure 12] This flowchart shows an example of face-to-face interaction processing. [Modes for carrying out the invention]

[0011] Figure 1 shows one embodiment of the wearable monitoring device 10 (hereinafter referred to as "device 10") according to the present invention, where (A) shows an example of the state in which each component is housed inside the case, and (B) shows the state in which it is worn around the neck.

[0012] In Figure 1, the device 10 comprises a case 10a made of, for example, resin and having a predetermined shape, with the necessary components housed inside. The case 10a can take on various shapes, and in this embodiment, it has a flat, for example, rectangular parallelepiped shape. The size of the rectangular parallelepiped is portable, and may correspond to the size of a breast pocket, for example, with dimensions of approximately 6 cm in width (left-right direction), 9 cm in height, and 2-4 cm in depth (front-back direction).

[0013] Inside the case 10a, an inertial sensor 11, a three-dimensional distance sensor 12, an airflow sensor 13, and a power supply unit 14 are arranged, along with a control unit 15 (control module) whose circuitry is designed on, for example, a flexible printed circuit board. The inertial sensor 11 is a small sensor that detects acceleration and angular velocity along the x-axis (left-right direction of the case 10a), y-axis (up-down direction of the case 10a), and z-axis (front-back direction of the case 10a), and preferably comprises sensors composed of MEMS (Micro Electro Mechanical Systems).

[0014] The three-dimensional distance sensor 12 is a small three-dimensional sensor that measures the distance to a target (reflection point) from the emission of infrared laser light into the front area and the signal received from the reflected light at the target. It is preferably composed of MEMS (Micro Electro Mechanical Systems), for example, LiDAR (Light Detection and Ranging). The light transmitting / receiving window portion 121 corresponds to the light emitting portion and the light incident portion of the laser light, and the front-rear direction (z-axis) is aligned with the front surface of the case 10a. Inside the light transmitting / receiving window portion 121, although not visible in the figure, a light emitting portion and a light receiving portion are arranged as is well known. When the three-dimensional distance sensor 12 is configured as LiDAR, it is possible to measure the distance to the target (reflection point) by using the time difference (dToF) from the emission to the reception of the pulsed laser light or by using the phase difference (iToF) of the received laser light with respect to the emitted light (sin wave). Since these measurement methods using LiDAR are well known, the description is omitted. Note that the three-dimensional distance sensor 12 is not limited to MEMS as long as it can be manufactured in a small size.

[0015] Also, a structured light type sensor included in a form of LiDAR can be adopted for the three-dimensional distance sensor 12. This structured light type sensor outputs (projects onto the target) a predetermined dot pattern such as a lattice pattern or a stripe pattern from a dot projector (corresponding to the light emitting portion) using infrared laser light, reflects from the uneven surface of the target, and calculates each distance in the depth direction by applying the principle of triangulation from the distortion of the dot pattern received by an image sensor (corresponding to the light receiving portion) arranged at a predetermined relative position, thereby acquiring the three-dimensional shape of the target surface as three-dimensional point cloud data.

[0016] FIG. 2 is a diagram for explaining a method of detecting a target by the three-dimensional distance sensor 12. (A) is a plan view for explaining laser scanning, (B) is a side view for explaining raster scanning, and (C) is a plan view for explaining detection of the surface of a target by laser scanning. The three-dimensional distance sensor 12 periodically transmits pulsed laser beams b1,…bj,b(j + 1) … while scanning in the circumferential direction. Further, the three-dimensional distance sensor 12 sweeps the laser light in the two-dimensional (x-axis, y-axis) direction by gradually tilting a MEMS mirror (not shown) to perform raster scanning of the laser beam. Note that B shown in the figure indicates the measurement area. Also, by shortening the emission interval of the laser light, reducing the angle (resolution) of adjacent beam lights, or increasing the sampling speed, within a required distance range (for example, several meters), multiple received light data can be obtained from a single target with high resolution, and multiple received light data are acquired as three-dimensional point cloud data so that, for example, the uneven shape of the surface of a human face can be measured.

[0017] In this embodiment, an example in which a dToF type LiDAR is applied will be described. The measurement operation will be described later.

[0018] Returning to FIG. 1, the air volume sensor 13 can adopt various types, and preferably, a known thermal flow sensor configured by MEMS (Micro Electro Mechanical Systems) can be adopted. The air volume sensor 13 is arranged at an appropriate position in the case 10a in a direction aligned with the front-rear direction. The air volume sensor 13 has, for example, a rectangular parallelepiped shape, and a cylindrical flow path 131 is formed across the front and rear. The flow path 131 has its front end exposed on the front surface of the case 10a and its rear end exposed on the rear surface of the case 10a (not visible in FIG. 1(A)), and is formed such that ambient air flows from the front end through the flow path 131 to the rear end.

[0019] A heater (not shown in the figure) and a pair of temperature sensors are placed along the flow path 131. The flow rate of the airflow, i.e., the ambient airflow, is calculated from the temperature difference between the pair of temperature sensors. Based on the calculated airflow, the airflow or stagnation state around case 10a can be determined. Furthermore, the orientation of the airflow sensor 13 relative to case 10a can be set according to the application. As shown in Figure 1, the flow path 131 may be in the front-to-back direction, or in the left-to-right direction, or it may be diagonal or curved along the way.

[0020] The power supply unit 14 can use batteries of a predetermined size, such as AA-size (JIS) dry cell batteries or cylindrical lithium batteries (e.g., 18.5 mm in diameter and 65.3 mm in length), taking into account the size of the case 10a. The power supply unit 14 houses the required number of batteries, for example, four batteries in Figure 1. In this embodiment, the control unit 15 is for low power consumption and is configured, for example, on a single printed circuit board, with a processor (CPU), peripheral components, input / output interfaces, and necessary connectors.

[0021] Case 10a has a suspension member, for example, through holes 10b drilled on the left and right sides of the upper part of case 10a. By passing a cord-like neck strap 20 through these through holes 10b, it can be suspended around the wearer's neck, as shown in Figure 1(B). In a normal posture, such as when the wearer is sitting or standing still, the back surface of the flat case 10a is in contact with the wearer's chest, so that the forward direction (z-axis) of case 10a coincides with the wearer's front direction. On the other hand, case 10a may sway slightly while walking. Depending on the application, it may also be attached to the front of the waist using a belt or similar means to concentrate on measuring the movement of the feet of other people in the surrounding area.

[0022] Figure 3 is a configuration diagram showing the functional blocks of device 10 and some of the functional blocks of the mobile communication terminal 30. The inertial sensor 11, the 3D distance sensor 12, and the airflow sensor 13 are connected to the control unit 15. The control unit 15 efficiently supplies power from the power supply unit 14 to itself and the necessary sensors at appropriate timings. The memory unit 16 includes a control program memory unit 160 for storing various control programs for monitoring, learned model memory units 161 and 162 for storing the respective learned models described later, a measurement data memory unit 163 for storing measurement data from each sensor, and a memory unit as a work area for temporarily storing data in progress.

[0023] A smartphone can be used as the mobile communication terminal 30. As described later, the mobile communication terminal 30 transmits the measurement result data or monitoring results of device 10 to the smartphone of the device 10 owner via a short-range communication unit 157 (see Figure 4) and a short-range communication unit 321, such as Wi-Fi, allowing for more effective use of the monitoring result information via the smartphone. The mobile communication terminal 30 has a display unit 311 in the center of the main body 31 and a speaker 312 at the top, and the monitoring result information is output from the speaker 312 via the audio processing unit 322. By utilizing the monitoring results on the mobile communication terminal 30 side, for example, by installing and using more comprehensive and advanced application functions for visually impaired people, effective voice guidance can be provided, and the power consumption of device 10 can be reduced.

[0024] Figure 4 is a functional configuration diagram of the control unit 15 of device 10. The control unit 15 functions as a front detection unit 151, a step detection unit 152, a point cloud segment creation unit 153, a classification determination unit 154, a behavior classification determination unit 155, a timing unit 156, and a short-range communication unit 157 by reading and executing a control program from the processor. The control unit 15 also performs front detection processing using the trained models stored in the trained model storage unit 161. Furthermore, the control unit 15 performs classification estimation processing and behavior classification estimation processing using each trained model stored in the trained model storage unit 162. Details will be described later.

[0025] The front detection unit 151 detects when the case 10a, which hangs around the wearer's neck, faces the wearer's front while walking. The orientation of the case 10a may swing continuously while walking, but at certain times it will face the wearer's front. Therefore, by causing the 3D distance sensor 12 to perform a measurement operation at the moment the case 10a faces the wearer's front, stable measurement is achieved without losing track of targets located in the wearer's front while conserving power.

[0026] In case 10a, the orientation is determined using the measurement results of the inertial sensor 11, while in this embodiment, a more accurate timing is detected through machine learning. Below, a method for generating a trained model to be applied to machine learning will be described. Various machine learning methods can be used, but supervised learning may be adopted to reduce the processing load. Supervised learning is a machine learning method in which a problem with an associated answer is given to a program that simulates a neural network, and the program learns by gradually and automatically adjusting the parameters (e.g., weight coefficients) within the program so that the answer calculated by the program approaches the prepared answer. A trained model includes at least the adjusted parameters and may also include the program.

[0027] Figures 5 to 7 illustrate the process of determining the front view. Figure 5 illustrates the relationship between y-axis acceleration (gravitational acceleration) during walking and determining the front view. Figure 6 is an explanatory diagram showing the relationship between vanishing points and determining the front view. Figure 7 is an explanatory diagram illustrating the creation of a trained model based on feature vectors extracted from inertial data and judgment values. Note that the so-called pre-training in Figures 7, 5, and 6 required scenery images for determining the front view. Therefore, as shown in Figure 7, a known mobile communication terminal 100 (model name Pixel 3, manufactured by Google), equipped with an inertial sensor 101 (corresponding to inertial sensor 11) and a camera 102 capable of shooting video, was hung around the neck with a string of the same length as the neck strap 20.

[0028] In Figure 5, (A) is a single frame image showing a street alley, illustrating the vibration state of the mobile communication terminal 100; (B) is detection data from the y-axis acceleration sensor of the inertial sensor 11 corresponding to the direction of gravity during walking; (C) is a single frame image (showing an alley) and detection data for three steps during a period Tf that includes the timing when the mobile communication terminal 100 is facing the direction in front of the owner; and (D) is a single frame image (not showing an alley) and detection data for three steps during a period Ts when the mobile communication terminal 100 is not facing the direction in front of the owner.

[0029] Figures 5 and 6 show data from a straight-line walk for approximately 30 seconds from a stationary state to the start of walking, utilizing the inertial sensor 101 and camera 102 mounted on the mobile communication terminal 100. As shown in Figure 6, "perspective lines" indicating distance are extracted from the frame images, and whether the "vanishing point," which is the intersection of the ends of these lines, is located in the central area of ​​the image frame (in the example of Figure 6, the central area of ​​the vertical and horizontal divisions) determines whether the smartphone, i.e., the mobile communication terminal 100, is facing the user directly, using a binary value ("0" or "1"). This binary value is called the determination value (see Figure 7). Depending on the application, the determination may also be made by whether the user is located in the central area of ​​the horizontal division of the forward direction into three equal parts.

[0030] As can be seen from Figures 5(A) to (D), during the walking period of one step, the period Tf that includes the timing when the mobile communication terminal 100 is facing the front of the owner is the period from the middle of the swing phase of one leg until immediately after landing (stance phase), and the period Ts when the mobile communication terminal 100 is not facing the front of the owner is the period from immediately after landing of that leg until the middle of the stance phase.

[0031] As shown in Figure 7, (1) each measurement value from the acceleration sensors and angular velocity sensors on each x, y, and z axis that constitute the inertial sensor 101 is acquired for each axis, from the most recent to a predetermined past time window, (2) each measurement value is standardized for each axis, and (3) pre-set features are extracted. The features are generated as the mean, standard deviation, minimum value, maximum value, and range ([0], [n-1]) for each sensor on each axis, as shown below, and the six axes are combined to form a feature vector. In pre-training, the parameters in the program are adjusted by repeatedly running the machine learning program with multiple inputs of the extracted feature vector and the judgment value which is the answer. Then, once a certain amount of training is completed, it is stored as a trained model in the trained model storage unit 161.

[0032] Figure 8 is a flowchart showing the pre-training and data collection process. Figure 9 is a flowchart showing the trained model creation process. Figure 10 is a flowchart showing the front-facing detection process.

[0033] The pre-training and data collection process shown in Figure 8 is performed using the mobile communication terminal 100. First, each time the sampling period of a predetermined time slot t is reached (steps S1, S3), one frame of an image captured by the camera 102 is acquired (step S5), and then inertial data for 6 axes is acquired by the inertial sensor 101 (step S7). Next, it is determined whether a predetermined time has elapsed (step S9), and if it has not finished, the process returns to step S1 and the same process is repeated, and if it has finished, the flow is exited.

[0034] Next, in Figure 9, the vanishing point for each time slot t is calculated from the frame images obtained for each time slot t, which were obtained in step S5 of Figure 8. As shown in Figure 5, the vanishing point is calculated by performing a perspective line extraction process and determining the intersection point (address) as the vanishing point. Then, it is determined whether the vanishing point is located in the central area of ​​the image frame based on the comparison of both addresses. If it is within the central area, the determination value is set to "1", and if it is outside the central area, the determination value is set to "0" (step S11).

[0035] Next, using the 6-axis data and image data obtained in step S7 of Figure 8, an (n+4)-dimensional vector is created for each axis using the detected data from a time T during walking up to n time slots prior, along with the mean, standard deviation, maximum, and minimum values ​​calculated from the detected data (step S13). Then, the six vector data sets for each axis are combined to create a feature vector (step S15).

[0036] Next, the combined feature vectors and the judgment values ​​indicating whether the object is facing forward are combined in pairs for each time slot t to create a trained model (step S17).

[0037] Next, it is determined whether the acquisition of data for a predetermined distance, such as one step to the left or right, has been completed (step S19). If data acquisition is not complete, the next time slot t is set (step S21), and the same process is repeated. On the other hand, if data acquisition is complete (Yes in step S19), this flow is terminated.

[0038] Furthermore, the determination of the vanishing point position in the frame image in step S11 may be performed automatically using a determination program, or it may be set using a supervised learning method. The learning process in Figures 8 and 9 is not limited to a single run, but is usually performed repeatedly to improve the determination accuracy when actually applied (see, for example, Figure 10), making it more practical.

[0039] The above describes a method in which, during pre-training, images are captured using the camera 102 of the mobile communication terminal 100, and the system is trained to determine whether or not the captured images are facing forward. The pre-training for frontal detection may also be performed in other ways, such as using the mobile communication terminal 100, by obtaining a determination value by referring to angular velocity data of each axis or a predetermined axis of the inertial sensor 11, using the same shape as case 10a, or by obtaining a determination value by referring to measurement data from the 3D distance sensor 12 obtained by walking in a predetermined location for pre-training.

[0040] Figure 10 shows the front detection process in an actual usage scenario of device 10. First, timing is performed, and each time a time slot t is reached (steps S31, S33), detection data from the inertial sensor 11 is acquired (step S35). Next, a feature vector is calculated from the acquired detection data (step S37) and input to the front detection module (front detection unit 151) (step S39).

[0041] The front-facing determination module determines, based on the input feature vector and the trained model, whether device 10 is facing the wearer's front direction relative to the inertial data acquired at time slot t (step S41). If device 10 is determined to be facing the wearer's front direction, the 3D distance sensor 12 is instructed to perform measurement (step S43). Then, it is determined whether to terminate this determination process (step S45). If not, the process returns to step S31; if it is terminated, the flow is exited. Also, if it is determined in step S41 that device 10 is not facing the wearer's front direction, the process skips to step S45. Through the above process, the timing of operation of the 3D distance sensor 12 during walking is determined.

[0042] In the above description, the acceleration and angular velocity of each of the three axes of the inertial sensor 11 were used for determining the front during walking, but this is not limited to that. Depending on the application, for example, only the acceleration in the y-axis direction and the angular velocity in the x-axis direction may be used, or other accelerations or angular velocities may be used.

[0043] Returning to Figure 4, the step determination unit 152 determines whether the wearer of the device 10 around their neck is walking or not. Various methods are known for determining whether or not the wearer is walking. Here, a determination method using the inertial sensor 11 will be described. All or some of the sensor data from the inertial sensor 11 may be used, but in this embodiment, for example, the detected values ​​of the acceleration sensor in the y-axis direction and the angular velocity sensor in the z-axis direction, which are likely to show the most distinctive features, are used. As shown in Figure 7, whether or not the wearer is walking is determined by utilizing the fact that the detected value of the acceleration sensor in the y-axis direction and the detected value of the angular velocity sensor in the z-axis direction increase or decrease periodically in accordance with walking.

[0044] The point cloud segment creation unit 153 extracts (cuts out) point clouds corresponding to people, objects, and backgrounds (walls, etc.) on a target-by-target basis from the 3D point cloud data acquired by the 3D distance sensor 12, as point cloud segments.

[0045] An example of the process of extracting target-level point cloud segments from 3D point cloud data of a 3D distance sensor 12 is described below. When a LiDAR using dToF is employed as the 3D distance sensor 12, as shown in Figure 2(C), when a laser beam b is emitted from the 3D distance sensor 12 in a certain direction, the distance to the reflection point p on the surface of the nearest target (e.g., a person 41) in that direction is accurately measured according to dToF. The set of distance information for each measured reflection point p is defined as 3D point cloud data. The 3D point cloud data includes at least the distance d to each reflection point p on the surface of the target (person 41, background 51, other objects, etc.) and the direction (scan angle θ and raster scan angle).

[0046] Next, the point cloud segment creation unit 153 extracts 3D point cloud data from the 3D point cloud data that can be considered as an individual based on the distance change of the target surface, and creates a point cloud segment. The extraction conditions for the point cloud segment may also include the size (diameter) of the target, the number of reflection points, the surface shape, and the degree of discontinuity (separation distance) between adjacent 3D point cloud data for people, objects, and backgrounds. Furthermore, for example, if the target is a person, the 3D point cloud data may show almost the entire body, or it may show the upper body or facial area, and the extraction conditions for the point cloud segment can be set appropriately according to these cases.

[0047] The classification determination unit 154 determines whether the extracted point cloud segment corresponds to a person, an object, or the background. The classification determination unit 154 may determine which category to classify by comparing it with pre-set determination conditions, but in this embodiment, machine learning is applied, and the determination is made using a pre-trained model obtained in advance. Furthermore, in machine learning, when acquiring parameters for the pre-trained model, the processing burden on the (machine learning) program is reduced by utilizing features related to the point cloud segment to perform appropriate data compression.

[0048] For example, a well-known supervised learning-based, low-intensity classification method called a Support Vector Machine (SVM) can be used as a machine learning technique. On the other hand, for the point cloud segments representing each target as input information, a Gaussian mixture model (GMM) is created for people, objects, and background, or more preferably, as even more compressed Fisher vector features, which are then input into the machine learning program. During pre-training, these Fisher vector features are linked to the answer and input into the Support Vector Machine (SVM), thereby adjusting the parameters of the classification program, which is a Support Vector Machine (SVM). Of the classification program and the adjusted parameters, at least the adjusted parameters are stored in the trained model storage unit 162 as a trained model for classification.

[0049] Then, in actual monitoring using device 10, the classification determination unit 154 identifies point cloud segments corresponding to people. Subsequently, the behavior of the person identified by the classification determination unit 154 is determined by the behavior classification determination unit 155.

[0050] The behavior classification determination unit 155 determines the human behavior identified by the classification determination unit 154 based on the point cloud segments. The behavior classification determination unit 155 also estimates behavioral elements for classifying human behavior, such as position and movement speed.

[0051] First, let's explain the process for estimating a person's position and movement speed. In this embodiment, the movement speed is obtained by performing so-called multi-object tracking (MOT) using a Kalman filter, which calculates the difference between the current position and the position of the person acquired immediately before. As is well known, a Kalman filter is used to estimate values ​​(position and movement speed) that change moment by moment from observations with discrete errors, and multi-object tracking (MOT) is a method of performing tracking based on movement by labeling multiple point cloud segments.

[0052] By knowing the detection locations of other people in the vicinity and the movement of the corresponding detection parts of those people, it becomes possible to estimate the actions of others. In this embodiment, the classification of other people's actions includes, for example, their "relative position and movement speed" and "relative stopping position." Furthermore, in the case where the point cloud segment focuses on the facial area, the movement of the lips and other parts can be used to illustrate various behavioral patterns such as "talking (or being silent)," "eating," and other behaviors, and feature quantities are set for each of these.

[0053] Knowing the movements of others around you while walking is particularly beneficial as a mobility aid for visually impaired individuals. For example, voice guidance such as "There is a person stopped 1 meter ahead" can be provided via speaker 312. In addition to the distance between you and others facing you, the airflow (stagnation) and time spent in close proximity between you, as measured by the airflow sensor 13 and timing unit 156, can also be factors in determining how to avoid close contact with others to reduce the risk of infection.

[0054] Next, the behavior classification unit 155 determines the person's behavior based on the person's position and movement speed information derived from the point cloud segments. The behavior classification unit 155 could also determine which category the behavior falls into by comparing it with pre-set determination conditions, but in this embodiment, machine learning is applied to make the determination using a pre-trained model obtained in advance. Furthermore, in determining the classification of a person's behavior, various features corresponding to the aforementioned behaviors are created and applied as features.

[0055] For example, as a machine learning method, a well-known supervised learning-based, low-intensity classification method called a Support Vector Machine (SVM) can be used, similar to the method applied in the classification determination unit 154. On the other hand, for the human point cloud segments as input information, a Gaussian mixture model (GMM) corresponding to human behavior is applied, or more preferably, further compressed Fisher Vector features are created and then input into the machine learning program. During pre-training, these Fisher Vector features are linked to the answers and input into the Support Vector Machine (SVM), thereby adjusting the parameters of the behavior classification determination program, which is a Support Vector Machine (SVM). Of the behavior classification determination program and the adjusted parameters, at least the adjusted parameters are separately stored in the trained model storage unit 162 as a trained model for behavior classification determination.

[0056] In actual monitoring using device 10, the behavior classification determination unit 155 obtains a behavior classification determination result corresponding to human behavior based on the point cloud segments measured by the 3D distance sensor 12.

[0057] Figure 11 is a flowchart showing an example of behavior classification processing. First, it is determined whether or not an action instruction has been input to the 3D distance sensor 12 (step S51). The action instruction is input, for example, when it is determined that the user is walking, at each moment the device 10 faces the user in front of them (every step), while when it is not determined that the user is walking (e.g., standing still, motionless, or sitting), it is input periodically (e.g., several times per second).

[0058] If no operation instruction is input, the process skips to step S65. If an operation instruction is input, the 3D distance sensor 12 is operated to acquire 3D point cloud data (step S53). Next, point cloud segments are obtained by extracting the 3D point cloud data of each target (step S55). Subsequently, feature value vectors are created from the acquired point cloud segments and input into a classification program to determine whether the point cloud segment is a person, an object, or the background (step S57).

[0059] Next, the position and velocity of the point cloud segment identified as a person are estimated (step S59), and a feature value vector including this position and velocity is created and input into a program for behavior classification determination, thereby determining the behavior of the point cloud segment (step S61).

[0060] Next, processing is performed according to the judgment result (step S63). Processing according to the judgment result refers to the output format corresponding to the judgment result, such as recording to the recording unit, outputting as audio or image, and notifying the mobile communication terminal 30 via short-range communication. Then, it is determined whether or not the process is finished (step S65). If it is not finished, the process returns to step S51; if it is finished, the process is exited.

[0061] Figure 12 is a flowchart illustrating an example of face-to-face behavior processing. Face-to-face behavior processing is a form of behavioral classification that involves monitoring the degree of closeness in a face-to-face situation with another person. First, it is determined whether or not the person is walking (step S71). If they are walking, they exit this flowchart. If they are not walking, a point cloud segment is obtained from the classification determination unit 154, and a behavioral classification is determined based on it (step S73). Next, information such as face-to-face distance, face-to-face situation (presence or absence of conversation), face-to-face time, and airflow stagnation status is comprehensively examined from the point cloud segment to monitor the degree of closeness in relation to the risk of infectious disease infection, and to issue necessary warnings (e.g., "It's a crowded situation.") (step S75).

[0062] For example, the higher the level of stagnation in the airflow, the greater the risk, and a shorter contact time may be set accordingly. Also, if there is another person in the face-to-face position, the closer the face-to-face distance and the longer the conversation time, the greater the risk, and a shorter contact time may be set accordingly. Monitoring and warnings are provided by the device 10 if it is equipped with an audio or image output unit, while if they are provided via the mobile communication terminal 30, the output unit of the mobile communication terminal 30 can be used for both. Once the monitoring operation (step S77) is completed, exit this flow.

[0063] In the embodiments described above, the device 10 was described as being worn around the neck of the wearer, but the present invention is also applicable to a configuration in which the device 10 is installed or attached to a predetermined location. That is, the device 10 can be attached to a suitable location on a wall or shelf in a retail store, home, or facility, or placed using a jig, to understand shopping behavior and measure pedestrian flow in a retail store, or to understand behavior in a home or facility. For shopping behavior, the device 10 is positioned toward a target shopping area in the store, and the number of people gathered or making purchases is counted, or the number of people crossing the measurement area is identified and counted to measure pedestrian flow. For behavioral monitoring, the device 10 is positioned toward a target area, and the behavior of people passing through or appearing in that area is understood.

[0064] As described above, the wearable monitoring device according to the present invention comprises a three-dimensional distance sensor having a light-emitting unit that periodically emits light for distance measurement to a forward region and a light-receiving unit that receives reflected light from a target among the emitted light; a control module that performs information processing on a person in the target based on three-dimensional point cloud data indicating the distance to each reflection point of the target acquired by the three-dimensional distance sensor; and a small case that houses the three-dimensional distance sensor and the control module, wherein the control module preferably includes determination means for determining the actions of the person based on the three-dimensional point cloud data.

[0065] According to the present invention, a three-dimensional distance sensor and a control module for information processing are housed in a small case, making it highly portable and allowing for easy acquisition of information about people in the surrounding area in a desired location. Furthermore, by using a three-dimensional distance sensor with laser light instead of acquiring information about people in the surrounding area using images from an imaging device, privacy is minimized. In addition, since the three-dimensional distance sensor is operated periodically, power consumption is reduced compared to simply operating it continuously. Moreover, by performing the distance measurement operation periodically, it is possible to compensate for the shaking and orientation of the case while carrying it, thereby enabling more stable distance measurement. The light used for distance measurement may be pulsed, sinusoidal, or scanning beam light, or light such as laser light or infrared light having a predetermined dot pattern. Distance measurement may be performed using time difference, phase difference, or the difference in dot patterns between transmission and reception.

[0066] Furthermore, it is preferable that the case also incorporates a power supply unit equipped with a battery to supply operating power to the 3D distance sensor and the control module. With this configuration, the device can be constructed using only the case, without the need for an external battery.

[0067] Furthermore, the case is preferably the size of a breast pocket. With this configuration, such a small size reduces the weight burden and allows for natural walking even when carrying it.

[0068] Furthermore, it is preferable that the present invention includes an inertial sensor housed in the case, the case includes a suspension member, the inertial sensor detects the sway of the case when suspended, and the control module includes an instruction means that instructs the 3D distance sensor to perform a measurement operation based on the detected direction of the case sway. With this configuration, for example, the case may sway while walking, but even in such cases, the orientation of the case can be determined by the inertial sensor, so that the distance measurement operation of the 3D distance sensor can be performed at the timing when the case is oriented in the desired direction.

[0069] Furthermore, it is preferable that the inertial sensor includes at least one of an acceleration sensor and an angular velocity sensor. With this configuration, it becomes possible to know the movement and orientation of the case sequentially and accurately, thereby enabling the 3D distance sensor to perform its measurement operation at a more appropriate timing.

[0070] Furthermore, it is preferable that the instruction means, when the inertial sensor detects a periodic swing of the case, instructs the 3D distance sensor to perform the measurement operation at the timing when it detects that the case is facing forward. With this configuration, it becomes possible to more appropriately determine the timing when the case is facing forward in the case of periodic swings such as those that occur when walking.

[0071] Furthermore, if no vibration of the case is detected, it is preferable that the instruction means instructs the measurement operation of the 3D distance sensor at a predetermined interval. With this configuration, when the case is not vibrating, the orientation of the case is assumed to be facing in a direction that enables the measurement operation of the 3D distance sensor, and measurement operation at a predetermined interval set in advance becomes possible. In this case, the timing of the measurement operation can be set to be shorter or longer than that of the measurement operation during walking, depending on the application.

[0072] Furthermore, in the present invention, it is preferable that the control module is equipped with a timing means, and the determination means measures the detection time by the timing means when a person is detected in the front direction of the case. With this configuration, if it is determined that another person is in the face-to-face position, monitoring can be performed by measuring the time during which the person is in close proximity to another person, taking into account the risk of infection from infectious diseases, and a warning can be issued.

[0073] Furthermore, the present invention preferably includes an airflow sensor that is positioned partially exposed on the outer surface of the case and measures the flow rate of the airflow around the case, and the determination means measures the flow rate of the airflow using the airflow sensor when a person is detected in the front direction of the case. With this configuration, if it is determined that another person is in the opposite position and the airflow between them is small, making it easy for a crowded situation to occur due to stagnation, the risk of infection from infectious diseases increases, and monitoring and warning can be provided by timing the detection period.

[0074] Furthermore, it is preferable that the determination means evaluates the density state using at least one of the detection time and the airflow rate. With this configuration, the density state is evaluated for a certain period of time according to the stagnation of the airflow.

[0075] Furthermore, it is preferable that the present invention includes a short-range communication unit housed within the case and capable of short-range communication with a specific mobile communication terminal equipped with a notification unit, and transmits the 3D point cloud data to the specific mobile communication terminal via the short-range communication unit. With this configuration, the 3D point cloud data can be more accurately and precisely notified from the notification unit, for example, by voice guidance, using an advanced processing program (for example, a dedicated application program) registered in a mobile communication terminal such as a smartphone. [Explanation of symbols]

[0076] 10 Wearable Monitoring Devices 10a case 10b Through hole (for suspension member) 11. Inertial Sensor 12 3D distance sensors 13. Airflow sensor 14 Power supply section 15. Control Unit (Control Module) 151 Frontal determination section (instruction means) 152 Step determination unit 153 Point Cloud Segment Creation Unit 154 Classification judgment unit (judgment means) 155 Behavior classification judgment unit (judgment means) 156 Timekeeping section (timekeeping means) 157 Near Field Communication Department 161,162 Pre-trained model memory

Claims

1. A three-dimensional distance sensor having a light-emitting unit that periodically emits light for distance measurement to a forward area and a light-receiving unit that receives reflected light from a target from the emitted light, A control module that performs information processing about a person in the target based on three-dimensional point cloud data indicating the distance to each reflection point of the target acquired by the three-dimensional distance sensor, The system comprises a small case housing the three-dimensional distance sensor and the control module, The control module is a wearable monitoring device that includes determination means for determining the actions of a person based on the three-dimensional point cloud data.

2. The wearable monitoring device according to claim 1, wherein the case further incorporates a power supply unit equipped with a battery that supplies operating power to the three-dimensional distance sensor and the control module.

3. The wearable monitoring device according to claim 1, wherein the case is chest pocket sized.

4. The case is equipped with an inertial sensor housed inside, The aforementioned case is equipped with a suspension member, The inertial sensor detects the vibration of the case when suspended, The wearable monitoring device according to claim 1, wherein the control module includes an instruction means for instructing the three-dimensional distance sensor to perform a measurement operation based on the detected direction of the case's swing.

5. The wearable monitoring device according to claim 4, wherein the inertial sensor comprises at least one of an acceleration sensor and an angular velocity sensor.

6. The wearable monitoring device according to claim 4, wherein the instruction means, when the inertial sensor detects a periodic vibration of the case, instructs the three-dimensional distance sensor to perform the measurement operation at the timing when it detects that the case is facing forward.

7. The wearable monitoring device according to claim 4, wherein the instruction means instructs the measurement operation of the three-dimensional distance sensor at a predetermined interval when no vibration of the case is detected.

8. The control module includes a timing means, The wearable monitoring device according to claim 1, wherein the determination means measures the detection time by the timing means when a person is detected in the front direction of the case.

9. A portion of the case is positioned on the outer surface of the case and is equipped with an airflow sensor that measures the flow rate of the air around the case. The wearable monitoring device according to claim 8, wherein the determination means measures the flow rate of the airflow using the airflow sensor when a person is detected in the front direction of the case.

10. The wearable monitoring device according to claim 9, wherein the determination means evaluates the density state using information of at least one of the detection time and the flow rate of the airflow.

11. The aforementioned case is housed within a specific mobile communication terminal equipped with a notification unit and a short-range communication unit capable of short-range communication with that terminal. A wearable monitoring device according to any one of claims 1 to 10, wherein the three-dimensional point cloud data is transmitted to the specific mobile communication terminal via the short-range communication unit.

Images