Detection determination device and human sensor system

Abstract

To provide a detection determination device for solving a problem that an object disappears (a person cannot be detected) for long time when there is almost no movement in the same place in a millimeter wave radar.SOLUTION: Provided is a detection determination device for detecting a target indoors or outdoors by using a millimeter wave and determining whether the target is an object, the detection determination device comprising a point group data control unit for controlling a point group data corresponding to the target based on millimeter wave data, where the point group data control unit calculates a signal-to-noise ratio (SNR) of the target, and determines a state of the target by using the signal-to-noise ratio.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a detection and determination device capable of detecting a person indoors or outdoors, and a human detection system including the same. [Background technology]

[0002] Conventionally, motion sensors that use millimeter-wave electromagnetic waves with frequencies of 30 GHz to 300 GHz have been developed. For example, Patent Document 1 discloses an intrusion detection system that uses millimeter waves to detect an object (person) entering a monitored area. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-98637 Summary of the Invention [Problem to be solved by the invention]

[0004] Millimeter waves have the advantage of being highly directional, with high range and angular resolution, allowing accurate measurement of the distance and speed of an object. However, they have the physical property that if there is little movement in the same place, the object will disappear for a long period of time (making it impossible to detect a person). Also, if an object that is not a person moves, it may be mistaken for a person. The present invention provides a detection and determination device and a human presence sensor system that use millimeter waves to solve such problems. [Means for solving the problem]

[0005] [1] A detection and determination device capable of detecting an object using millimeter waves indoors or outdoors and determining that the object is a person, a point cloud data control unit that controls point cloud data corresponding to the object based on millimeter wave data; The point cloud data control unit calculates a signal-to-noise ratio (SNR value) of the object and determines the state of the object using the SNR. [2] The detection and determination device according to [1], wherein the point cloud data control unit determines the state of the object and then registers it in a grid list. [3] The detection and determination device according to [2], wherein the point cloud data control unit identifies that the object is in one of the following states: (A) Active state when a new object is detected. (B) Live state, in which the object is recognized as a moving person or object; (C) Vanish state, which is not recognized as an object; (D) Guard state, which is recognized as a motionless or slightly moving person or object; and (E) A transition state from (B) to (D) (Live_Inc state). [4] In the Active state, the SNR value of the object is calculated, and if the SNR value is lower than a predetermined threshold, the object is identified as being in the Vanish state, and the point cloud data of the object is not registered in the grid list or is deleted from the grid list. [5] A detection judgment device as described in claim 3, wherein when point cloud data of the object in the Live state is no longer detected, the device transitions from the Live state to the Guard state, and if a predetermined time elapses in the Guard state and the object is not re-detected as a point cloud object, the device is judged to be in the Vanish state, and if the object is re-detected, the device transitions to the Live state. [6] The human presence sensor system described in [1], wherein the point cloud data control unit continuously generates frames of the measurement area and identifies the object by overlaying the resulting multiple frames. [7] A human presence sensor system that detects people indoors or outdoors, [1] to [6], and a detection device according to any one of [1] to [6]. a millimeter wave radar including a transmitter for transmitting millimeter wave radio waves and a receiver for measuring the millimeter wave radio waves reflected by an object; A human presence sensor system including a display unit that displays a grid plan view corresponding to a millimeter wave reception area and the grid corresponding to the object. [Effects of the Invention]

[0006] The detection and determination device of the present invention uses the SNR value, preferably by assigning a detection target to multiple statuses, to overcome the physical characteristics of millimeter-wave radar detection, where if there is little human movement in the same place, the person may be detected and disappear (become unable to be detected), and the system can discriminate without disappearance. Also, preferably, even if an object is identified as a person, the state of the object can be changed to identify it as something other than a person (e.g., a non-living object) by understanding the state of the object thereafter. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a human presence sensor system according to the present invention. [Figure 2] FIG. 2 is a flowchart of a detection and judgment process using the detection and judgment device of the present invention. [Figure 3A] FIG. 3A shows an example of a frame synthesis after noise filtering in the detection and determination device of the present invention. [Figure 3B] FIG. 3B shows an example of the change over time of the point cloud in the detection and determination device of the present invention. [Figure 4] FIG. 4 is a graph showing the number of point clouds over time for one object in a filtered frame in the detection and determination device of the present invention. [Figure 5] FIG. 5 is a diagram showing the steps of measurement grid processing in the detection and determination device of the present invention. [Figure 6]FIG. 6 is a diagram showing the steps of performing object determination processing in the detection and determination device of the present invention. [Figure 7] FIG. 7 is an example of an image of the interior of a room viewed from the millimeter wave radar 2. [Figure 8] FIG. 8 is a plan view of the room of FIG. [Figure 9] FIG. 9 is an example of an image of the interior of a room viewed from the millimeter wave radar 2. [Figure 10] FIG. 10 is a plan view of the room of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0008] The detection and judgment device and human presence sensor system of the present invention will be described in detail below, but the present invention is not limited to the description below, and those skilled in the art can make modifications as necessary based on their knowledge as long as the intent of the present invention is not compromised. In this specification, a person detected by the detection and judgment device does not only include a person but also any living organism, while an object judged by the detection and judgment device means an object that is not a person or other living organism (non-living object).

[0009] Figure 1 is an example of a configuration diagram of a human presence sensor system of the present invention. As shown in Figure 1, the human presence sensor system 1 of the present invention can be composed of a millimeter wave radar 2, a detection and judgment device 3, and a display unit 4. The connections between these devices can be wireless or wired as long as data can be sent and received, and they may also be connected over the Internet.

[0010] [Millimeter wave radar] The millimeter-wave radar 2 includes a transmitter 21 that transmits millimeter-wave radio waves and a receiver 22 that measures the radio waves reflected by a target object. Here, the transmitter 21 and the receiver 22 do not have to be located in the same housing, but may be located separately. In this specification, the millimeter-wave radar 2 is conceptually defined as an entity that includes the transmitter 21 that transmits millimeter-wave electromagnetic waves and the receiver 22 that receives them. The millimeter wave radar 2 is not particularly limited as long as it includes a transmitter 21 that transmits millimeter wave radio waves and a receiver 22 that measures the radio waves reflected by an object.

[0011] [Detection and judgment device] The configuration of the detection and determination device 3 is shown in Fig. 1(b). The detection and determination device includes a point cloud data control unit 31, which includes an area determination unit 311, an object identification unit 312, and an object state determination unit 313, an information receiving unit 32, an information transmitting unit 33, and also includes a storage unit 34 for storing information. Data from the millimeter-wave radar 2 is received by the information receiving unit 32, and the obtained information is analyzed in the point cloud data control unit 31, and the result is sent to the display unit 4 by the information transmitting unit 33.

[0012] The detection and judgment device 3 only needs to have the function of receiving, analyzing, and transmitting information from the millimeter-wave radar 2. The point cloud data control unit 31 is composed of a microcomputer capable of controlling signals such as a CPU, other information analysis devices, memory, etc., in order to analyze the information from the millimeter-wave sensor 2. The millimeter-wave radar 2 may also have a function for acquiring the signal-to-noise ratio (SNR value) of an object, or an SNR value acquisition device different from the millimeter-wave radar 2 may be used. Conventional technology can be used to acquire the SNR value.

[0013] (Point cloud data control unit 31) The detection and determination device 3 of the present invention includes a point cloud data control unit 31. By including the point cloud data control unit 31, an object (a person or a non-personal object (hereinafter simply referred to as an "object")) is detected in the antenna effective range (measurement area) 6. The point cloud data control unit 31 can include an area determination unit 311, an object identification unit 312, and an object state determination unit 313, and the roles of each unit will be described below.

[0014] (Area determination unit 311) In the present invention, the area determination unit 311 acquires frames (point clouds) from millimeter waves reflected by targets within the antenna effective range 6 of the millimeter-wave radar 2. The acquired frames generate information for identifying targets in the target identification unit 312, which will be described later.

[0015] The stacked frame used for the object identification unit 312 can be obtained by the area determination unit 311 as follows, with reference to the flowchart in FIG. First, detection by the millimeter-wave radar 2 is initiated, and the detection judgment device 3 is activated and started (AS1). Next, a grid of a predetermined size is virtually placed in the antenna effective range 6 of the millimeter-wave radar 2 (AS2). Here, the predetermined size can be changed depending on the performance of the millimeter-wave radar 2, the measurement area, and the area to be displayed, but for example, a 50 cm x 50 cm grid can be virtually placed. By placing this grid, it is possible to identify points in the measurement target space.

[0016] Next, the millimeter wave radar 2 is used to acquire point cloud data (AS3). At this time, the direction of the radar is changed as necessary to obtain a group of points that are thought to be objects. In Figure 3, f j-2 Now, let's consider the object f of O1. j-1 Now, let's consider the object of O2, f j This will give you a point cloud of the O3 object, but at this point it is recognized as a collection of points rather than an object. Figure 3f j-2 , f j-1 , f j In the above, noise no1, noise no2, and noise no3 are also included. The noises are outliers in the acquired point cloud. Using conventional noise cancellation techniques, the noises are removed by a filter (AS4). Here, a point cloud is, in principle, a representation of the position and shape of an object's surface at points by measuring the relative distance to the object in a pulse-like manner, but does not represent the situation inside the object. By repeating AS3 to AS5, basic data for the point cloud can be obtained.

[0017] Then, in FIG. 3A, frame f j-2 , f j-1 , f j A composite frame fk can be obtained by synthesizing the three objects O1, O2, and O3 (AS6 in FIG. 2). An example of the change over time in the number (n) of point clouds that make up an object is shown in Figure 3B. The horizontal axis indicates the nth fn. If the number of point clouds is higher than a threshold, the object that the point cloud makes up is recognized as an object (person or object). A specific example is shown in Figure 4. This shows the number of points of a specific target over time in the obtained point cloud frames. Then, multiple overlapping frames are clustered and distinguished into point cloud aggregates (point cloud target data). The distinguished point cloud aggregates are recognized as objects and become data elements of the target object identification unit 312, which determines whether they are people or objects, and then performs measurement grid processing (AS7). Furthermore, after measurement grid processing, object determination processing (AS8) is performed to determine the object.

[0018] (Target identification unit 312) The target identification unit 312 performs measurement grid processing to identify the obtained object. The flow of the process performed by the target identification unit 312 until the point cloud set is identified as an object will be described with reference to FIG.

[0019] First, the point cloud that constitutes the object is placed on a map (BS1). Here, the map is a map of the measurement area of ​​the millimeter wave radar 2 that has been acquired in advance, and by placing the point cloud on this map, it becomes clear where the points of the point cloud are most frequently located.

[0020] Next, the point cloud is clustered (BS2). Clustering here means grouping densely packed points (point cloud collection) into a single object. Clustering can be performed, for example, based on the distance between points or the density of points. Next, the points in the point cloud collection are connected (BS3), and the connected points are further clustered and recognized as an object (BS4). The obtained object has information such as coordinate axes and SNR, and is given a predetermined name and registered in the grid list (BS5).The named object is then updated (BS6).

[0021] Next, since an object contains many points, the same points are deleted based on the center of the object (BS7). This deletion reduces the data volume.

[0022] The object thus obtained is subjected to the following object determination process (BS8) to determine whether it is a person or an object. The object determination process will be explained in detail in the next section. In the object determination process, the object is determined and updated in BS9. If the object is a physical object, it is deleted from the list by updating BS9.

[0023] (Target state determination unit 313) The target state determination unit 313 has a function of determining the state of an object configured by point data, which will be explained with reference to FIG. When a new object is detected in BS1-7, as explained above, it transitions from the idle state (IDLE state) to the active state (state (A)) (CS1). From this state onwards, the state of the object (including signal noise) is determined.

[0024] In the Active state, if there is movement within a specified time, the state changes to CA, but if there is no movement of the object within the specified time, the state changes to Vanish, which will be described later, and the object is deleted by the target state determination unit 313 through an update of BS9.

[0025] It is determined whether an object has existed for a predetermined time from the CA state in Figure 6. An object that has existed for a predetermined time becomes a BA state (CS4), and a newly recognized object is determined to be an AA state (CS5). If the object is in the BA state (an object that has existed for a predetermined time), it is determined that the object is in the Live state (CS8). Alternatively, even if the object is in the AA state (a new object), if the number of points in the point cloud or the signal-to-noise ratio (SNR value) is higher than a threshold, it becomes the Live state (CS6). On the other hand, if the number of points in the point cloud is small or the signal-to-noise ratio (SNR) is lower than a threshold from the AA state, it becomes the Vanish state (CS7) and is deleted from the target state determination unit 313. The Live state is a state in which the object is determined to be a moving person or object.

[0026] In the Live state, the object has been continuously measured, but if the object stops moving, the state transitions to the Live_inc state (state (E)) (CS10). The Live_inc state is a transitional state from the Live state to the Guard state (D) below. In the Live_inc state, the time during which there is no movement is counted. If movement is detected within a predetermined time, the state transitions to the Live state (state B) in which the object is recognized as moving (CS11). On the other hand, if no movement is detected within a predetermined time, the state transitions to the Guard state (D) below (CS12). Note that from the Live_inc state, if the object's movement distance, maximum movement distance, and movement range satisfy predetermined conditions, the state transitions to the Vanish state (state (C)), in which it is recognized that there is no real object, and it is deleted from the target state determination unit 313 (CS14, CS17). Even if the object's movement distance, maximum movement distance, and movement range satisfy the predetermined conditions, if the object was previously determined to be a person, the state transitions to the Guard state (CS14, CS16).

[0027] Also, even if the device is in the Live state, if it moves outside the antenna effective range 6 and remains motionless for a long period of time, it will enter the Vanish state (CS9) described below.

[0028] The Guard state D above is a state in which the target is recognized as an object, but remains motionless. If there is movement during the Guard state, the state returns to the Live state (CS13), which indicates human movement. On the other hand, if the state of no target movement continues until a predetermined timeout threshold, the target is identified as not being an object, and the state transitions to the Vanish state (state (C)) (CS18). The timeout threshold is set appropriately by a person skilled in the art, but it is preferable to set it to a time period that would prevent a normal human from remaining motionless for a long period of time.

[0029] The Vanish state (state (C)) refers to a state in which this object is deleted from the object state determination unit 313 (state machine). In other words, the Vanish state is a state in which the object is determined to be something other than an object. The Vanish state is entered when a predetermined condition is met in the Live_Inc state (state (E)), or when there is no movement after the timeout threshold has elapsed in the Guard state (state (D)), or when a person moves out of the antenna effective range 6 even in the Live state (state (B)).

[0030] Note that measurement grid processing and object judgment operations are performed whenever cloud data is obtained, but measurement grid processing is affected by the results of the object judgment processing performed before it, and object judgment processing is also affected by the results of measurement grid processing.

[0031] (device) In the embodiment, the detection and determination device 3 equipped with such a point cloud data control unit 31 can be an electronic device that can perform the above detection and control. The electronic device may be, for example, a computer, a server, a workstation, a laptop computer, a smartphone, etc., but the embodiment of the present invention is not limited to these.

[0032] The detection and determination device 3 according to the embodiment of the present invention includes a processor (e.g., a central processing unit (CPU)) and a memory. The memory is connected to the processor. The memory may store various data and may store programs. The programs can be executed under the control of the processor.

[0033] In one embodiment, the functionality of the detection determination device 3 may be integrated into a processor, where the processor may be configured to implement the detection determination method described in the embodiments.

[0034] In another embodiment, the detection judgment device 3 may be arranged in separate processors for each function, or the detection judgment device 3 may be a chip connected to a processor and configured to realize the functions of the detection judgment device 3 under the control of the processor.

[0035] For example, the processor may acquire point cloud data using millimeter wave radar, and determine whether the point cloud data acquired within a predetermined period is an object using the function of the point cloud data control unit.

[0036] The detection and judgment device 3 may be connected to a display unit 4 that displays a grid plan view corresponding to the millimeter wave receiving area and a grid corresponding to the target, and may be used together with the millimeter wave radar 2 to form a human presence sensor system 1.

[0037] A computer-readable program is provided that, when executed in the detection and determination device 3, causes a computer to execute the detection and determination method described in the embodiments in the electronic device.

[0038] Furthermore, a storage medium is provided that stores a computer-readable program for causing the detection and determination device 3 to execute the detection and determination method.

[0039] The above-described apparatus and method of the present invention may be realized by hardware or a combination of hardware and software. The present invention relates to a computer-readable program that, when executed by a logic unit, causes the logic unit to realize the above-described apparatus or components, or to implement the above-described various methods or steps. The present invention also relates to a storage medium for storing the above-described program, such as a hard disk, magnetic disk, optical disk, DVD, flash memory, etc.

[0040] The methods / apparatuses described with reference to the embodiments of the present invention may be implemented in hardware, software modules executed by a processor, or a combination of both. For example, one or more of the functional block diagrams shown in the drawings, or one or more combinations of the functional block diagrams, may correspond to software modules in a computer program flow or hardware modules. These software modules may correspond to steps shown in the drawings. These hardware modules may be implemented by implementing these software modules in hardware, for example, using a field programmable gate array (FPGA).

[0041] The software module may be located in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, mobile hard disk, CD-ROM, or any other form of storage medium known to those skilled in the art. The storage medium may be connected to the processor so that the processor reads information from or writes information to the storage medium, or the storage medium may be a component of the processor. The processor and the storage medium are located in an ASIC. The software module may be stored in the memory of the mobile terminal or in a memory card inserted into the mobile terminal. For example, if a device (e.g., a mobile terminal) uses a relatively large-capacity MEGA-SIM card or a large-capacity flash memory device, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.

[0042] One or more of the functional blocks and / or one or more combinations of functional blocks depicted in the figures may be implemented with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or any suitable combination thereof to perform the functions described herein. One or more of the functional blocks and / or one or more combinations of functional blocks depicted in the figures may be implemented, for example, with a combination of computing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, one or more microprocessors in combination with a DSP communication, or any other configuration.

[0043] (verification) A millimeter-wave radar 2 and a detection and judgment device 3 connected to the millimeter-wave radar 2 were placed in an office, and the appearance of people and the identification of objects by the detection and judgment device 3 were compared. Figure 7 shows the millimeter-wave radar 2 placed in a predetermined position in the office, and displays the positions of people in the room. The image in Figure 7 shows the presence of people P1, P2, P3, P4, and P5. FIG. 8 is a floor plan of the office shown in the image of FIG. 7, displayed on a display connected to the millimeter-wave radar 2 and the detection and judgment device 3. In the floor plan of FIG. 8, the detection and judgment device 3 detects and judges objects based on information obtained by the millimeter-wave radar 2, and those identified as objects are displayed with numbered circles. This floor plan has been grid-processed, and is a grid floor plan. Note that the object numbers are numbers used solely for software management and are different from numbers that identify people in the office (such as employee numbers). As shown in Fig. 8, when a person is present within the antenna effective range 6, the detection and determination device 3 detects the person and can display the person's location. When the person moves, the displayed symbol (in this embodiment, a number in a circle) moves in accordance with the person's movement. In this way, SNR can be used to identify people as objects, and non-people such as office furniture as non-objects.

[0044] Furthermore, an oscillating electric fan was placed as an example of a moving object other than a person. As shown in Figure 9, when the electric fan is placed on the right side of the photograph, it is indicated by a triangle mark 9, which is different from the circle mark that indicates a person, as shown in the plan view of Figure 10. In this way, even if a moving object is detected as something other than a person, it can be detected as something other than a person. At this time, the fan A9 is recognized as being in the following state. First, when the fan A9 is moved and regarded as an object, it is recognized as being in the Active state, and then in the initial stage as being in the Live state. However, since the fan A9 does not move as a clustered object, it is recognized as being in the Live state as being in the Live_Inc state. After that, even if it continues to be regarded as an object, it will be recognized as being in the Guard state or the Live_Inc state. 9, the person at P4 has moved and is now outside the antenna effective range 6 of the millimeter-wave radar 2, so P4 is not displayed in Fig. 10. At this time, the point cloud data control unit 31 sets the state of P4 to the Vanish state and deletes it from the grid list. On the other hand, even in cases where people like P3 and P5 are sitting at a desk and making little movement, such as working, people P3 and P5 can be recognized as objects in Figure 10 (a photo taken a certain time after Figure 8) and do not disappear.

[0045] Although the present invention has been described above with reference to specific embodiments, the above description is merely illustrative and does not limit the scope of protection of the present invention. Various modifications and changes may be made to the present invention without departing from the spirit and principles of the present invention, and these modifications and changes also fall within the scope of the present invention.

Claims

1. A detection and determination device that can detect an object using millimeter waves indoors or outdoors and determine that the object is a person, a point cloud data control unit that controls point cloud data corresponding to the object based on millimeter wave data; The point cloud data control unit calculates a signal-to-noise ratio (SNR value) of the object and determines the state of the object using the SNR.

2. The detection and determination device according to claim 1 , wherein the point cloud data control unit determines a state of the object before registering the object in a grid list.

3. The detection and determination device according to claim 2 , wherein the point cloud data control unit determines whether the object is in one of the following states: (A) Active state when newly detected as an object, (B) Live state in which the object is recognized as a moving person or object; (C) Vanish state, where the object is not recognized; (D) Guard state, which is perceived as a motionless or slightly moving person or object; and (E) A transition state from (B) to (D) (Live_Inc state).

4. 4. The detection and determination device according to claim 3, wherein, in the Active state, an SNR value of the object is calculated, and if the SNR value is lower than a predetermined threshold, the object is identified as being in the Vanish state, and point cloud data of the object is not registered in the grid list or is deleted from the grid list.

5. 4. The detection determination device according to claim 3, wherein when point cloud data of the object in the Live state is no longer detected, the device transitions from the Live state to the Guard state, and if a predetermined time has passed in the Guard state and the object is not re-detected as a point cloud object, the device determines that the object is in the Vanish state, and if the object is re-detected, the device transitions to the Live state.

6. The human presence sensor system according to claim 1 , wherein the point cloud data control unit continuously generates frames of a measurement area and identifies the object by superimposing the resulting frames.

7. A human presence sensor system that detects people indoors or outdoors, A detection device according to any one of claims 1 to 6; a millimeter wave radar including a transmitter for transmitting millimeter wave radio waves and a receiver for measuring the millimeter wave radio waves reflected by an object; A human presence sensor system including a display unit that displays a grid plan view corresponding to a millimeter wave reception area and the grid corresponding to the object.

Images